Abstract:
This invention provides a system and method for automatically inferring when a target that is being tracked by an aerial vehicle is doing something significant (e.g. stopping suddenly, changing direction quickly or getting out of track view), and consequently alerting an operator. The alerting system also provides a classification of what the target is doing. It frees the operator from continuously monitoring the imagery stream so the operator can perform other tasks.

Description:
GOVERNMENT LICENSE RIGHTS 
     The U.S. Government may have certain rights in the present invention as provided for by the terms of Contract No. FA8650-04-C-7142 with the Defense Advanced Research Projects Agency. 
    
    
     BACKGROUND TECHNOLOGY 
     Aerial vehicles (AVs) are manned or unmanned aircraft that can be remotely piloted or self-piloted by an onboard computer system and can carry cameras, sensors, communications equipment, or other payloads. They have been used in a reconnaissance and intelligence-gathering role for many years. More recently, AVs have been developed for the purpose of surveillance and target tracking. 
     Autonomous surveillance and target tracking performed by AVs in either military or civilian environments is becoming an important aspect of intelligence-gathering. Typically, when a target is being tracked from aerial vehicles (e.g. an AV), human operators must closely monitor imagery streamed from the aircraft to assess target behavior and ensure that the target continues to be in view. 
     SUMMARY 
     This invention provides a system and method for automatically inferring when the target is doing something significant (e.g. stopping suddenly, changing direction quickly or getting out of track view) and alerting the operator. The alerting system also provides a classification of what the target is doing. It frees the operator from continuously monitoring the imagery stream so the operator can perform other tasks. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Features of the present invention will become apparent to those skilled in the art from the following description with reference to the drawings. Understanding that the drawings depict only typical embodiments of the invention and are not therefore to be considered limiting in scope, the invention will be described with additional specificity and detail through the use of the accompanying drawings, in which: 
         FIG. 1  is a schematic diagram depicting a system for aerial tracking of a ground vehicle according to one embodiment of the invention. 
         FIG. 2  is a simplified block diagram of an entity arranged to implement aspects of the exemplary embodiment. 
         FIG. 3  is a flow chart depicting functions that can be carried out in accordance with the exemplary embodiment. 
         FIG. 4  is a diagram depicting an example of a sensor footprint. 
         FIG. 5  is a diagram depicting the variables needed to calculate an instantaneous track metric. 
         FIG. 6  depicts a forward-looking sensor footprint that has been normalized. 
     
    
    
     DETAILED DESCRIPTION 
     In the following detailed description, embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It is to be understood that other embodiments may be utilized without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense. 
       FIG. 1  is a simplified diagram depicting a system  100  for automatically tracking a target from an AV. As shown in  FIG. 1 , the system includes (1) an AV  112  equipped with at least one sensor  114 , (3) a target  116  and (4) a remote operator  118 . It should be understood that while remote operator  118  is shown in  FIG. 1 , remote operator  118  may be many miles away from AV  112  and target  116 . 
     The AV  112  is an aircraft capable of being remotely piloted or self-piloted. AV  112  may carry cameras, sensors, communications equipment, or other payloads. AV  112  may be a hover-capable aerial vehicle or a fixed-wing aerial vehicle. Sensor  114  may be any device capable of imaging a target, such as a camera or radar. Target  116  may be anything being monitored by AV  112 . For example, target  116  may be a ground-based vehicle, an air-based vehicle, or a person. To acquire a target, AV  112  typically sends images from sensor  114  to remote operator  118 . Remote operator  118  then defines an area in the image as the target, and sends the target to AV  112 . 
     Remote operator  118  may be any device capable of communicating with AV  112 . In addition, remote operator  118  may be configured to remotely control AV  112 . Remote operator  118  may be a device such as a desktop computer equipped with a joystick, laptop, or personal data assistant (“PDA”), for example. 
     Aspects of the present invention may be carried out by AV  112  and/or remote operator  118  (or any other entity capable of controlling AV  112 ).  FIG. 2  depicts functional components that may be included in AV  112  and/or remote operator  118  to carry out various aspects of the invention. As shown in  FIG. 2 , the components include a communication interface  200 , a processing unit  202 , and data storage  206 , all of which may be coupled together by a system bus, network, or other mechanism  210 . 
     Communication interface  200  comprises a mechanism for communicating over an air interface, so as to facilitate communication between AV  112  and remote operator  118 . Further, communication interface  200  may include one or more antennas to facilitate air interface communication. 
     Processing unit  202  comprises one or more general purpose processors (e.g., INTEL microprocessors) and/or one or more special purpose processors (e.g., digital signal processors). Data storage  204 , in turn, comprises one or more volatile and/or non-volatile storage mechanisms, such as memory and/or disc-drive storage for instance, which may be integrated in whole or in part with processing unit  202 . 
     As shown, data storage  204  includes program logic  206  and reference data  208 . Program logic  206  comprises one or more logic modules (applications), and preferably includes machine language instructions executable by processing unit  204  to carry out various functions described herein, such as (1) identifying the coordinates of the footprint of sensor  114 , (2) computing a normalized track metric as a function of how close the target is to the center of the footprint, (3) comparing the track metric to a threshold value, and (4) alerting remote operator  118  if the track metric (or its time derivatives) falls outside a threshold value. Reference data  208 , in turn, may include data such as imaging data acquired by sensor  114 . 
       FIG. 3  is a flow chart depicting automatically alerting a remote operator about the status of a vehicle being tracked by an AV in accordance with an embodiment of the invention. In particular,  FIG. 3  depicts (1) identifying the coordinates of the footprint of sensor  114 , (2) computing a normalized track metric as a function of how close the target is to the center of the footprint, (3) comparing the track metric to a threshold value, and (4) alerting remote operator  118  if the track metric (or its time derivatives) falls outside the threshold value. 
     As shown in  FIG. 3 , at step  302 , AV  112  identifies the coordinates of the vertices and center of the footprint (i.e., the viewing window) of sensor  114 . Examples of sensor footprints are depicted in  FIG. 4 . As shown in  FIG. 4 , AV  112  is equipped with forward and side looking sensors. Forward looking sensor footprint  402  includes vertices {a, b, c, d}. The center of footprint  402  is identified as {i}. Side-looking sensor footprint  404  includes vertices {e, f, g, h}. The center of side-looking sensor footprint is identified as {j}. 
       FIG. 6  depicts a forward-looking sensor footprint that has been normalized (i.e., displayed as a rectangle). As shown in  FIG. 6 , the footprint includes vertices {a, b, c, d}, center {i} midpoints {ad c , ab c , bc c , dc c }, and angles 
               {         α   v     2     ,       α   h     2       }     ,         
where α h  and α v  are the horizontal and vertical field of view angles for sensor  114 .
 
     Returning to  FIG. 3 , the coordinates of the vertices and center of the sensor footprint may be computed using the following data: 
     [α h , α v ], the horizontal and vertical field of view for sensor  114 ; 
     [θ, φ, ψ], the attitude angles of AV  112 , where θ, is the pitch, φ is the roll, and ψ is the yaw. In this example climb requires a positive pitch, the right wing down is a positive roll and clockwise from the top of the vehicle is a positive yaw; 
     [θ c , φ c , ψ c ], the attitude angles of sensor  114 , where θ, is the pitch, φ is the roll, and ψ the yaw. In this example, pitch is measured between 0 and 90 degrees measured from straight down. The Camera lookdown angle is (1−θ c ), the roll angle is positive right and the yaw angle is positive in the clockwise direction. Consequently, a forward facing sensor  114  has ψ c =0, while a left-pointing camera has a ψ c =−90 degrees; and 
     [N, E, h], the position coordinates of AV  112  where N=north, E=east, and h=height from some reference point (such as UTM northings, eastings and altitude). 
     The local coordinates of the vertices and center of the footprint are identified as follows: 
     
       
         
           
             
               [ 
               
                 
                   
                     a 
                   
                 
                 
                   
                     b 
                   
                 
                 
                   
                     c 
                   
                 
                 
                   
                     d 
                   
                 
                 
                   
                     i 
                   
                 
               
               ] 
             
             = 
             
               [ 
               
                 
                   
                     
                       tan 
                       ⁡ 
                       
                         ( 
                         
                           
                             α 
                             v 
                           
                           2 
                         
                         ) 
                       
                     
                   
                   
                     
                       - 
                       
                         tan 
                         ⁡ 
                         
                           ( 
                           
                             
                               α 
                               h 
                             
                             2 
                           
                           ) 
                         
                       
                     
                   
                   
                     1 
                   
                 
                 
                   
                     
                       tan 
                       ⁡ 
                       
                         ( 
                         
                           
                             α 
                             v 
                           
                           2 
                         
                         ) 
                       
                     
                   
                   
                     
                       tan 
                       ⁡ 
                       
                         ( 
                         
                           
                             α 
                             h 
                           
                           2 
                         
                         ) 
                       
                     
                   
                   
                     1 
                   
                 
                 
                   
                     
                       - 
                       
                         tan 
                         ⁡ 
                         
                           ( 
                           
                             
                               α 
                               v 
                             
                             2 
                           
                           ) 
                         
                       
                     
                   
                   
                     
                       tan 
                       ⁡ 
                       
                         ( 
                         
                           
                             α 
                             h 
                           
                           2 
                         
                         ) 
                       
                     
                   
                   
                     1 
                   
                 
                 
                   
                     
                       - 
                       
                         tan 
                         ⁡ 
                         
                           ( 
                           
                             
                               α 
                               v 
                             
                             2 
                           
                           ) 
                         
                       
                     
                   
                   
                     
                       - 
                       
                         tan 
                         ⁡ 
                         
                           ( 
                           
                             
                               α 
                               h 
                             
                             2 
                           
                           ) 
                         
                       
                     
                   
                   
                     1 
                   
                 
                 
                   
                     0 
                   
                   
                     0 
                   
                   
                     1 
                   
                 
               
               ] 
             
           
         
       
     
     At step  304 , the local coordinates of the midpoints for each side of the sensor footprint are identified as follows: 
     
       
         
           
             
               [ 
               
                 
                   
                     
                       ab 
                       c 
                     
                   
                 
                 
                   
                     
                       bc 
                       c 
                     
                   
                 
                 
                   
                     
                       dc 
                       c 
                     
                   
                 
                 
                   
                     
                       ad 
                       c 
                     
                   
                 
               
               ] 
             
             = 
             
               [ 
               
                 
                   
                     
                       tan 
                       ⁡ 
                       
                         ( 
                         
                           
                             α 
                             v 
                           
                           / 
                           2 
                         
                         ) 
                       
                     
                   
                   
                     0 
                   
                   
                     1 
                   
                 
                 
                   
                     0 
                   
                   
                     
                       tan 
                       ⁡ 
                       
                         ( 
                         
                           
                             α 
                             h 
                           
                           / 
                           2 
                         
                         ) 
                       
                     
                   
                   
                     1 
                   
                 
                 
                   
                     
                       - 
                       
                         tan 
                         ⁡ 
                         
                           ( 
                           
                             
                               α 
                               v 
                             
                             / 
                             2 
                           
                           ) 
                         
                       
                     
                   
                   
                     0 
                   
                   
                     1 
                   
                 
                 
                   
                     0 
                   
                   
                     
                       - 
                       
                         tan 
                         ⁡ 
                         
                           ( 
                           
                             
                               α 
                               h 
                             
                             / 
                             2 
                           
                           ) 
                         
                       
                     
                   
                   
                     1 
                   
                 
               
               ] 
             
           
         
       
     
     At step  306 , each local coordinate is transformed to global inertial coordinates by multiplying the coordinate by pitch-roll-yaw rotation matrices [R] and [R c ], where
 
 [R]=[R (θ)][ R (φ)][ R (ψ)]; and
 
 [R   c   ]=[R (θ c )][ R (φ c )][ R (ψ c )].
 
Thus,
 
 A=a[R][R   c ]
 
 B=b[R][R   c ]
 
 C=c[R][R   c ]
 
 D=d[R][R   c ]
 
 I=i[R][R   c ]
 
 AB   c   =ab   c   [R][R   c ]
 
 BC   c   =bc   c   [R][R   c ]
 
 DC   c   =dc   c   [R][R   c ]
 
 AD   c   =ad   c   [R][R   c ]
 
     Rotational matrices are well known in the art, and are not described in detail here. 
     At step  308  the scaled coordinates of the sensor footprint are computed by scaling the inertial coordinates by the height (h) that AV  112  is flying above the ground (if target  116  is a ground target), or the height of AV  112  is flying above the target  116  (if target  116  is not necessarily on the ground). The footprint is calculated as follows: 
     
       
         
           
               
             
               
                 
                   
                     
                       A 
                       g 
                     
                     = 
                       
                     ⁢ 
                     
                       A 
                       × 
                       
                         h 
                         
                           A 
                           ⁡ 
                           
                             ( 
                             3 
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       B 
                       g 
                     
                     = 
                       
                     ⁢ 
                     
                       B 
                       × 
                       
                         h 
                         
                           B 
                           ⁡ 
                           
                             ( 
                             3 
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       C 
                       g 
                     
                     = 
                       
                     ⁢ 
                     
                       C 
                       × 
                       
                         h 
                         
                           C 
                           ⁡ 
                           
                             ( 
                             3 
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       D 
                       g 
                     
                     = 
                       
                     ⁢ 
                     
                       D 
                       × 
                       
                         h 
                         
                           D 
                           ⁡ 
                           
                             ( 
                             3 
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       I 
                       g 
                     
                     = 
                       
                     ⁢ 
                     
                       I 
                       × 
                       
                         h 
                         
                           I 
                           ⁡ 
                           
                             ( 
                             3 
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       AB 
                       cg 
                     
                     = 
                       
                     ⁢ 
                     
                       
                         AB 
                         c 
                       
                       × 
                       
                         h 
                         
                           
                             AB 
                             c 
                           
                           ⁡ 
                           
                             ( 
                             3 
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       BC 
                       cg 
                     
                     = 
                       
                     ⁢ 
                     
                       
                         BC 
                         c 
                       
                       × 
                       
                         h 
                         
                           
                             BC 
                             c 
                           
                           ⁡ 
                           
                             ( 
                             3 
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       DC 
                       cg 
                     
                     = 
                       
                     ⁢ 
                     
                       
                         DC 
                         c 
                       
                       × 
                       
                         h 
                         
                           
                             DC 
                             c 
                           
                           ⁡ 
                           
                             ( 
                             3 
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   
                     
                       AD 
                       cg 
                     
                     = 
                       
                     ⁢ 
                     
                       
                         AD 
                         c 
                       
                       × 
                       
                         h 
                         
                           
                             AD 
                             c 
                           
                           ⁡ 
                           
                             ( 
                             3 
                             ) 
                           
                         
                       
                     
                   
                 
               
             
           
         
       
     
     After computing the various coordinates of sensor  114 &#39;s sensor footprint, at step  310 , a track metric ρ is calculated by (1) calculating a series of instantaneous normalized track metrics (ρ TR ) over time, and (2) calculating a weighted moving average of the instantaneous normalized track metrics. 
     ρ TR  is calculated using (1) the target&#39;s position relative to the center of the camera footprint (r target ) on the ground, (2) the distance from the center of the camera footprint to the side (e.g., [AB cg  BC cg  DC cg  AD cg ] of the footprint that is closest to the target (r side ), and (3) the distance from the center of the frame to the target (r t ). These positions are illustrated in  FIG. 5 , which is a diagram of a footprint that illustrates variables needed to calculate ρ TR . As shown in  FIG. 5 , the frame includes a center point, a target, r target , r t , r side , and the direction of the frame of motion. 
     In order to calculate ρ TR , the value of r target  and r side  is first calculated. r target  is calculated by using the following equation:
 
 r   target =( ê   ct   ·ê   ce ) r   t  
 
     where ê ct  and ê ce  are unit vectors along a line from the target to the center of the footprint and from the mid-point of the closest side to the center respectively. That is, ê ct  is the unit vector along r t , while ê ce  is the unit vector along r side . 
     r side  is calculated using the following equation: 
     
       
         
           
             
               r 
               side 
             
             = 
             
               arg 
               ⁢ 
               
                   
               
               ⁢ 
               min 
               ⁢ 
               
                  
                 
                   
                     
                       
                         
                           r 
                           
                             AB 
                             cg 
                           
                         
                         - 
                         
                           r 
                           target 
                         
                       
                     
                   
                   
                     
                       
                         
                           r 
                           
                             BC 
                             cg 
                           
                         
                         - 
                         
                           r 
                           target 
                         
                       
                     
                   
                   
                     
                       
                         
                           r 
                           
                             DC 
                             cg 
                           
                         
                         - 
                         
                           r 
                           target 
                         
                       
                     
                   
                   
                     
                       
                         
                           r 
                           
                             AD 
                             cg 
                           
                         
                         - 
                         
                           r 
                           target 
                         
                       
                     
                   
                 
                  
               
             
           
         
       
     
     where r AB     cg   -r AB     cg    is the distance from the center of the frame to the side of the frame. 
     After calculating r target  and r side , ρ TR  is calculated as follows: 
     
       
         
           
             
               ρ 
               TR 
             
             = 
             
               
                 r 
                 target 
               
               
                 r 
                 side 
               
             
           
         
       
     
     If ρ TR =0, then the target  114  is directly over the center of the footprint. This is considered perfect tracking. If ρ TR ≧1, then AV  112  has lost track of target  114 . Because AV  112  (and possibly target  116 ) are moving, ρ TR  should be calculated at regular time intervals (i.e., once every 10 ms), although ρ TR  could be calculated at random time intervals as well. 
     After calculating ρ TR , the track metric ρ is calculated as the weighted moving average of ρ TR , with more weight being placed on recently calculated values of ρ TR . 
     
       
         
           
             ρ 
             = 
             
               
                 
                   
                     
                       [ 
                       
                         b 
                         k 
                       
                     
                   
                   
                     
                       b 
                       
                         k 
                         - 
                         1 
                       
                     
                   
                   
                     … 
                   
                   
                     
                       
                         b 
                         
                           k 
                           - 
                           n 
                         
                       
                       ] 
                     
                   
                 
               
               ⁡ 
               
                 [ 
                 
                   
                     
                       
                         
                           ρ 
                           TR 
                         
                         ⁡ 
                         
                           ( 
                           k 
                           ) 
                         
                       
                     
                   
                   
                     
                       
                         
                           ρ 
                           TR 
                         
                         ⁡ 
                         
                           ( 
                           
                             k 
                             - 
                             1 
                           
                           ) 
                         
                       
                     
                   
                   
                     
                       ⋮ 
                     
                   
                   
                     
                       
                         
                           ρ 
                           TR 
                         
                         ⁡ 
                         
                           ( 
                           
                             k 
                             - 
                             n 
                           
                           ) 
                         
                       
                     
                   
                 
                 ] 
               
             
           
         
       
     
     where b i  are the weights, and 
     
       
         
           
             
               
                 ∑ 
                 
                   i 
                   = 
                   0 
                 
                 n 
               
               ⁢ 
               
                 b 
                 
                   k 
                   - 
                   i 
                 
               
             
             = 
             1 
           
         
       
     
     where k is the sampling instant and n is the moving window over which the averaging is done. For example, if the sample time of the algorithm is t s  and the averaging is done over a time window of t seconds, then: 
     
       
         
           
             n 
             = 
             
               t 
               
                 t 
                 s 
               
             
           
         
       
     
     In addition to calculating ρ, the values of the first and second time derivatives of ρ ({dot over (ρ)},{umlaut over (ρ)}) may be calculated. 
     After obtaining ρ, at step  312 , remote operator  118  is alerted when the value of ρ, {dot over (ρ)}, or {umlaut over (ρ)} exceeds a threshold value (ρ threshold , {dot over (ρ)} threshold , or {umlaut over (ρ)} threshold ). The threshold values may be a static or dynamic number. For example, the threshold for ρ may be set to be slightly less than one, and remote operator  118  will be alerted if ρ exceeds the threshold. For a fixed-wing AV with a non-steerable track sensor and minimum turn radius R min , flying with airspeed V, the threshold values for {dot over (ρ)} may be determined as follows:
 
{dot over (ρ)} threshold =min(( ê   ct   ·ê   ce )× V,V   max )
 
     Thus, remote operator  118  will be alerted if the value of {dot over (ρ)} is greater than the lesser of AV  112 &#39;s maximum airspeed and its current airspeed resolved along the direction that the target is moving in the track sensor frame. 
     The value of {umlaut over (p)} may be determined as follows: 
     
       
         
           
             
               
                 ρ 
                 ¨ 
               
               threshold 
             
             = 
             
               
                 
                   ( 
                   
                     
                       
                         e 
                         ^ 
                       
                       ct 
                     
                     · 
                     
                       
                         e 
                         ^ 
                       
                       ce 
                     
                   
                   ) 
                 
                 
                   2 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   
                     R 
                     min 
                   
                 
               
               ⁢ 
               
                 V 
                 2 
               
             
           
         
       
     
     Therefore, the remote operator  118  will be alerted if {umlaut over (ρ)} is greater than the vehicle&#39;s acceleration along the direction of target movement in the frame. 
     Alerts sent to remote operator  118  in step  312  may be classified into categories. For example, the alert could indicate that AV  112  is about to lose track of target  116  and occurs when {umlaut over (p)} is slightly less than 1 (i.e., 0.9), and {dot over (ρ)} is positive. The alert could also indicate that target  116  is doing something significant, such as stopping suddenly, changing direction quickly, or engaging in aggressive or evasive motion maneuvers. Such an alert could be triggered if {dot over (ρ)} and {umlaut over (ρ)} both exceed their threshold values. 
     The present invention may be embodied in other specific forms without departing from its essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is therefore indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.