# Real Flight Application of a Monocular Image-Based Aircraft Collision Decision Method

Authors: Peter Bauer; Antal Hiba; Tamas Zsedrovits
Journal: International Journal of Aerospace Engineering
 (2019)

---

## Abstract

This paper considers real flight application of a monocular image-based aircraft collision decision algorithm developed in a previous paper. First, it briefly summarizes the theory based on the previous paper and extends the results with a detailed evaluation of possible special cases. Second, it introduces the UAVs and flight test scenarios together with the camera system and the steps of image processing used in flight testing. A brief analysis about intruder detectability is also provided referencing a more exhaustive work of the authors. The main contribution is the detailed comparison of the image-based estimated collision parameters to the flight trajectory-based ones together with threshold selection for collision decision. Different threshold combinations are evaluated offline, and finally, real flight decision results based on one of the threshold combinations are also discussed. The paper ends with the setup of future research directions.


---

## Body

## 1. Introduction

Sense and avoid (S&A) capability is a crucial ability for future unmanned aerial vehicles (UAVs). It is vital to integrate civilian and governmental UAVs into the common airspace according to [1] for example. At the highest level of integration, airborne sense and avoid (ABSAA) systems are required to guarantee airspace safety [2].In this field, the most critical question is the case of noncooperative S&A for which usually complicated multisensor systems (see [3–6] for example) or radar-based solutions (see [7, 8] for example) are developed. However, in the case of small UAVs, the size, weight, and power consumption of the onboard S&A system should be minimal. Monocular vision-based solutions can be cost and weight effective and therefore especially good for small UAVs [9–16]. These systems measure the position (bearing) and size of the intruder aircraft (A/C) camera image without range and physical intruder size information. This scale ambiguity makes the decision about the possibility of collision (collision means both mid-air collision or near mid-air collision in this article) complicated (see, e.g., [9] where a conservative assumption on intruder’s minimum size is used to make the decision possible but this can lead to false alarms). In another source [13] where the task is to follow a leader aircraft based on a monocular camera, special maneuvers are required to provide range observability. However, [10] points out that the relative distance of the intruder from the own A/C called the closest point of approach (CPA) well characterizes the possibility of collision together with the time to closest point of approach (TTCPA) and there is a chance to estimate these parameters without any physical intruder information. In the literature, there are several works dealing with TTCPA estimation based on optic flow or first-order motion models such as [17–19]. However, it is hard to find references about the estimation of CPA. For example, [9, 20] target to estimate the absolute distance between the A/C the former with the conservative intruder size assumption and the latter by assuming known intruder position and relative velocity. Consequently, previous efforts of the authors [21–24] focused on the estimation of TTCPA and CPA values together with the direction of CPA from solely the monocular image parameters. The only restrictive assumption was that the aircraft follows straight flight paths with constant velocities (in these previous works, TTCPA was sometimes referenced as TTC (time to collision)). Formulae to obtain these values with forward looking and even with oblique cameras (see our system in Figure 10) were developed and tested mainly in software in-the-loop (SIL) simulations. The proposed simple solution is based on a least squares (LS) optimal line fit method which well serves real-time execution needs. Error analysis considering image pixelization errors was done in [21].Our latest work [24] presents the theory of evaluation of the collision situation in three dimensions (3D) estimating TTCPA, horizontal and vertical CPA, and the direction of the horizontal one (βCPA) as it is not necessarily perpendicular to the own A/C forward body axis (see [23] and Figure 2). The only information lost is the absolute distance between the two A/C; the relative CPA values are determined instead, but this information is appropriate for the decision about the possibility of collision and for the design of an avoidance maneuver. SIL test results are presented, but in-flight testing of such methods is also very important as underlined by [4, 13] for example.So, the main goal of the current article is to test the developed method on real flight data. That is why after a summary of the theory published before (to make this article self-contained) and the additional evaluation of special cases, it introduces our real flight test system (A/C and camera system) and applies the derived 3D collision characterization on real flight image data. Knowing the sizes and real flight GPS trajectories of the A/C, the obtained camera-based results (TTCPA, CPA, andβCPA) are compared to the real data. Finally, suggestions are done for decision threshold selection and real flight decision results based on one possible combination of the thresholds are evaluated. The structure of the paper is as follows. Section 2 briefly summarizes the 3D collision characterization from our previous article [24] by possibly partly reproducing the wording used there. Then it includes detailed evaluation of the possible special cases. Section 3 introduces our test UAVs and the test scenarios. Section 4 presents our camera system together with the image processing issues. Section 5 summarizes the results of collision situation evaluation in real flight and presents some results about intruder detectability. Finally, Section 6 concludes the paper.


## 2. Theory of 3D Collision Possibility Decision

This section includes a summary of the results obtained in [24] estimating first the horizontal closest point between the A/C and the related vertical distance (P1) and second the vertical closest point and the related horizontal distance (P2). These points are shown in Figure 1 together with the own aircraft body X,Y,Z coordinate system. The dashed lines symbolize the aircraft flight paths. Note that the order of body coordinate system axes differs from aerospace convention to better match camera coordinate system conventions of the image processing community.Figure 1
Special points along the 3D A/C trajectories. The dashed line betweenP1 and P2 is the straight trajectory of the intruder. Note that βCPA/x is in the X-Z horizontal plane.Considering the interpolation betweenP1 and P2, the real 3D minimum CPA can be obtained by applying the normalized equivalent of the formula presented in [20] equation (E4).The underlying assumption of all of the derivations is that the A/C fly on straight paths with constant velocities. The collision situation is evaluated relative to theX,Y,Z body coordinate system of the own A/C. The XC,YC,ZC separate camera coordinate system will be used to characterize camera orientation relative to the body if it is not aligned with the body system (see, e.g., Figure 3).The horizontal situation in theZ-X plane of the own body coordinate system is shown in Figure 2 denoting also P1, the horizontal closest point. To evaluate this situation (obtain P1), one has to determine tCPA/x, CPAx/x=Xa/R, and βCPA/x. Here, subscript x denotes that it is a horizontal parameter, while the second subscript /x denotes that the parameter is determined by looking for the horizontal minimum distance. Xa is the absolute minimum horizontal distance while R is the characteristic horizontal intruder size.Figure 2
DefinetCPA/x (TTCPA), CPAx/x=Xa/R, and βCPA/x(intruder: red on the left, own A/C: blue on the right, and P1: the horizontal closest position of the intruder).In the horizontal plane, the A/C can be modeled as a disc in the same way as in [9]. Considering that the camera of the S&A system can be oblique relative to the body coordinate system (see our system in Figure 10), the disc projection model of the sense and avoid situation is shown in Figure 3. For a detailed derivation of the disc-based camera projection model for an oblique camera, see [22].Figure 3
Oblique camera disc projection model.Finally, a system of equations (1) results which includes the modified image parameters S¯x=Sxcosβ1+cosβ2 and x¯=x1−S¯x2/16f2. Where x is the horizontal position and Sx is the horizontal size of the intruder image, f is the camera focal length, βC is the camera angle, and β1 and β2 are the view angles of the edges of the intruder image as shown in Figure 3. All x,Sx,β1,β2 are measurable parameters in the image and f, βC are known. The other parameters tCPA/x, CPAx/x, and βCPA/x and the constant intruder relative velocities Vx,Vz are unknown (note that βCPA/x is shown in Figure 2 and its estimation will be discussed later).
(1)1S¯x=CPAx/x2sinβCf−VxsinβC+VzcosβC2fRtCPA/x,x¯S¯x=CPAx/x2cosβC−VxcosβC−VzsinβC2RtCPA/x.In the vertical situation, the unknown parameters to obtainP2 decrease to tCPA/y, CPAy/y as the relative angle will be surely 90° up or down. In the vertical plane, the A/C cannot be modeled as a disc since it is rather similar to an ellipse (length almost equal to the horizontal size of the intruder and height equal to the height of the intruder). However, the pinhole camera projection model of the ellipse is very complicated and later interpolation for the 3D minimum point requires normalization with the same size in the horizontal and vertical CPA. Thus, an extension to the same disc size R=2r as in the horizontal plane is done considering also the distortion effect dependent on the α angles. This is shown in Figure 4. For details, see [24]. Finally, the extended vertical size in the image plane can be denoted as Sy′.Figure 4
Vertical camera ellipse and disc projection models.This leads to a similar system of equations as in the horizontal case (see (1)) with modified image parameters S¯y′=Sy′cosα10+cosα20 and y¯=y1−S¯′y2/16f2.
(2)1S¯y′=CPAx2sinβCf−VxsinβC+VzcosβC2fRtCPA,y¯S¯y′=CPAy′2−Vy2RtCPA.The first equation is exactly the same as that in the horizontal plane so the mean value of the two equations can be considered. Summarizing the results up to now leads to three image parameter-based equations from which the first two characterize the horizontal situation (P1) while the last characterizes the vertical situation (P2):
(3)121S¯x+1S¯y′=CPAx/x2sinβCf−VxsinβC+VzcosβC2fRtCPA/x,x¯S¯x=CPAx/x2cosβC−VxcosβC−VzsinβC2RtCPA/x,y¯S¯y′=CPAy/y′2−Vy2RtCPA/y.In this system of equations, the unknowns areCPAx/x, CPAy/y′, tCPA/x, and tCPA/y. The terms such as the camera angle βC and focal length f, the relative velocities Vx,Vy,Vz, and the intruder size R are all constant. Considering this and tCPA/x=tC/x−t,tCPA/y=tC/y−t, one gets another form of the equations in (4) and (5). Note that t is the actual varying absolute time while tC/x and tC/y are the constant absolute times when the intruder is closest to the own A/C in the horizontal and vertical planes, respectively.
(4)121S¯x+1S¯y′=sinβCfCPAx/x2−a1tC/x+a1t=c1+a1t,x¯S¯x=cosβCCPAx/x2−a2tC/x+a2t=c2+a2t,(5)y¯S¯y′=CPAy/y′2−a3tC/y+a3t=c3+a3t.The horizontal minimumCPAx/x and the related tC/x can be determined from (4). Fitting LS optimal linear curves to the expressions on the left side with independent variable t gives a1,a2,c1,c2, and this leads to a system of linear equations for CPAx/x/2 and tC/x:
(6)sinβCf−a1cosβC−a2CPAx/x2tC/x=c1c2.This system gives a solution forCPAx/x and tC/x considering the crossing of the body-fixed x-axis by the intruder as the closest point. However, as Figure 2 shows, this is not necessarily the closest point. Perturbing the camera angle to get βC/x′=βC+ΔβC/x (this means −∆βC/x virtual rotation of the body system), one can derive the optimal value of ∆βC/x which gives real minimum CPAx/x and the related tC/x and βCPA/x in the horizontal plane.
(7)sinβC+ΔβC/xf−a1cosβC+ΔβC/x−a2CPAx/x2tC/x=c1c2,minΔβC/xCPAx/x⇒tanΔβC/x=−a2cosβC/f−a1sinβC−a2sinβC/f+a1cosβC,βCPA/x=−ΔβC/x+signCPAx/xπ2.Turning back to the vertical situation from (5), that equation characterizes CPA in the vertical plane. In the case where Vy=0, a3=0 and CPAy/y′≠0 can be determined simply from the time average of y¯/S¯y′ values. For this case, the A/C flight paths are parallel in the vertical plane. In the other case, Vy≠0 and a3≠0 mean that CPAy/y′=0 will be the closest vertical point as the trajectories cross each other in the vertical plane. For this case, c3=CPAy/y′/2−a3tC/y=−a3tC/y. Doing a line fit to the y¯/S¯y′,t data pairs will give c3 and a3 from which tC/y can be determined: tC/y=−c3/a3.It can be proven that ifVy=0 and CPAy/y′≠0, the closest horizontal point P1 will also be the closest 3D point (for details, see [24]). Its coordinates result as follows (where v stands for the vertical parameter and h for the horizontal parameter):
(8)tCMIN=tC/x,tCPAMIN=tCMIN−t,CPAvMIN′=CPAy/y′,CPAhMIN=CPAx/x,βCPAMIN=βCPA/x.In the other case, ifVy≠0, one should determine the vertical position of the intruder (CPAy/x′) in point P1 while the horizontal position is at tC/x, CPAx/x, and βCPA/x and also determines the horizontal position of P2 while the vertical position is at tC/y and CPAy/y′=0.InP1, CPAy/x′ can be determined from (5) by substituting tC/x into c3 obtained from LS optimal line fit. In P2, CPAx/y and the related ∆βC/y (and so βCPA/y) values should be determined. This can be done considering the system of equations for the averaged image size and horizontal data from (4) and (7) (βC/y′=βC+∆βC/y).
(9)sinβC/y′f−a1cosβC/y′−a2CPAx/y2tC/y=c1c2.AstC/y is known, the system can be reformulated having the known values on the right-hand side:
(10)sinβC+ΔβC/yCPAx/y2f=a1tC/y+c1=b1,cosβC+ΔβC/yCPAx/y2=a2tC/y+c2=b2,where the unknowns are CPAx/y and ΔβC/y. Depending on the values of b1 and b2, there can be different solutions. For details, see [24].Summarizing the developments, one has two points,P1 with parameters CPAx/x,βCPA/x,CPAy/x′,tC/x and P2 with CPAx/y,βCPA/y,CPAy/y,tC/y as shown in Figure 1.The coordinates of pointsP1 and P2 in the body coordinate system (X,Y,Z in Figure 1) so result as follows:
(11)P1Z1=cosβCPA/xCPAx/x,X1=sinβCPA/xCPAx/x,Y1=CPAy/x′,(12)P2Z2=cosβCPA/yCPAx/y,X2=sinβCPA/yCPAx/y,Y2=0.An interpolated point betweenP1 and P2 can be represented as follows. Note that the given representation with the tM parameter makes extrapolation before P1 or after P2 also possible.
(13)ZM=Z1+tMZ2−Z1tC/y−tC/x=Z1+tMΔZ,XM=X1+tMX2−X1tC/y−tC/x=X1+tMΔX,YM=Y1+tMY2−Y1tC/y−tC/x=Y1+tMΔY.Minimizing the 3D distanceD=ZM2+XM2+YM2 gives tM=−Z1ΔZ+X1ΔX+Y1ΔY/ΔZ2+ΔX2+ΔY2 which is exactly the same as the tCPA value calculated in [20] (E4) from absolute trajectory parameters, but here, this is an extension of the formula relative to CPA distances. Considering this result, the time of minimum 3D distance and the related relative CPA values in case the A/C trajectories cross the vertical plane can be determined as follows:
(14)tCMIN=tC/x+tM,tCPAMIN=tCMIN−t,CPAvMIN′=YM,CPAhMIN=XM2+ZM2,tanβCPAMIN=XMZM.Finally,CPAvMIN related to the H real vertical size (Sy real vertical image size) of the intruder should be calculated from CPAvMIN′ (calculated from the extended Sy′ vertical image size) considering the ratio k between horizontal and vertical sizes and taking its time average k¯:
(15)k=HR≈SySx,CPAvMIN=YaH=1k¯CPAvMIN′.The approximation in (15) shows that there can be different distortions of Sy and Sx. A method to correct this is proposed in [24] and so k should be calculated based on the corrected values.


### 2.1. Possible Special Cases

After developing the formulae, they should be examined for special cases to avoid any singularity in the calculations. Sources such as [25, 26] point out that a critical special case in A/C collision avoidance is the case of collision with zero miss distance where the intruder A/C will be steady in the pilot’s view. For this case, it will be also steady in the camera system’s view and so this can be a problematic case in image-based S&A also. Considering an oblique camera setup, three special cases can occur in the horizontal plane:Case 1.
Xa=0, Vx=0, and βC≠0 which means a head-on scenario with zero miss distance and an oblique observer camera.Case 2.
Xa=0, Vx≠0, and βC=0 which means a crossing scenario with zero miss distance and a forward-looking camera.Case 3.
Xa=0, Vx=0, and βC=0 which means a head-on scenario with zero miss distance and a forward-looking camera.These cases are examined in the sequel considering also the possible vertical situation and thatCPAx/x=0 in all cases because Xa=0.


#### 2.1.1. Case 1 (Vx=0,βC≠0)

In this case, the horizontal equations from (3) and (4) degrade to
(16)121S¯x+1S¯y′=−VzcosβC2fRtCPA/x=−a1tC/x+a1t=c1+a1t,x¯S¯x=VzsinβC2RtCPA/x=−a2tC/x+a2t=c2+a2t.This shows that the LS optimal line fits can be done flawlessly and soa1,a2,c1,c2 are obtained with a1=VzcosβC/2fR and a2=VzsinβC/2R theoretical values. Substituting these into the expression of ∆βC/x in (7) gives
(17)tanΔβC/x=VzsinβCcosβC/2fR−VzsinβCcosβC/2fRVzsin2βC/2fR/2fR+Vzcos2βC/2fR=0.This gives∆βC/x=0 and the following form of the equation system to be solved from (7) (considering also c1=−a1tC/x and c2=−a2tC/x theoretical values):
(18)sinβCf−a1cosβC−a2U=−a1tC/x−a2tC/x.It is easy to derive the solution as(19)U=0tC/x,which gives CPAx/x=0 and tC/x. Considering βCPA/x from (7), one gets βCPA/x=0 as −∆βC/x=0 and signCPAx/x=0=0.Considering now the possible vertical situations ifVy=0, it is either CPAy/y′=const≠0 or CPAy/y′=0 if the A/C fly in the same horizontal plane y=0→y¯=0. The calculation of both of them is straightforward from the mean value of y¯/S¯y′ and will be the same in the further two cases also. The only vertical scenario left is the crossing of the own horizontal plane by the intruder with CPAy/y′=0 and Vy≠0. For this case, after determining tC/y, CPAx/y should be determined from (10) considering that b1=a1tC/y−tC/x, b2=a2tC/y−tC/x. The relation of tC/y and tC/x gives three possible cases shown in Figure 5.Figure 5
Three crossing possibilities of the own horizontal plane (red cross: intruder; blue circle: own A/C).The vertical crossing can be earlier than the horizontal one (in front of the own A/C) (tC/y<tC/x; no. 1 in Figure 5), and it can be at the same time (tC/y=tC/x; no. 2 in Figure 5) or later (behind the own A/C) (tC/y>tC/x; no. 3 in Figure 5). In the second case, b1=0, b2=0, and CPAx/y=0 according to [24]. In the other cases, b1≠0 and so the formula from [24] can be used to determine ∆βC/y
(20)tanΔβC/y=cosβC−b2/b1fsinβCsinβC+b2/b1fcosβC.Substituting the theoretical values ofa1, a2 into b1, b2 and the latter into (20) equation gives
(21)tanΔβC/y=cosβC+tanβCsinβCsinβC−tanβCcosβC=∞.This gives∆βC/y=π/2 and CPAx/y can be calculated from this. Finally, βCPA/y=−π/2+signCPAx/yπ/2 which is either 0 or −π radian giving the expected in-front or behind crossing.Finally, the calculation of the 3D minimum distance with interpolation should be examined for the special case. The final parameters areβCPA/x=0, CPAx/x=0, CPAy/x′≠0, βCPA/y=0 or −π, and CPAx/y≠0 (CPAy/x′ can be determined from (5) knowing c3 and tC/x). This gives the coordinates of points P1 and P2 as Z1=0,X1=0,Y1=CPAy/x′ and Z2=±CPAx/y, X2=0, Y2=0 from (11) and (12), respectively. Considering the interpolation, the only singular case is tC/y=tC/x where the horizontal and vertical crossing coincides and this gives immediately the 3D minimum. Otherwise, there is no problem in calculating the minimizing tM parameter with ∆Z≠0, ∆X=0, and ∆Y≠0.


#### 2.1.2. Case 2 (Vx≠0,βC=0)

In this case, the LS optimal line fits can also be done flawlessly and soa1,a2,c1,c2 are obtained with a1=Vz/2fR and a2=Vx/2R theoretical values. Substituting these into the expression of ∆βC/x in (7) gives
(22)tanΔβC/x=−Vx/2fRVz/2fR=−VxVz≠0.This is the calculation formula that resulted from a forward-looking camera in [23] so it is natural to get this result. Now, the equation system to be solved from (7) results as (considering also βC=0, c1=−a1tC/x,c2=−a2tC/x)
(23)sinΔβC/xf−a1cosΔβC/x−a2U=−a1tC/x−a2tC/x,giving again
(24)U=0tC/x,which gives CPAx/x=0 and tC/x. Considering βCPA/x from (7), one gets βCPA/x=−∆βC/x=0 as signCPAx/x=0=0. This is only a theoretical value because CPAx/x=0 and has no definite direction.Considering the only special vertical case withCPAy/y′=0 and Vy≠0, no. 1 and no. 3 cases with b1≠0 give the following:
(25)tanΔβC/y=VzVx.From this,∆βC/y=−π/2−βv∞ and ∆βCPA/y=π/2+βv∞+signCPAx/yπ/2. βv∞=const is the view angle of the intruder from the own aircraft if Xa=0 and can be obtained from tanβv∞=−Vx/−Vz (for details, see [23]). So finally, the crossing of horizontal plane will be along the line characterized by βv∞ in this direction or in the opposite direction.Regarding the calculation of the 3D minimum distance with interpolation, the final parameters here result asβCPA/x≠0, CPAx/x=0, CPAy/x′≠0, βCPA/y=βv∞ or βv∞+π, and CPAx/y≠0. This gives the coordinates of points P1 and P2 as Z1=0, X1=0, Y1=CPAy/x′ and Z2≠0, X2≠0, Y2=0 from (11) and (12), respectively. Considering the interpolation, the only singular case is tC/y=tC/x where the horizontal and vertical crossing coincides and this gives immediately the 3D minimum. Otherwise, there is no problem in calculating the minimizing tM parameter with ∆Z≠0, ∆X≠0, and ∆Y≠0.


#### 2.1.3. Case 3 (Vx=0,βC=0)

In this case, (16) further simplifies and the LS optimal line fits will give a1,c1,a2=0, c2=0 by fitting the second line on a constant zero parameter.This leads to(26)tanΔβC/x=0a1=0,which gives ∆βC/x=0 and so βCPA/x=0. The first equation in (7) simplifies to
(27)0−a110U=−a1tC/x0,giving again
(28)U=0tC/x,which gives CPAx/x=0 and tC/x. Considering the only special vertical case with CPAy/y′=0 and Vy≠0, no. 1 and no. 3 cases with b1≠0, b2=0 give the following:
(29)tanΔβC/y=10=∞.This gives∆βC/y=π/2 as in Case 1 and βCPA/y=−π/2+signCPAx/yπ/2 which is either 0 or −πradian giving in-front (of the own A/C) or behind crossing.Regarding the calculation of the 3D minimum distance with interpolation, the final parameters here have the same result as in Case1, βCPA/x=0, CPAx/x=0, CPAy/x′≠0, βCPA/y=0 or π, and CPAx/y≠0. From this point, the same deduction can be done.


#### 2.1.4. Summary about Special Cases

All of these derivations show that original formulae (4), (5), (7), (10), and (13) can be well applied in these special cases without any modification. This is underlined by the Monte-Carlo test run in [24] where Case 2 and Case 3 scenarios are included both with special tC/y=tC/x and crossing in-front and behind vertical setups and there were no singularity or any problem with the formulae. Of course, for real data, the theoretical values will be corrupted by noise but considering multiple measurements will smoothen this out in some extent.After developing a method to determine the 3D minimum distance (relative to intruder size) between two A/C in an encounter scenario, it should be tested in detail. Reference [24] presented promising results of a SIL Monte-Carlo simulation campaign. The current article presents results based on real flight data. This will be done in Section 5 after introducing the test A/C and scenarios and the used camera system and image processing methods.


## 2.1. Possible Special Cases

After developing the formulae, they should be examined for special cases to avoid any singularity in the calculations. Sources such as [25, 26] point out that a critical special case in A/C collision avoidance is the case of collision with zero miss distance where the intruder A/C will be steady in the pilot’s view. For this case, it will be also steady in the camera system’s view and so this can be a problematic case in image-based S&A also. Considering an oblique camera setup, three special cases can occur in the horizontal plane:Case 1.
Xa=0, Vx=0, and βC≠0 which means a head-on scenario with zero miss distance and an oblique observer camera.Case 2.
Xa=0, Vx≠0, and βC=0 which means a crossing scenario with zero miss distance and a forward-looking camera.Case 3.
Xa=0, Vx=0, and βC=0 which means a head-on scenario with zero miss distance and a forward-looking camera.These cases are examined in the sequel considering also the possible vertical situation and thatCPAx/x=0 in all cases because Xa=0.


### 2.1.1. Case 1 (Vx=0,βC≠0)

In this case, the horizontal equations from (3) and (4) degrade to
(16)121S¯x+1S¯y′=−VzcosβC2fRtCPA/x=−a1tC/x+a1t=c1+a1t,x¯S¯x=VzsinβC2RtCPA/x=−a2tC/x+a2t=c2+a2t.This shows that the LS optimal line fits can be done flawlessly and soa1,a2,c1,c2 are obtained with a1=VzcosβC/2fR and a2=VzsinβC/2R theoretical values. Substituting these into the expression of ∆βC/x in (7) gives
(17)tanΔβC/x=VzsinβCcosβC/2fR−VzsinβCcosβC/2fRVzsin2βC/2fR/2fR+Vzcos2βC/2fR=0.This gives∆βC/x=0 and the following form of the equation system to be solved from (7) (considering also c1=−a1tC/x and c2=−a2tC/x theoretical values):
(18)sinβCf−a1cosβC−a2U=−a1tC/x−a2tC/x.It is easy to derive the solution as(19)U=0tC/x,which gives CPAx/x=0 and tC/x. Considering βCPA/x from (7), one gets βCPA/x=0 as −∆βC/x=0 and signCPAx/x=0=0.Considering now the possible vertical situations ifVy=0, it is either CPAy/y′=const≠0 or CPAy/y′=0 if the A/C fly in the same horizontal plane y=0→y¯=0. The calculation of both of them is straightforward from the mean value of y¯/S¯y′ and will be the same in the further two cases also. The only vertical scenario left is the crossing of the own horizontal plane by the intruder with CPAy/y′=0 and Vy≠0. For this case, after determining tC/y, CPAx/y should be determined from (10) considering that b1=a1tC/y−tC/x, b2=a2tC/y−tC/x. The relation of tC/y and tC/x gives three possible cases shown in Figure 5.Figure 5
Three crossing possibilities of the own horizontal plane (red cross: intruder; blue circle: own A/C).The vertical crossing can be earlier than the horizontal one (in front of the own A/C) (tC/y<tC/x; no. 1 in Figure 5), and it can be at the same time (tC/y=tC/x; no. 2 in Figure 5) or later (behind the own A/C) (tC/y>tC/x; no. 3 in Figure 5). In the second case, b1=0, b2=0, and CPAx/y=0 according to [24]. In the other cases, b1≠0 and so the formula from [24] can be used to determine ∆βC/y
(20)tanΔβC/y=cosβC−b2/b1fsinβCsinβC+b2/b1fcosβC.Substituting the theoretical values ofa1, a2 into b1, b2 and the latter into (20) equation gives
(21)tanΔβC/y=cosβC+tanβCsinβCsinβC−tanβCcosβC=∞.This gives∆βC/y=π/2 and CPAx/y can be calculated from this. Finally, βCPA/y=−π/2+signCPAx/yπ/2 which is either 0 or −π radian giving the expected in-front or behind crossing.Finally, the calculation of the 3D minimum distance with interpolation should be examined for the special case. The final parameters areβCPA/x=0, CPAx/x=0, CPAy/x′≠0, βCPA/y=0 or −π, and CPAx/y≠0 (CPAy/x′ can be determined from (5) knowing c3 and tC/x). This gives the coordinates of points P1 and P2 as Z1=0,X1=0,Y1=CPAy/x′ and Z2=±CPAx/y, X2=0, Y2=0 from (11) and (12), respectively. Considering the interpolation, the only singular case is tC/y=tC/x where the horizontal and vertical crossing coincides and this gives immediately the 3D minimum. Otherwise, there is no problem in calculating the minimizing tM parameter with ∆Z≠0, ∆X=0, and ∆Y≠0.


### 2.1.2. Case 2 (Vx≠0,βC=0)

In this case, the LS optimal line fits can also be done flawlessly and soa1,a2,c1,c2 are obtained with a1=Vz/2fR and a2=Vx/2R theoretical values. Substituting these into the expression of ∆βC/x in (7) gives
(22)tanΔβC/x=−Vx/2fRVz/2fR=−VxVz≠0.This is the calculation formula that resulted from a forward-looking camera in [23] so it is natural to get this result. Now, the equation system to be solved from (7) results as (considering also βC=0, c1=−a1tC/x,c2=−a2tC/x)
(23)sinΔβC/xf−a1cosΔβC/x−a2U=−a1tC/x−a2tC/x,giving again
(24)U=0tC/x,which gives CPAx/x=0 and tC/x. Considering βCPA/x from (7), one gets βCPA/x=−∆βC/x=0 as signCPAx/x=0=0. This is only a theoretical value because CPAx/x=0 and has no definite direction.Considering the only special vertical case withCPAy/y′=0 and Vy≠0, no. 1 and no. 3 cases with b1≠0 give the following:
(25)tanΔβC/y=VzVx.From this,∆βC/y=−π/2−βv∞ and ∆βCPA/y=π/2+βv∞+signCPAx/yπ/2. βv∞=const is the view angle of the intruder from the own aircraft if Xa=0 and can be obtained from tanβv∞=−Vx/−Vz (for details, see [23]). So finally, the crossing of horizontal plane will be along the line characterized by βv∞ in this direction or in the opposite direction.Regarding the calculation of the 3D minimum distance with interpolation, the final parameters here result asβCPA/x≠0, CPAx/x=0, CPAy/x′≠0, βCPA/y=βv∞ or βv∞+π, and CPAx/y≠0. This gives the coordinates of points P1 and P2 as Z1=0, X1=0, Y1=CPAy/x′ and Z2≠0, X2≠0, Y2=0 from (11) and (12), respectively. Considering the interpolation, the only singular case is tC/y=tC/x where the horizontal and vertical crossing coincides and this gives immediately the 3D minimum. Otherwise, there is no problem in calculating the minimizing tM parameter with ∆Z≠0, ∆X≠0, and ∆Y≠0.


### 2.1.3. Case 3 (Vx=0,βC=0)

In this case, (16) further simplifies and the LS optimal line fits will give a1,c1,a2=0, c2=0 by fitting the second line on a constant zero parameter.This leads to(26)tanΔβC/x=0a1=0,which gives ∆βC/x=0 and so βCPA/x=0. The first equation in (7) simplifies to
(27)0−a110U=−a1tC/x0,giving again
(28)U=0tC/x,which gives CPAx/x=0 and tC/x. Considering the only special vertical case with CPAy/y′=0 and Vy≠0, no. 1 and no. 3 cases with b1≠0, b2=0 give the following:
(29)tanΔβC/y=10=∞.This gives∆βC/y=π/2 as in Case 1 and βCPA/y=−π/2+signCPAx/yπ/2 which is either 0 or −πradian giving in-front (of the own A/C) or behind crossing.Regarding the calculation of the 3D minimum distance with interpolation, the final parameters here have the same result as in Case1, βCPA/x=0, CPAx/x=0, CPAy/x′≠0, βCPA/y=0 or π, and CPAx/y≠0. From this point, the same deduction can be done.


### 2.1.4. Summary about Special Cases

All of these derivations show that original formulae (4), (5), (7), (10), and (13) can be well applied in these special cases without any modification. This is underlined by the Monte-Carlo test run in [24] where Case 2 and Case 3 scenarios are included both with special tC/y=tC/x and crossing in-front and behind vertical setups and there were no singularity or any problem with the formulae. Of course, for real data, the theoretical values will be corrupted by noise but considering multiple measurements will smoothen this out in some extent.After developing a method to determine the 3D minimum distance (relative to intruder size) between two A/C in an encounter scenario, it should be tested in detail. Reference [24] presented promising results of a SIL Monte-Carlo simulation campaign. The current article presents results based on real flight data. This will be done in Section 5 after introducing the test A/C and scenarios and the used camera system and image processing methods.


## 2.1.1. Case 1 (Vx=0,βC≠0)

In this case, the horizontal equations from (3) and (4) degrade to
(16)121S¯x+1S¯y′=−VzcosβC2fRtCPA/x=−a1tC/x+a1t=c1+a1t,x¯S¯x=VzsinβC2RtCPA/x=−a2tC/x+a2t=c2+a2t.This shows that the LS optimal line fits can be done flawlessly and soa1,a2,c1,c2 are obtained with a1=VzcosβC/2fR and a2=VzsinβC/2R theoretical values. Substituting these into the expression of ∆βC/x in (7) gives
(17)tanΔβC/x=VzsinβCcosβC/2fR−VzsinβCcosβC/2fRVzsin2βC/2fR/2fR+Vzcos2βC/2fR=0.This gives∆βC/x=0 and the following form of the equation system to be solved from (7) (considering also c1=−a1tC/x and c2=−a2tC/x theoretical values):
(18)sinβCf−a1cosβC−a2U=−a1tC/x−a2tC/x.It is easy to derive the solution as(19)U=0tC/x,which gives CPAx/x=0 and tC/x. Considering βCPA/x from (7), one gets βCPA/x=0 as −∆βC/x=0 and signCPAx/x=0=0.Considering now the possible vertical situations ifVy=0, it is either CPAy/y′=const≠0 or CPAy/y′=0 if the A/C fly in the same horizontal plane y=0→y¯=0. The calculation of both of them is straightforward from the mean value of y¯/S¯y′ and will be the same in the further two cases also. The only vertical scenario left is the crossing of the own horizontal plane by the intruder with CPAy/y′=0 and Vy≠0. For this case, after determining tC/y, CPAx/y should be determined from (10) considering that b1=a1tC/y−tC/x, b2=a2tC/y−tC/x. The relation of tC/y and tC/x gives three possible cases shown in Figure 5.Figure 5
Three crossing possibilities of the own horizontal plane (red cross: intruder; blue circle: own A/C).The vertical crossing can be earlier than the horizontal one (in front of the own A/C) (tC/y<tC/x; no. 1 in Figure 5), and it can be at the same time (tC/y=tC/x; no. 2 in Figure 5) or later (behind the own A/C) (tC/y>tC/x; no. 3 in Figure 5). In the second case, b1=0, b2=0, and CPAx/y=0 according to [24]. In the other cases, b1≠0 and so the formula from [24] can be used to determine ∆βC/y
(20)tanΔβC/y=cosβC−b2/b1fsinβCsinβC+b2/b1fcosβC.Substituting the theoretical values ofa1, a2 into b1, b2 and the latter into (20) equation gives
(21)tanΔβC/y=cosβC+tanβCsinβCsinβC−tanβCcosβC=∞.This gives∆βC/y=π/2 and CPAx/y can be calculated from this. Finally, βCPA/y=−π/2+signCPAx/yπ/2 which is either 0 or −π radian giving the expected in-front or behind crossing.Finally, the calculation of the 3D minimum distance with interpolation should be examined for the special case. The final parameters areβCPA/x=0, CPAx/x=0, CPAy/x′≠0, βCPA/y=0 or −π, and CPAx/y≠0 (CPAy/x′ can be determined from (5) knowing c3 and tC/x). This gives the coordinates of points P1 and P2 as Z1=0,X1=0,Y1=CPAy/x′ and Z2=±CPAx/y, X2=0, Y2=0 from (11) and (12), respectively. Considering the interpolation, the only singular case is tC/y=tC/x where the horizontal and vertical crossing coincides and this gives immediately the 3D minimum. Otherwise, there is no problem in calculating the minimizing tM parameter with ∆Z≠0, ∆X=0, and ∆Y≠0.


## 2.1.2. Case 2 (Vx≠0,βC=0)

In this case, the LS optimal line fits can also be done flawlessly and soa1,a2,c1,c2 are obtained with a1=Vz/2fR and a2=Vx/2R theoretical values. Substituting these into the expression of ∆βC/x in (7) gives
(22)tanΔβC/x=−Vx/2fRVz/2fR=−VxVz≠0.This is the calculation formula that resulted from a forward-looking camera in [23] so it is natural to get this result. Now, the equation system to be solved from (7) results as (considering also βC=0, c1=−a1tC/x,c2=−a2tC/x)
(23)sinΔβC/xf−a1cosΔβC/x−a2U=−a1tC/x−a2tC/x,giving again
(24)U=0tC/x,which gives CPAx/x=0 and tC/x. Considering βCPA/x from (7), one gets βCPA/x=−∆βC/x=0 as signCPAx/x=0=0. This is only a theoretical value because CPAx/x=0 and has no definite direction.Considering the only special vertical case withCPAy/y′=0 and Vy≠0, no. 1 and no. 3 cases with b1≠0 give the following:
(25)tanΔβC/y=VzVx.From this,∆βC/y=−π/2−βv∞ and ∆βCPA/y=π/2+βv∞+signCPAx/yπ/2. βv∞=const is the view angle of the intruder from the own aircraft if Xa=0 and can be obtained from tanβv∞=−Vx/−Vz (for details, see [23]). So finally, the crossing of horizontal plane will be along the line characterized by βv∞ in this direction or in the opposite direction.Regarding the calculation of the 3D minimum distance with interpolation, the final parameters here result asβCPA/x≠0, CPAx/x=0, CPAy/x′≠0, βCPA/y=βv∞ or βv∞+π, and CPAx/y≠0. This gives the coordinates of points P1 and P2 as Z1=0, X1=0, Y1=CPAy/x′ and Z2≠0, X2≠0, Y2=0 from (11) and (12), respectively. Considering the interpolation, the only singular case is tC/y=tC/x where the horizontal and vertical crossing coincides and this gives immediately the 3D minimum. Otherwise, there is no problem in calculating the minimizing tM parameter with ∆Z≠0, ∆X≠0, and ∆Y≠0.


## 2.1.3. Case 3 (Vx=0,βC=0)

In this case, (16) further simplifies and the LS optimal line fits will give a1,c1,a2=0, c2=0 by fitting the second line on a constant zero parameter.This leads to(26)tanΔβC/x=0a1=0,which gives ∆βC/x=0 and so βCPA/x=0. The first equation in (7) simplifies to
(27)0−a110U=−a1tC/x0,giving again
(28)U=0tC/x,which gives CPAx/x=0 and tC/x. Considering the only special vertical case with CPAy/y′=0 and Vy≠0, no. 1 and no. 3 cases with b1≠0, b2=0 give the following:
(29)tanΔβC/y=10=∞.This gives∆βC/y=π/2 as in Case 1 and βCPA/y=−π/2+signCPAx/yπ/2 which is either 0 or −πradian giving in-front (of the own A/C) or behind crossing.Regarding the calculation of the 3D minimum distance with interpolation, the final parameters here have the same result as in Case1, βCPA/x=0, CPAx/x=0, CPAy/x′≠0, βCPA/y=0 or π, and CPAx/y≠0. From this point, the same deduction can be done.


## 2.1.4. Summary about Special Cases

All of these derivations show that original formulae (4), (5), (7), (10), and (13) can be well applied in these special cases without any modification. This is underlined by the Monte-Carlo test run in [24] where Case 2 and Case 3 scenarios are included both with special tC/y=tC/x and crossing in-front and behind vertical setups and there were no singularity or any problem with the formulae. Of course, for real data, the theoretical values will be corrupted by noise but considering multiple measurements will smoothen this out in some extent.After developing a method to determine the 3D minimum distance (relative to intruder size) between two A/C in an encounter scenario, it should be tested in detail. Reference [24] presented promising results of a SIL Monte-Carlo simulation campaign. The current article presents results based on real flight data. This will be done in Section 5 after introducing the test A/C and scenarios and the used camera system and image processing methods.


## 3. Test Aircraft and Scenarios

Real flight tests are conducted with two UAVs on an airfield near Budapest (Hungary). The own A/C is a large 3.5 m wingspan, about 10-12 kg twin-engine A/C called Sindy (see Figure6) developed in our institute (SZTAKI) to carry large payloads such as the sense and avoid system (for details and building instructions (as it is an open source project), see [27]).Figure 6
Sindy, twin-engine own aircraft.The intruder A/C is a small 1.27 m wingspan 1.5-2 kg single-engine A/C which is the E-flite Ultra Stick 25e (see Figure7). The small size of the intruder makes detection and decision about the collision difficult as the numerical results will show. However, as the S&A system works acceptably in this scenario which is close to the worst case, the system will perform even better as the conditions get better.Figure 7
Ultrastick single-engine intruder.Both A/C are equipped withIMU+GPS+Pitottube and an air pressure sensor and an MPC5200 onboard microcontroller with attitude estimation and autopilot algorithms. Waypoint or trajectory tracking (constructed from straight lines and arcs) is also possible with them. For the details of the sense and avoid hardware/software system on Sindy, see Section 4.The S&A test flights were conducted in October/November 2016 by flying the two A/C on parallel, horizontal, straight flight paths with 15 m (close flight) and 30 m (far flight) horizontal side distances between them. The flying on nonparallel paths would be much more complicated as [28] points out. The own A/C air relative velocity was set to be 20 m/s while that of the intruder was 17 m/s giving 37 m/s closing speed in the ideal case. During the first set of test flights, the avoidance was deactivated to make in detail offline algorithm evaluation possible. Note that if an avoidance maneuver is done, the own A/C trajectory is modified and it is impossible to evaluate the closest point as if there was no avoidance. In the second set of test flights, avoidance in the horizontal plane (only considering P1 parameters) was activated and maximum bank turn avoidance maneuvers were done when required. Setup of a safety altitude separation (15 m for close test and 10 m for far test) was mandatory in the first set and advised in the second (that is why only horizontal avoidance was applied). Depending on the initial conditions, actual wind disturbance, and turbulence, the tracking of the straight line can show some oscillatory behaviour which can compromise camera data and the precision of the S&A algorithm. A close test case and a far test case with almost straight trajectories are shown in Figure 8 while oscillating ones are shown in Figure 9.Figure 8
Almost straight close (1) and far (2) trajectories.Figure 9
Oscillating close (1) and far (2) trajectories.


## 4. Camera System

Figure10 presents the camera system hardware and the main characteristics of the components. The system includes two Basler dcA 1280 cameras which have a 65.4° field of view (FOV). They are setup with βC=±30° camera angles which gives finally ±62.7° total FOV. This is in good agreement with the ±60° angle range suggested for pilots to scan for intruders in [25]. The NVIDIA Jetson TK1 board is responsible for image processing and decision making. This hardware is capable of processing and saving the 2x HD monochrome video input at about 8 fps. Communication between the autopilot and camera system is minimal. The autopilot sends the orientation of the aircraft while the camera system sends only an evasion alert and suggested avoidance direction.Figure 10
Vision system mounted on Sindy UAV.


### 4.1. Image Processing

Recent papers on flying object detection and tracking utilize deep neural networks [29]. The accuracy of these methods is impressive; however, these techniques have too large computational complexity for onboard real-time computation. The evolution of new hardware and research on more efficient neural network design will lead to robust flying object detection in the future; however, now, different methods were applied. Here, we introduce our approach which was used in the real flight tests. The task of detecting flying objects in images is easier being against the sky than against the ground. All objects which are not a cloud or the sun and have no connection to the ground are flying objects and can be determined on a single frame. Against the ground, we could detect and track objects which are moving compared to the background considering multiple frames for the detection. Our method now focused on detection against sky background with the following three stages: sky-ground separation, blob detection against sky background, and false object filtering.


#### 4.1.1. Sky-Ground Separation

We perform sky-ground separation by horizon detection. The basic method was published in [30]. The main idea is to calculate a horizon line based on the orientation of the aircraft and the known relative orientation of the camera. This horizon line is not accurate enough, because of the small airframe deformations during maneuvers and calibration errors, but this makes it very easy to create a good sky-ground separation (Figure 11).Figure 11
Horizon detection in the image with radial distortion. The blue line represents the horizon calculated from orientation, and the red line corresponds to the improved sky-ground separation.


#### 4.1.2. Blob Detection against Sky Background

Algorithm1 is an example preprocessing method to extract candidate objects. Candidate objects include real flying objects with false positives (cloud edges and so forth). A successfully detected aircraft and its enlarged and shaded region can be seen in Figure 12. Thus, an object filter is applied to classify flying objects.Algorithm 1. Preprocessing.
1. Step:5×5 erosion (grayscale)
2. Step:13×13 Gaussian filter
3. Step: adaptive threshold: pixelthreshold=9×9average−2
4. Step:5×5 erosion (binary)
5. Step:5×5 dilation (binary).Figure 12
Aircraft detection. The red region in the enlarged image shows the object mask from preprocessing.


#### 4.1.3. Object Filter

Most of the false sky objects can be eliminated efficiently with simple filtering rules. The main rule defines threshold on the pixel variance in the neighborhood of the object mask which eliminates cloud edges. These handcrafted rules were enough for successful flight tests in good conditions (homogeneous clear or cloudy background); however, in some cases, we had still too many false positive objects which can cause false evasions. With the next-generation faster NVIDIA TX1-based camera system, classification of candidate objects could be performed with a small convolutional neural network.


#### 4.1.4. Parameters to the Collision Decision

Finally, the horizontal and vertical centroid positions (x,y) and object sizes (Sx,Sy) are obtained after detection. From these parameters, a rectangle is created which is egomotion compensated (see [10]). This means rotation from the camera to the body, from the body to the trajectory, and then from the trajectory to the camera coordinate system. The trajectory system is a body system rotated to be parallel to the straight flight trajectory. This way, the intruder direction is evaluated relative to the trajectory direction irrespective of the current orientation of the own aircraft. After egomotion compensation, (x,y) and (Sx,Sy) are recalculated and then passed to the algorithm in Section 2.


## 4.1. Image Processing

Recent papers on flying object detection and tracking utilize deep neural networks [29]. The accuracy of these methods is impressive; however, these techniques have too large computational complexity for onboard real-time computation. The evolution of new hardware and research on more efficient neural network design will lead to robust flying object detection in the future; however, now, different methods were applied. Here, we introduce our approach which was used in the real flight tests. The task of detecting flying objects in images is easier being against the sky than against the ground. All objects which are not a cloud or the sun and have no connection to the ground are flying objects and can be determined on a single frame. Against the ground, we could detect and track objects which are moving compared to the background considering multiple frames for the detection. Our method now focused on detection against sky background with the following three stages: sky-ground separation, blob detection against sky background, and false object filtering.


### 4.1.1. Sky-Ground Separation

We perform sky-ground separation by horizon detection. The basic method was published in [30]. The main idea is to calculate a horizon line based on the orientation of the aircraft and the known relative orientation of the camera. This horizon line is not accurate enough, because of the small airframe deformations during maneuvers and calibration errors, but this makes it very easy to create a good sky-ground separation (Figure 11).Figure 11
Horizon detection in the image with radial distortion. The blue line represents the horizon calculated from orientation, and the red line corresponds to the improved sky-ground separation.


### 4.1.2. Blob Detection against Sky Background

Algorithm1 is an example preprocessing method to extract candidate objects. Candidate objects include real flying objects with false positives (cloud edges and so forth). A successfully detected aircraft and its enlarged and shaded region can be seen in Figure 12. Thus, an object filter is applied to classify flying objects.Algorithm 1. Preprocessing.
1. Step:5×5 erosion (grayscale)
2. Step:13×13 Gaussian filter
3. Step: adaptive threshold: pixelthreshold=9×9average−2
4. Step:5×5 erosion (binary)
5. Step:5×5 dilation (binary).Figure 12
Aircraft detection. The red region in the enlarged image shows the object mask from preprocessing.


### 4.1.3. Object Filter

Most of the false sky objects can be eliminated efficiently with simple filtering rules. The main rule defines threshold on the pixel variance in the neighborhood of the object mask which eliminates cloud edges. These handcrafted rules were enough for successful flight tests in good conditions (homogeneous clear or cloudy background); however, in some cases, we had still too many false positive objects which can cause false evasions. With the next-generation faster NVIDIA TX1-based camera system, classification of candidate objects could be performed with a small convolutional neural network.


### 4.1.4. Parameters to the Collision Decision

Finally, the horizontal and vertical centroid positions (x,y) and object sizes (Sx,Sy) are obtained after detection. From these parameters, a rectangle is created which is egomotion compensated (see [10]). This means rotation from the camera to the body, from the body to the trajectory, and then from the trajectory to the camera coordinate system. The trajectory system is a body system rotated to be parallel to the straight flight trajectory. This way, the intruder direction is evaluated relative to the trajectory direction irrespective of the current orientation of the own aircraft. After egomotion compensation, (x,y) and (Sx,Sy) are recalculated and then passed to the algorithm in Section 2.


## 4.1.1. Sky-Ground Separation

We perform sky-ground separation by horizon detection. The basic method was published in [30]. The main idea is to calculate a horizon line based on the orientation of the aircraft and the known relative orientation of the camera. This horizon line is not accurate enough, because of the small airframe deformations during maneuvers and calibration errors, but this makes it very easy to create a good sky-ground separation (Figure 11).Figure 11
Horizon detection in the image with radial distortion. The blue line represents the horizon calculated from orientation, and the red line corresponds to the improved sky-ground separation.


## 4.1.2. Blob Detection against Sky Background

Algorithm1 is an example preprocessing method to extract candidate objects. Candidate objects include real flying objects with false positives (cloud edges and so forth). A successfully detected aircraft and its enlarged and shaded region can be seen in Figure 12. Thus, an object filter is applied to classify flying objects.Algorithm 1. Preprocessing.
1. Step:5×5 erosion (grayscale)
2. Step:13×13 Gaussian filter
3. Step: adaptive threshold: pixelthreshold=9×9average−2
4. Step:5×5 erosion (binary)
5. Step:5×5 dilation (binary).Figure 12
Aircraft detection. The red region in the enlarged image shows the object mask from preprocessing.


## 4.1.3. Object Filter

Most of the false sky objects can be eliminated efficiently with simple filtering rules. The main rule defines threshold on the pixel variance in the neighborhood of the object mask which eliminates cloud edges. These handcrafted rules were enough for successful flight tests in good conditions (homogeneous clear or cloudy background); however, in some cases, we had still too many false positive objects which can cause false evasions. With the next-generation faster NVIDIA TX1-based camera system, classification of candidate objects could be performed with a small convolutional neural network.


## 4.1.4. Parameters to the Collision Decision

Finally, the horizontal and vertical centroid positions (x,y) and object sizes (Sx,Sy) are obtained after detection. From these parameters, a rectangle is created which is egomotion compensated (see [10]). This means rotation from the camera to the body, from the body to the trajectory, and then from the trajectory to the camera coordinate system. The trajectory system is a body system rotated to be parallel to the straight flight trajectory. This way, the intruder direction is evaluated relative to the trajectory direction irrespective of the current orientation of the own aircraft. After egomotion compensation, (x,y) and (Sx,Sy) are recalculated and then passed to the algorithm in Section 2.


## 5. Real Flight Test Results

From the first set of flight tests (without avoidance), several important parameters can be evaluated such as the detection distancedD of the intruder, the time frame Ft to make a decision and execute avoidance (from time of detection to time of CPA), and the fps (frames per second) rate of the camera system. 8 close and 9 far encounters were flown, and data synchronization between the two A/C was done considering the global GPS UTC (coordinated universal time) time stamps. Note that the GPS frequency is 5 Hz so the time resolution of data is 0.2 s. The minimum, maximum, and mean values and the standard deviations (STD) of the above parameters are summarized in Table 1 for the close and far encounters.Table 1
Basic essential parameters of S&A system application.
Parameter (unit)
MIN
MAX
Mean
STD
Close
dD (m)
122.2
204.5
171.9
26.6
Ft (s)
3.47
5.59
4.68
0.77
fps (Hz)
7
8
7.52
0.37
Far
dD (m)
54.7
244.4
163.7
53.3
Ft (s)
1.77
6.35
4.48
1.36
fps (Hz)
6.34
8
7.41
0.56Considering the data in the table, thefps of the camera system with onboard image processing and decision making is between 6.3 and 8 Hz with a mean of about 7.5 Hz. This is about half or less than the 15-20 Hz in [12], but in that paper, they did ground-based calculations with a laptop computer while we applied an onboard system carriable by the Sindy A/C and making the S&A system autonomous. Future hardware developments can easily increase this rate.The detection distance is between 122 and 204 m in the close scenarios while it is between 55 and 244 m in the far ones. Here, detection means that the tracking of the intruder by the camera system is stable and continuous from that point. The mean values are 172 and 164 m, respectively. Regarding the means, these results outperform the 60-70 m detection distance of a 1.6 m wingspan A/C in [12] (note that our intruder wingspan is only 1.27 m). The very small (55 m) detection distance in the far scenarios is an outlier (the mean 164 m is close to the close scenario 172 m mean), and we could not figure out what caused it. The flight trajectories were good, and so, the intruder could not move out of the camera system FOV so possibly some image blur or distortion could cause it.The time frame to make a decision is 3.47-5.59 s for close and 1.77 6.35 s for far scenarios because of the same reasons as fordD. The means are 4.7 s and 4.5 s, respectively, which do not provide too much time to make a decision and execute avoidance. Considering the later selected 3 s TTCPA threshold which provides 2 to 4 s real time until the closest point and the required 2-3 s convergence time of our algorithm, it would be better to have 6-7 s from observation until the closest point. That is why such a small A/C as an Ultrastick aircraft is a critical case which makes avoidance with large safety distance impossible. On the contrary, our article [31] points out that for general aviation A/C (7 m and above wingspan), the time from detection to the closest point is at least 7 seconds which can provide enough time for successful avoidance. It is also worth noting that approximately 10 s [25] and 12.5 s [26] are the required time for a human pilot to detect, identify, and avoid an intruder which is outperformed by the 6-7 s time of our system. From dD and Ft, the average closing speed of the A/C can be calculated which results between 31 and 40 m/s with a mean of about 36.5 m/s close to the theoretical 37 m/s value.


### 5.1. More Details about Intruder Detectability

An exhaustive study about the detectability of intruders with different camera systems, sizes, and speed is published in [31]. Some of the observations deduced there are repeated here to put our previous observations into context. At first, it is worth to mention that the detectability of an object by a camera mainly depends on the sensor area occupied by the image on the sensor. This area can be directly related to the object pixels through the so-called pixel pitch parameter. Another possibility can be to use the subtended angle of the object, but this suffers from distortion by approaching the edges of the image plane. So finally, the covered area (size) was considered in [31]. Table 2 repeats the considered camera parameters from there.Table 2
Camera data.
Type
Basler dcA 1280
Basler acA 1440
Basler acA 2040
Basler acA 2500
Pixel count (Mp)
1.2
1.6
3
5
Camera resolution
1280×960
1440×1080
2048×1536
2592×1944
Pixel pitch (μm)
3.75
3.45
3.45
2.2
Focal lengthf (px)
997.3
1217.4
1739.1
1818.2
View angle (°)
65.4
61.2
61
71
MIN sizeS (px)
8
9
9
14Our camera in the flight tests is the Basler dcA 1280. Considering theR=1.27m size of the Ultrastick aircraft and the Z=164m average detection distance makes it possible to calculate the average detectable size in pixels as S=f⋅R/Z≈8px. Considering the pixel pitch, this gives about 30 μm size on the sensor. The MIN sizes for the other cameras in the table were calculated from this value. The minimum and maximum detection distances for several aircraft sizes were calculated considering the worst (dcA 1280) and best (acA 2040) cameras as shown in Table 3.Table 3
Camera system detectability results.
AC type
Size (m)
Worst range (m)
Best range (m)
Boeing 747
68
8477
13140
Boeing C-17
52
6482
10048
Embraer ERJ-145
25
3117
4830
MQ-9 Reaper
15.6
1945
3014
MQ-8 Fire Scout
7.7
960
1488
Ultrastick
1.27
159
245For the Ultrastick A/C, the worst distance (159 m) is close to our mean detection distance (164 m) which is not surprising as we calculated the parameters from this. Our maximum detection distance (244 m) is very close to the best result (245 m) which means that even the worst camera can give superior results in some cases but, of course, using higher-resolution cameras can improve the results. These tests in [31] and the pinhole projection formula Z=f⋅R/S show that the camera should have a large focal length (f) and small detectable occupied sensor area S. That is why acA 2040 outperformed acA 2500 because S is much larger and the focal length is almost the same in the latter case.The time to the closest point of approach at the time of A/C detection strongly depends on the speed of the own and intruder A/C. A thorough evaluation of this problem is done in [31].


### 5.2. Results and Thresholding with Flight Test Data

The first set of flights (without avoidance) makes it possible to compare S&A system estimates with real physical data. Real 2D (only horizontal) and 3D closest points were calculated from GPS data for all close and far encounters. Then the calculated distances were normalized by intruder wingspanb=R=1.27m and height H=0.33m (the real height/size ratio thus results as k=0.33/1.27=0.26) to finally get tCPAMIN,CPAvMIN′,CPAhMIN,βCPAMIN. As the A/C fly on horizontal paths, there is no crossing in the vertical plane and so the vertical CPAvMIN′≠0 and the 2D horizontal CPA is the same as the that of the 3D. So, it is enough to compare the estimated and real 3D results.The S&A algorithm was run offline for all close and far encounters, and the estimates are compared to the A/C trajectory-based real values in Figures13–19. The LS line fits were done on 7 data points to consider about 1 s data (fps is about 7.5).Figure 13
Estimation errors of 3D TTCPA (red x: close encounters; blue circle: far encounters).Figure 14
Estimation errors of 3D horizontal CPA (red x: close encounters; blue circle: far encounters).Figure 15
TTCPA-horizontal CPA nomogram (red x: close encounters; blue circle: far encounters).Figure 16
TTCPA-vertical CPA (R relative) nomogram (red x: vertical far encounters; blue circle: vertical close encounters).Figure 17
Estimation errors of the direction of CPA (red x: close encounters; blue circle: far encounters).Figure 18
Estimation errors of size ratio (red x: close encounters; blue circle: far encounters).Figure 19
TTCPA-vertical CPA (H relative) nomogram (red x: vertical far encounters; blue circle: vertical close encounters).Figure13 shows the estimated tCPAMIN on the horizontal axis and the error realtCPAMIN−estimatedtCPAMIN on the vertical axis. Note that tCPAMIN changes with t in every time step as the intruder approaches the own A/C. These tCPAMIN time series values are plotted together with their instantaneous errors in the figure. If the error is positive, there is more time to CPA than estimated. This representation is advantageous to consider the possibilities for decision threshold (THS) selection. The figure shows that first (when the estimated tCPAMIN is large), the TTCPA is overestimated and then, as the intruder approaches, it is underestimated. Table 4 summarizes the observations.Table 4
tCPAMIN estimation results.
EstimatedtCPAMIN (s)
4
3
2
1
Error (s)
-2/+1
-1/+1
-0.5/+2.7
-0.5/+1
RealtCPAMIN (s)
2/5
2/4
1.5/4.7
1.5/2According to the table, an estimatedtCPAMIN THS of 3 s will result in a decision between 2 and 4 s real time before CPA. Selecting a larger THS will not increase the possible smallest real value (2 s) while smaller THS will decrease it, so 3 s is the ideal selection.Figure14 shows the estimated CPAhMIN on the horizontal axis and the error realCPAhMIN−estimatedCPAhMIN on the vertical axis. Here, the real CPAhMIN is a constant given value in every encounter but it changes encounter by encounter depending on the shape of the flight trajectories. At the same time, the estimated one changes every time step as new and new images are considered in the estimation method. If the error is positive, the intruder is farther than estimated. It is interesting to see that for close encounters, CPAhMIN is usually underestimated, while for far encounters, it is usually overestimated. Table 5 summarizes the observations. According to the table, the avoidance of an intruder with a horizontal CPA of 4 or below can be guaranteed by any THS and only intruders above 25-28 CPA can be surely considered as nonthreatening. Avoidance of any intruder at or below 4 CPA will not give well clear but can provide the last chance to solve the situation without collision.Table 5
CPAhMIN estimation results.
EstimatedCPAhMIN (-)
26
20
15
10
Error (-)
-22/+5
-16/+7
-11/+10
-4.2/+15
RealCPAhMIN (-)
4/28
4/27
4/25
5.8/25In case of CPA THS selection, the relation of CPA estimates with the TTCPA estimates is vital to evaluate decidability (between collision and noncollision). This is because collision decision is done examining the absolute CPA estimate when estimated TTCPA is below the selected THS. That is why TTCPA-CPA nomograms should be plotted and applied for THS selection. Figure15 shows the estimated tCPAMIN on the horizontal axis (plotted only below 3.5 s as its THS is 3 s) and CPAhMIN on the vertical axis together with two possible horizontal CPA THS values. The first (magenta dashed line) is 20 while the second (green continuous line) is 26. The first (20) has more close scenario crosses above it and so can result in missed detection with higher probability. Below 2.25 s estimated TTCPA, all the red crosses are below the second (26) so this should guarantee detection of close threat at least below this time. Of course, the larger THS will result in more false alarms as several far scenario circles are below it.Table6 shows estimation error evaluation results for vertical CPA (CPAvMIN′) similarly to Table 5 based on a figure similar to Figure 14 but not presented here. In this case, intruders below 4-5 vertical CPA (relative to intruder horizontal size) can be guaranteed to be avoided and those above 10-11 CPA can be considered as nonthreatening.Table 6
Vertical CPA estimation results.
EstimatedCPAv (-)
25
20
15
10
Error (-)
-20/-15
-16/-10
-10/-4
-5/+1
RealCPAv (-)
5/10
4/10
5/11
5/11Figure16 shows the estimated tCPAMIN on the horizontal axis and CPAvMIN′ on the vertical axis together with two possible vertical CPA THS values. The first (magenta dashed line) is 5 while the second (green continuous line) is 20. The first has several circles (which is the vertical close scenario with 10 m altitude separation) above it and so can result in missed detections. The second has only a few circles above it, but all the crosses are below if tCPAMIN<1.25s. This means that on one side, it possibly will not do missed detection but on the other side, it will generate false alarms for all far vertical encounters.Figure17 shows the real tCPAMIN on the horizontal axis and the error estimatedβCPAMIN−realβCPAMIN on the vertical axis. The figure shows that the direction estimation is very uncertain so it is not advisable to use it for precise avoidance. This large uncertainty is caused by the oscillating flight paths despite the egomotion compensation of image data. Note that egomotion compensation removes only the rotational component of the oscillation; the position change disturbances are not removed. Detailed examination of the estimates shows that their sign is correct and so at least the side of the intruder can be decided based on them.Figure18 shows the real tCPAMIN on the horizontal axis and the size ratio error estimatedk−0.26/0.26·100% on the vertical axis. Here, 0.26 is the real size ratio. The figure shows that k is overestimated most of the time.Figure19 shows the tCPAMIN−CPAvMIN nomogram with thresholds from Figure 16 transformed by dividing by 0.26 which is the real size ratio. The figure shows that here again, thresholds which cause missed detections or false alarms can only be selected as the vertically close and far cases are overlapping even for small estimated TTCPA.Finally, a decision test campaign was run offline on the real flight data applying five THS pairs shown in Table7. The decision is done if the estimated tCPAMIN is below its threshold. Collision is decided if the estimated CPA is below the given THS (this makes possible a more precise decision than in [12] where every intruder is avoided which is in the image for a longer time). Results are summarized in Table 8 for the close encounters and Table 9 for the far encounters referencing the THS case numbers from the first row of Table 7. In the close encounters the horizontal decision should be collision and the vertical non-collision (NC). In the far encounters horizontal non-collision and vertical collision decisions are expected. In Tables 8 and 9, the numbers mean the real TTCPA time in seconds at the time of collision decision if collision is decided. In case of NC decision, an NC symbol is used.Table 7
Selected threshold combinations.
Number
1
2
3
4
5
tCPAMIN (s)
3
3
2
3
3
EstimatedCPAh (-)
26
20
20
—
—
EstimatedCPAv′ (-)
—
—
—
5
20Table 8
Decision results in the close encounters.
Flight case
1
2
3
4
5
6
7
8
No. 1 H
1.56
4.15
3.1
4.65
2.7
3
4.2
2.5
No. 2 H
1.43
3.14
2.97
4.65
2.7
3
4.2
2.5
No. 3 H
1.43
3.14
2.97
4.52
2.33
1.8
4.2
2.5
No. 4 V
1.31
2.73
2.3
NC
NC
NC
4.2
NC
No. 5 V
1.94
4.15
3.1
4.52
2.7
3
4.2
2.5
H denotes horizontal decisions, while V denotes vertical decisions.Table 9
Decision results in the far encounters.
Flight case
1
2
3
4
5
6
7
8
9
No. 1 H
4.02
1
NC
3.7
3.13
3.3
1.3
3.7
1.8
No. 2 H
4.02
NC
NC
3.7
3.13
3.3
NC
3.7
1.8
No. 3 H
NC
NC
NC
1.58
NC
2.8
NC
1.8
1.5
No. 4 V
2.87
1
2.29
1.96
1.58
3.3
2.2
1.8
1
No. 5 V
4.02
1
2.42
3.7
2.73
3.9
2.2
3.7
1.8
H denotes horizontal decisions, while V denotes vertical decisions.Table8 shows that no. 1 to no.3 THS pairs well detect the possibility of collision in the horizontal plane. The only difference is at the time of detection.The earliest detection is achieved by thetCPAMIN<3s and CPAhMIN<26 pair (no. 1) while the latest by the tCPAMIN<2s and CPAhMIN<20 (no. 3). The real TTCPA values are between 2 and 4 s in most of the cases as predicted in Table 4. The only outlier is the first flight where the estimated TTCPA crosses the THS late after a sudden decrease. In the vertical (nos. 4 and 5) plane, NCs are expected while 50% false alarm is achieved by no. 4 and 100% by no. 5. In view of the 8 − 10 range of real CPAv′ values based on GPS data, this is not surprising as only in above 10-11 is a guarantee of NC decision present both with 5 and with 20 THS (see Table 6).Table9 shows that no. 1 to no. 3 THS pairs generate a lot of false alarms as follows: 8/9 (89%), 6/9 (67%), and 4/9 (44%). Given the 18−30 range with mean 25 of real CPAh values based on GPS data, this is not surprising as only in above 25-28 is a guarantee of NC decision present both with 20 and with 26 THS (see Table 5). In the vertical plane, there is no missed detection; the only difference between tCPAMIN<3s, CPAvMIN<5 (no. 4) and tCPAMIN<3s, CPAvMIN<20 (no. 5) is the time of decision. The latter decides earlier on which can be advantageous to execute avoidance.Summarizing the results, the no. 2 THS pair (tCPAMIN<3s and CPAhMIN<20) can be useful in the horizontal plane as it provides more time for avoidance than the no. 3 pair though it gives very high false alarm rate. In the vertical plane, the no. 5 pair (tCPAMIN<3s and CPAhMIN<20) is better as it provides more time for avoidance despite the 100% false alarm rate. Note that the false alarm rates should get lower as the horizontal or vertical distance is increased between the A/C.A second set of flight tests was done in 2016 (on a different day) by applying thetCPAMIN<2s and CPAhMIN<20 (no. 3) THS pair only in the horizontal plane and activating the S&A avoidance. In the close scenarios, a 0/8 (0%) missed detection rate was observed while in the far ones, the false alarm rate was 4/8 (50%). All these rates are close to the ones estimated from the tables. Checking the A/C trajectories with the avoidance has shown that the collision decision was made sometimes too late to significantly increase the distance between A/C. This could be predicted from the detection time results in Table 8. Possibly, the 3 s TTCPA THS can improve these results; this requires a future test campaign. Ground and aerial (onboard) videos of the second set of flight tests in 2016 can be found in the supplementary material and on our Youtube channel called AeroGNC.


## 5.1. More Details about Intruder Detectability

An exhaustive study about the detectability of intruders with different camera systems, sizes, and speed is published in [31]. Some of the observations deduced there are repeated here to put our previous observations into context. At first, it is worth to mention that the detectability of an object by a camera mainly depends on the sensor area occupied by the image on the sensor. This area can be directly related to the object pixels through the so-called pixel pitch parameter. Another possibility can be to use the subtended angle of the object, but this suffers from distortion by approaching the edges of the image plane. So finally, the covered area (size) was considered in [31]. Table 2 repeats the considered camera parameters from there.Table 2
Camera data.
Type
Basler dcA 1280
Basler acA 1440
Basler acA 2040
Basler acA 2500
Pixel count (Mp)
1.2
1.6
3
5
Camera resolution
1280×960
1440×1080
2048×1536
2592×1944
Pixel pitch (μm)
3.75
3.45
3.45
2.2
Focal lengthf (px)
997.3
1217.4
1739.1
1818.2
View angle (°)
65.4
61.2
61
71
MIN sizeS (px)
8
9
9
14Our camera in the flight tests is the Basler dcA 1280. Considering theR=1.27m size of the Ultrastick aircraft and the Z=164m average detection distance makes it possible to calculate the average detectable size in pixels as S=f⋅R/Z≈8px. Considering the pixel pitch, this gives about 30 μm size on the sensor. The MIN sizes for the other cameras in the table were calculated from this value. The minimum and maximum detection distances for several aircraft sizes were calculated considering the worst (dcA 1280) and best (acA 2040) cameras as shown in Table 3.Table 3
Camera system detectability results.
AC type
Size (m)
Worst range (m)
Best range (m)
Boeing 747
68
8477
13140
Boeing C-17
52
6482
10048
Embraer ERJ-145
25
3117
4830
MQ-9 Reaper
15.6
1945
3014
MQ-8 Fire Scout
7.7
960
1488
Ultrastick
1.27
159
245For the Ultrastick A/C, the worst distance (159 m) is close to our mean detection distance (164 m) which is not surprising as we calculated the parameters from this. Our maximum detection distance (244 m) is very close to the best result (245 m) which means that even the worst camera can give superior results in some cases but, of course, using higher-resolution cameras can improve the results. These tests in [31] and the pinhole projection formula Z=f⋅R/S show that the camera should have a large focal length (f) and small detectable occupied sensor area S. That is why acA 2040 outperformed acA 2500 because S is much larger and the focal length is almost the same in the latter case.The time to the closest point of approach at the time of A/C detection strongly depends on the speed of the own and intruder A/C. A thorough evaluation of this problem is done in [31].


## 5.2. Results and Thresholding with Flight Test Data

The first set of flights (without avoidance) makes it possible to compare S&A system estimates with real physical data. Real 2D (only horizontal) and 3D closest points were calculated from GPS data for all close and far encounters. Then the calculated distances were normalized by intruder wingspanb=R=1.27m and height H=0.33m (the real height/size ratio thus results as k=0.33/1.27=0.26) to finally get tCPAMIN,CPAvMIN′,CPAhMIN,βCPAMIN. As the A/C fly on horizontal paths, there is no crossing in the vertical plane and so the vertical CPAvMIN′≠0 and the 2D horizontal CPA is the same as the that of the 3D. So, it is enough to compare the estimated and real 3D results.The S&A algorithm was run offline for all close and far encounters, and the estimates are compared to the A/C trajectory-based real values in Figures13–19. The LS line fits were done on 7 data points to consider about 1 s data (fps is about 7.5).Figure 13
Estimation errors of 3D TTCPA (red x: close encounters; blue circle: far encounters).Figure 14
Estimation errors of 3D horizontal CPA (red x: close encounters; blue circle: far encounters).Figure 15
TTCPA-horizontal CPA nomogram (red x: close encounters; blue circle: far encounters).Figure 16
TTCPA-vertical CPA (R relative) nomogram (red x: vertical far encounters; blue circle: vertical close encounters).Figure 17
Estimation errors of the direction of CPA (red x: close encounters; blue circle: far encounters).Figure 18
Estimation errors of size ratio (red x: close encounters; blue circle: far encounters).Figure 19
TTCPA-vertical CPA (H relative) nomogram (red x: vertical far encounters; blue circle: vertical close encounters).Figure13 shows the estimated tCPAMIN on the horizontal axis and the error realtCPAMIN−estimatedtCPAMIN on the vertical axis. Note that tCPAMIN changes with t in every time step as the intruder approaches the own A/C. These tCPAMIN time series values are plotted together with their instantaneous errors in the figure. If the error is positive, there is more time to CPA than estimated. This representation is advantageous to consider the possibilities for decision threshold (THS) selection. The figure shows that first (when the estimated tCPAMIN is large), the TTCPA is overestimated and then, as the intruder approaches, it is underestimated. Table 4 summarizes the observations.Table 4
tCPAMIN estimation results.
EstimatedtCPAMIN (s)
4
3
2
1
Error (s)
-2/+1
-1/+1
-0.5/+2.7
-0.5/+1
RealtCPAMIN (s)
2/5
2/4
1.5/4.7
1.5/2According to the table, an estimatedtCPAMIN THS of 3 s will result in a decision between 2 and 4 s real time before CPA. Selecting a larger THS will not increase the possible smallest real value (2 s) while smaller THS will decrease it, so 3 s is the ideal selection.Figure14 shows the estimated CPAhMIN on the horizontal axis and the error realCPAhMIN−estimatedCPAhMIN on the vertical axis. Here, the real CPAhMIN is a constant given value in every encounter but it changes encounter by encounter depending on the shape of the flight trajectories. At the same time, the estimated one changes every time step as new and new images are considered in the estimation method. If the error is positive, the intruder is farther than estimated. It is interesting to see that for close encounters, CPAhMIN is usually underestimated, while for far encounters, it is usually overestimated. Table 5 summarizes the observations. According to the table, the avoidance of an intruder with a horizontal CPA of 4 or below can be guaranteed by any THS and only intruders above 25-28 CPA can be surely considered as nonthreatening. Avoidance of any intruder at or below 4 CPA will not give well clear but can provide the last chance to solve the situation without collision.Table 5
CPAhMIN estimation results.
EstimatedCPAhMIN (-)
26
20
15
10
Error (-)
-22/+5
-16/+7
-11/+10
-4.2/+15
RealCPAhMIN (-)
4/28
4/27
4/25
5.8/25In case of CPA THS selection, the relation of CPA estimates with the TTCPA estimates is vital to evaluate decidability (between collision and noncollision). This is because collision decision is done examining the absolute CPA estimate when estimated TTCPA is below the selected THS. That is why TTCPA-CPA nomograms should be plotted and applied for THS selection. Figure15 shows the estimated tCPAMIN on the horizontal axis (plotted only below 3.5 s as its THS is 3 s) and CPAhMIN on the vertical axis together with two possible horizontal CPA THS values. The first (magenta dashed line) is 20 while the second (green continuous line) is 26. The first (20) has more close scenario crosses above it and so can result in missed detection with higher probability. Below 2.25 s estimated TTCPA, all the red crosses are below the second (26) so this should guarantee detection of close threat at least below this time. Of course, the larger THS will result in more false alarms as several far scenario circles are below it.Table6 shows estimation error evaluation results for vertical CPA (CPAvMIN′) similarly to Table 5 based on a figure similar to Figure 14 but not presented here. In this case, intruders below 4-5 vertical CPA (relative to intruder horizontal size) can be guaranteed to be avoided and those above 10-11 CPA can be considered as nonthreatening.Table 6
Vertical CPA estimation results.
EstimatedCPAv (-)
25
20
15
10
Error (-)
-20/-15
-16/-10
-10/-4
-5/+1
RealCPAv (-)
5/10
4/10
5/11
5/11Figure16 shows the estimated tCPAMIN on the horizontal axis and CPAvMIN′ on the vertical axis together with two possible vertical CPA THS values. The first (magenta dashed line) is 5 while the second (green continuous line) is 20. The first has several circles (which is the vertical close scenario with 10 m altitude separation) above it and so can result in missed detections. The second has only a few circles above it, but all the crosses are below if tCPAMIN<1.25s. This means that on one side, it possibly will not do missed detection but on the other side, it will generate false alarms for all far vertical encounters.Figure17 shows the real tCPAMIN on the horizontal axis and the error estimatedβCPAMIN−realβCPAMIN on the vertical axis. The figure shows that the direction estimation is very uncertain so it is not advisable to use it for precise avoidance. This large uncertainty is caused by the oscillating flight paths despite the egomotion compensation of image data. Note that egomotion compensation removes only the rotational component of the oscillation; the position change disturbances are not removed. Detailed examination of the estimates shows that their sign is correct and so at least the side of the intruder can be decided based on them.Figure18 shows the real tCPAMIN on the horizontal axis and the size ratio error estimatedk−0.26/0.26·100% on the vertical axis. Here, 0.26 is the real size ratio. The figure shows that k is overestimated most of the time.Figure19 shows the tCPAMIN−CPAvMIN nomogram with thresholds from Figure 16 transformed by dividing by 0.26 which is the real size ratio. The figure shows that here again, thresholds which cause missed detections or false alarms can only be selected as the vertically close and far cases are overlapping even for small estimated TTCPA.Finally, a decision test campaign was run offline on the real flight data applying five THS pairs shown in Table7. The decision is done if the estimated tCPAMIN is below its threshold. Collision is decided if the estimated CPA is below the given THS (this makes possible a more precise decision than in [12] where every intruder is avoided which is in the image for a longer time). Results are summarized in Table 8 for the close encounters and Table 9 for the far encounters referencing the THS case numbers from the first row of Table 7. In the close encounters the horizontal decision should be collision and the vertical non-collision (NC). In the far encounters horizontal non-collision and vertical collision decisions are expected. In Tables 8 and 9, the numbers mean the real TTCPA time in seconds at the time of collision decision if collision is decided. In case of NC decision, an NC symbol is used.Table 7
Selected threshold combinations.
Number
1
2
3
4
5
tCPAMIN (s)
3
3
2
3
3
EstimatedCPAh (-)
26
20
20
—
—
EstimatedCPAv′ (-)
—
—
—
5
20Table 8
Decision results in the close encounters.
Flight case
1
2
3
4
5
6
7
8
No. 1 H
1.56
4.15
3.1
4.65
2.7
3
4.2
2.5
No. 2 H
1.43
3.14
2.97
4.65
2.7
3
4.2
2.5
No. 3 H
1.43
3.14
2.97
4.52
2.33
1.8
4.2
2.5
No. 4 V
1.31
2.73
2.3
NC
NC
NC
4.2
NC
No. 5 V
1.94
4.15
3.1
4.52
2.7
3
4.2
2.5
H denotes horizontal decisions, while V denotes vertical decisions.Table 9
Decision results in the far encounters.
Flight case
1
2
3
4
5
6
7
8
9
No. 1 H
4.02
1
NC
3.7
3.13
3.3
1.3
3.7
1.8
No. 2 H
4.02
NC
NC
3.7
3.13
3.3
NC
3.7
1.8
No. 3 H
NC
NC
NC
1.58
NC
2.8
NC
1.8
1.5
No. 4 V
2.87
1
2.29
1.96
1.58
3.3
2.2
1.8
1
No. 5 V
4.02
1
2.42
3.7
2.73
3.9
2.2
3.7
1.8
H denotes horizontal decisions, while V denotes vertical decisions.Table8 shows that no. 1 to no.3 THS pairs well detect the possibility of collision in the horizontal plane. The only difference is at the time of detection.The earliest detection is achieved by thetCPAMIN<3s and CPAhMIN<26 pair (no. 1) while the latest by the tCPAMIN<2s and CPAhMIN<20 (no. 3). The real TTCPA values are between 2 and 4 s in most of the cases as predicted in Table 4. The only outlier is the first flight where the estimated TTCPA crosses the THS late after a sudden decrease. In the vertical (nos. 4 and 5) plane, NCs are expected while 50% false alarm is achieved by no. 4 and 100% by no. 5. In view of the 8 − 10 range of real CPAv′ values based on GPS data, this is not surprising as only in above 10-11 is a guarantee of NC decision present both with 5 and with 20 THS (see Table 6).Table9 shows that no. 1 to no. 3 THS pairs generate a lot of false alarms as follows: 8/9 (89%), 6/9 (67%), and 4/9 (44%). Given the 18−30 range with mean 25 of real CPAh values based on GPS data, this is not surprising as only in above 25-28 is a guarantee of NC decision present both with 20 and with 26 THS (see Table 5). In the vertical plane, there is no missed detection; the only difference between tCPAMIN<3s, CPAvMIN<5 (no. 4) and tCPAMIN<3s, CPAvMIN<20 (no. 5) is the time of decision. The latter decides earlier on which can be advantageous to execute avoidance.Summarizing the results, the no. 2 THS pair (tCPAMIN<3s and CPAhMIN<20) can be useful in the horizontal plane as it provides more time for avoidance than the no. 3 pair though it gives very high false alarm rate. In the vertical plane, the no. 5 pair (tCPAMIN<3s and CPAhMIN<20) is better as it provides more time for avoidance despite the 100% false alarm rate. Note that the false alarm rates should get lower as the horizontal or vertical distance is increased between the A/C.A second set of flight tests was done in 2016 (on a different day) by applying thetCPAMIN<2s and CPAhMIN<20 (no. 3) THS pair only in the horizontal plane and activating the S&A avoidance. In the close scenarios, a 0/8 (0%) missed detection rate was observed while in the far ones, the false alarm rate was 4/8 (50%). All these rates are close to the ones estimated from the tables. Checking the A/C trajectories with the avoidance has shown that the collision decision was made sometimes too late to significantly increase the distance between A/C. This could be predicted from the detection time results in Table 8. Possibly, the 3 s TTCPA THS can improve these results; this requires a future test campaign. Ground and aerial (onboard) videos of the second set of flight tests in 2016 can be found in the supplementary material and on our Youtube channel called AeroGNC.


## 6. Conclusions

This paper presented the real flight validation of 3D collision decision based on monocular images in aircraft sense and avoid. After the review of literature and author’s previous works, the theoretical basics were briefly summarized targeting the estimation of 3D time to the closest point of approach and closest point of approach. The feasibility of possible special cases with the derived formulae is examined, and there is no need to handle any of them separately—the original formulae can handle everything well. Then the test UAVs and flight test scenarios were introduced together with the onboard camera and computing system. A short analysis of intruder aircraft detectability with different camera systems is provided referencing the author’s separate exhaustive work about this topic. Real test flights with close and far encounters without avoidance were conducted, and the estimated collision parameters were compared to the aircraft trajectory-based real ones. A strategy to select the decision thresholds was proposed, and then the offline decision results were presented and evaluated and finally compared to real flight test results (with avoidance) with a given threshold pair. The final conclusion is that the proposed system is capable of detecting and attempting to avoid close intruder aircraft even if it is as small as 1.27 m (wingspan) but of course larger size intruders are easier to be detected and avoided. Intruders with 7 m wingspan and above are predicted to be avoided with enough safety distance, but this also depends on own aircraft maneuvering capabilities and intruder speed. By applying other camera systems, this can be further improved.Future research should focus on two different topics. First, a relation between software-in-the-loop simulation results in previous works of the authors [23, 24] and the real flight tests should be established to make it possible to run close to real simulations with virtual intruders. The final goal is to tune the decision thresholds based solely on the knowledge of own aircraft dynamics, camera system parameters, and possible intruder size and velocity ranges all integrated in a realistic simulation environment. Second, an appropriate avoidance strategy based on the dynamic capabilities of the own aircraft and the estimated collision parameters should be proposed and tested.



---

*Source: 7191839-2019-05-19.xml*