Forward motion compensated flight path

With reference to FIG. 1, camera 12 mounted on airframe 10 captures an image of first field of view 20 along first optical axis 21 aimed at first object of interest 23. During the time of exposure, airframe 10 flies first flight path arc 22 centered on first object of interest 23 with a radius substantially equal to the distance between camera 12 and first object of interest 23. Airframe 10 pivots camera 12 around first object of interest 23 while the shutter in camera 12 is open. This is repeated around each subsequent object of interest to produce a scalloped or slalom path, namely Forward Motion Compensated (FMC) flight path 33.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of provisional patent application Ser. No. 61/930,424 filed 2014 Jan. 22 by the present inventor.

BACKGROUND

Prior Art

Motion of a camera during image exposure causes blur in the image. In aerial imaging one source of camera motion is unexpected gusts or turbulence disturbing the airframe. This motion can be mitigated with gyroscopes such as those produced by Kenyon Labs (U.S. Pat. No. 2,811,042 and U.S. Pat. No. 2,570,130); or with image stabilization available in many consumer cameras and smart phones. This image stabilization uses gyroscope, inertial measurement unit (IMU), and accelerometer sensors to moves lenses or the sensor array in the camera, thereby compensating for the camera motion.

A second source of motion is the planned forward movement of the airframe across the field of view. This motion is present even in wind-still conditions, and generally cannot be detected or compensated by gyros or accelerometers since there is little acceleration or change in angular velocity in straight, level flight. Prior approaches to Forward Motion Compensation (FMC) includemechanically translating the film, sensor, or a lens;rotating a mirror, prism, or camera (CN 202748029 U); ortranslating the charges in the photo sites on an electro-optical array (U.S. Pat. No. 5,155,597, time delay and integration, astronomy orthogonal transfer CCD).
The last option is implemented for many expensive medium and large format cameras designed especially for aerial imaging, where it is known as time delay and integration.

Providing FMC for a camera in smooth flight is a difficult problem (U.S. Pat. No. 5,155,597 cites 40 sources) that is still very active (U.S. Pat. No. 5,155,597 is in turn referenced by 74! patents and applications). The breakthrough in the FMC flight path described here came when we turned the problem inside out; to paraphrase “Ask not what FMC can do for the camera in your airframe, ask what your airframe can do for FMC in your camera”.

Blur due to forward motion for vertical aerial photography is measured by the amount of a ground sample distance (GSD) the airframe moves during exposure. The GSD is calculated as the object distance times the pixel dimension over the lens focal length. So for a 100 m flying height using a camera with 0.005 mm size pixels and a 25 mm focal length lens taking vertical pictures, the GSD is 2 cm. A common criterion for acceptable blur is that the airframe not move more than a fraction, say ½, of a GSD during the exposure time. So for an exposure time of 1/1000 second, this criterion limits the ground speed in this scenario to 1 cm in 1/1000 s or 10 m/s (36 km/hr or 22 mph).

Unmanned aerial vehicles (UAVs) operate primarily under autopilot control. They are typically lighter and more maneuverable than manned aircraft, and they fly at much lower heights (smaller GSD) due to government regulations. To achieve good flight speeds with a small GSD, forward motion compensation should be used. UAVs don't need to worry about airsickness.

SUMMARY

Forward motion compensation reduces blur due to planned airframe motion in images. As described in the following embodiments, this can be done by modifying the flight path to fly an arc around the object of interest during the time of exposure.

With reference toFIG. 1, camera12mounted on airframe10captures an image of first field of view20along first optical axis21aimed at first object of interest23. During the time of exposure, airframe10flies first flight path arc22centered on first object of interest23with a radius substantially equal to the distance between camera12and first object of interest23. Airframe10pivots camera12around first object of interest23while the shutter in camera12is open. This is repeated around each subsequent object of interest to produce a scalloped or slalom path, namely Forward Motion Compensated (FMC) flight path33.

ADVANTAGES

Although forward motion compensation is well known in the prior art, various aspects of the embodiments of my FMC flight path are improvements because:They do not add anya. weight,b. mechanical complexity, norc. electronic complexity.For the same blur criterion they allow one or more ofa. a longer exposure (more light so less noise),b. a slower lens f# (lighter, less expensive lens),c. imaging in lower light conditions (sunrise, twilight, or overcast),d. greater depth of field (focusing less critical), ore. faster flight.

Other advantages of one or more aspects will be apparent from a consideration of the drawings and ensuing description.

DETAILED DESCRIPTION

This section describes several embodiments of the Forward Motion Compensated Flight Path with reference toFIGS. 1-8.

FIG. 1. is a perspective view (not to scale) of an aerial survey taking images of multiple overlapping fields of view. Camera12, autopilot14, control surfaces8, and propulsion system16are mounted on airframe10flying in planned flight path18. Control surfaces8consist of one or more of elevator, rudder, ailerons, flaps, or combinations thereof (e.g. elevon). Camera12images first field of view20along first optical axis21aimed at first object of interest23. First flight path arc22is centered on first object of interest23with radius substantially equal to the distance between camera12and first object of interest23along optical axis21.

Second optical axis25points to second object of interest27at the center of second field of view24. Second flight path arc26is centered on second object of interest27with radius matching the distance between camera12when it gets to this point in the flight path and second object of interest27along second optical axis25.

Third flight path arc29is centered on third object of interest28and has radius equal to the distance between camera12when it gets to this point in the flight path and third object of interest28. As the fourth object of interest, hill30increases the height of the ground so fourth flight path arc32has a smaller radius corresponding to the reduced object distance along fourth optical axis31. The combination of arcs22,26,29, and32, and the flight segments joining them produces forward motion compensated flight path33.

In preparing for exposure for first field of view20, autopilot14adjusts control surfaces8to orient and maintain airframe10in flight along first flight path arc22. First flight path arc22has radius equal to the distance between first object of interest23and camera12along first optical axis21and is oriented to keep the object end of first optical axis21in one position. Airframe10pivots camera12around the object end of first optical axis21, i.e. around first object of interest23, while the shutter in camera12is open. Autopilot14may also turn off propulsion system16to reduce blur due to vibration.

Images used for photogrammetry in aerial surveys should be taken vertically, with tilts up to 3 degrees acceptable. For vertical images, first flight path arc22is oriented along the pitch axis of airframe10. For the example given previously with a flying height of 100 m, +/−3 degrees corresponds to an arc length of ˜10.5 m. The FMC ground speed is thus limited by the length 10.5 m in 1/1000s exposure time, rather than the ½*2 cm GSD. Thus airframe10could fly a thousand times faster, or the exposure time can be increased, or the lens stopped down, or some combination of these.

After the shutter (not shown inFIG. 1) in camera12closes at the end of first flight path arc22, autopilot14starts propulsion system16(if it was stopped), adjusts control surfaces8to reduce pitch and fly towards the second field of view24or the next waypoint. At second field of view24the process is repeated to fly second flight path arc26pivoting second optical axis25around second object of interest27while the shutter for camera12is open. The resulting FMC flight path33is a series of arcs22,26,29,32centered on the objects of interest at the center of the fields of view, connected by more freeform paths shown as dashed lines.

FIG. 2. is a section of the flight path along planned flight path18for one exposure. A flight plan is usually generated prior to flight to guide autopilot14. It is often communicated as a series of waypoints that are stored on autopilot14prior to flight. Autopilot14contains a processor and memory and controls the flight using control surfaces8. During flight autopilot14navigates airframe10between waypoints using segments such a planned flight path18. Images are acquired either as quickly as possible for camera12, or at predefined waypoints. To take a blur-free image of first field of view20along optical axis21pointing to object50with camera12at vertical position44, the flight path is slightly modified. Rather than simply flying straight and level along planned flight path18, the flight path is curved from start of arc42to end of arc46, while the shutter on camera12is open. The linear translational motion of planned flight path18is replaced with angular motion along first flight path arc22while the shutter is open. The arc is centered on object50and the radius of the arc is object distance48, from camera12to object50.

For fairly level terrain, planned flight path18is often straight and level flight. For terrain with large elevation changes, the flight plan may follow the terrain elevations more closely. The object distance is still calculated as the distance from the ground to the camera at the point of exposure. The FMC flight path is a small variation on the flight plan with arcs at each exposure.

FIG. 3. is a perspective view (not to scale) of aerial inspection of power lines. In this example for 345 kV transmission lines, shield wires60and62are 1-2 cm in diameter and have to be inspected for lighting strikes. Phase conductors64,65, and66are 3-5 cm in diameter with a steel core and stranded aluminum conductors that have to be inspected for corrosion, broken strands, failing spacers, failing dampers, or splice failures. Phase conductors are ˜10 m apart, and towers68and70are 200-600 m apart. Airframe10with autopilot14is flying a planned catenary arc above shield wire60with camera12at a side oblique angle along optical axis72. Rather than a linear path such as planned flight path18, the planned flight path is a catenary arc to closely follow shield wires60and62and phase conductors64,65, and66. Superimposed on the planned catenary flight path are FMC arcs74and76to give the FMC catenary flight path78.

FIG. 4is a section across the transmission lines and flight path78. To inspect all five wires60,62,64,65, and66in one flight, camera12is oriented with field of view80. Optical axis72is roughly 45 degrees off the horizontal to provide better stereo images when the wires are imaged with a second flight on the other side. The two flights together provide good coverage of the uppermost parts of the wires that are not visible from the ground. Virtual object location82is chosen to reduce the forward motion blur for all the wires in the image, as described below.

To be able to see damage on phase conductor65requires resolution of a number of pixels, say 6, across the diameter of the wire. This gives a wire sample distance (by analogy to our GSD) of 40/6˜7 mm, so for our example camera with 25 mm focal length lens and 0.005 mm pixels, camera12should be within 33 m of phase conductor65. With a 1/1000s exposure time and a blur criterion of ½ wire sample distance, this gives a maximum inspection speed of 3.3 m/s. This is painfully slow, and is probably below the stall speed of a fixed wing UAV that can carry even a lightweight consumer camera.

To make this inspection practical, autopilot14controls airframe10to fly a catenary arc to track phase conductor65elevation, with superimposed flight path arcs74,76, and one more for each inspection image. Flight path arcs74and76lie in a plane oblique to the horizon in the plane swept out by optical axis72, with a radius equal to the object distance along optical axis72. The object in the example of the previous paragraph being phase conductor65, so the radius is 33 m. For this example, forward motion corrected catenary flight path78is the catenary of the conductors, modified by flight path arcs74,76, and one more for each image, as well as the flight segments joining the arcs.

When a single object is the primary focus of the inspection, like phase conductor65in the previous two paragraphs, then the object distance is simply the distance from the camera to that object. When additional objects at different distances are to be inspected in one image, then a virtual object location82can be calculated to reduce forward motion blur for all the objects of interest. InFIG. 4. camera12would be flown in an arc coming out of the page towards the viewer. Consider camera12as pivoting around virtual object location82, then the relative motion at shield wire60and phase conductor64would also be out of the page, whereas the relative motion at shield wire62and phase conductors65and66would be into the page, away from the viewer. As a first approximation, virtual object location82is chosen to be the average of the distances from camera12to each of the objects of interest60,62,64,65, and66, i.e. at the average location. If some objects are more important, then they can be weighted more heavily in the average. For a known arrangement of objects of interest60,62,64,65, and66, virtual object location82is calculated using a simple or weighted average by a separate computer doing flight planning (not shown) or during the flight using autopilot14.

The autofocus in consumer cameras calculates object distances for thousands of points and tries to determine the best setting for the focus. However, it does not know, a priori, the importance of different points in the inspection. For a field of view such as80, it mostly measures the distant ground and not the five wires of interest. For the inspection illustrated inFIGS. 3 and 4, the configuration of wires is known. Based on the inspection objectives, a better estimate of virtual location82can often be calculated either during flight planning or during flight by autopilot14.

FIG. 5is a front view of a focal plane shutter typical of many consumer cameras. Just in front of the photosensitive array are opaque top shutter84and bottom shutter88. Prior to an exposure, both shutters are at the top of the frame. The exposure starts with bottom shutter88moving down at a speed ˜1-10 m/s. For exposure times longer than the flash sync time (typically 1/125- 1/250s), bottom shutter88reaches the bottom of the frame before top shutter84starts to move. The whole photosensitive array is exposed during the time of the flash.

For exposure times shorter than the flash sync speed, top shutter84starts to move down at the same speed as bottom shutter88, after a delay equal to the exposure time, thus forming a slit86between the shutters. The height of slit86divided by the shutter speed equals the exposure time.

The focal plane shutters in most consumer cameras are driven by springs and move at 5-8 m/s. The Panasonic Lumix GM1™ uses a shutter stepper motor89. Instead of bottom shutter88, this camera uses an electronic first curtain which clears the rows of pixels of charge as it moves down the sensor. This shutter has a sync speed of 1/50s and it travels at about 1 m/s. The slower speed and controllable shutter stepper motor89are a good match for the FMC applications described below. All-electronic shutters can be even slower and potentially easier to match to airframe responsiveness.

Note that for exposure times shorter than the flash sync time, the photosensitive elements are not all exposed at the same instant in time. The top of the image (bottom of the object since the lens inverts) is exposed before the bottom of the image (top of object). For fast moving objects this can lead to rolling shutter artifacts, such as leaning forward in the direction of motion.

This difference in time of exposure can be used to advantage in forward motion compensated flight paths.FIG. 6. shows airframe10with control surfaces8, camera12, autopilot14, and propulsion system16flying along planned flight path18. Camera12is mounted upside down pointing forward and down to image upcoming terrain in forward looking field of view90along forward oblique optical axis91. On flat ground, forward looking field of view90is a trapezoid. Further points at larger object distances at the front of trapezoidal field of view90need less FMC, whereas closer points at shorter object distances at the back of trapezoidal field of view90need more FMC. Decreasing radius FMC arcs92,93, and94start fairly straight and then pitch more and more in smaller radii during the time of exposure.

The exposure time on camera12is set shorter than the flash sync time, so slit86sweeps across the photosensitive array from bottom to top (camera is upside down compared toFIG. 5). The furthest points in forward looking field of view90are exposed first. These correspond to large object distances so radius of arc92is large at first. As slit86sweeps up the photosensitive array, closer points in forward looking field of view90are exposed, so the object distances decrease, and the radius of arc92decreases. This decreasing radius flight arc corresponds well with the response of an airframe to dive or nose down pitch control. Control of the slit speed using shutter stepper motor89or an all-electronic shutter allows control over how fast the radius is reduced, making it easier to match to responsiveness of airframe10. Decreasing radius FMC arcs93and94correspond to subsequent images and join together with connecting segments to make forward oblique FMC flight path95.

FIG. 7. shows airframe10with control surfaces8, camera12, autopilot14, and propulsion system16flying along planned flight path18. Camera12is mounted upside down pointing sideways and down to image terrain in side looking field of view96along side oblique optical axis97. On level ground, side looking field of view96is a trapezoid. Further points at larger object distances at the far side of trapezoidal field of view96need less FMC, whereas closer points at shorter object distances at the near side of trapezoidal field of view96need more FMC. Decreasing radius FMC arcs98and99start fairly straight and then curve in smaller and smaller radii during the time of exposure.

The exposure time on camera12is set shorter than the flash sync time, so slit86sweeps across the photosensitive array from bottom to top (camera is upside down compared toFIG. 5). The furthest points in side looking field of view96are exposed first. These correspond to large object distances so radius of arc98is large at first. As slit86sweeps up the photosensitive array, closer points in side looking field of view96are exposed, so the object distances decrease, and the radius of arc98decreases. Decreasing radius FMC arcs99corresponds to a subsequent image. These arcs join together with connecting segments to make side oblique FMC flight path100.

FIG. 8is a flowchart showing how to implement forward motion compensated flight path33to take a single image. This would be repeated for each subsequent image. The assumption is that airframe10is flying along planned flight path18either from the start of the flight, from a prior waypoint, or from taking a prior image. Setup for picture decision110is whether airframe10is close enough that camera12can image field of view20. Taking a picture has to be anticipated because both camera12and airframe10take time to set up.

Consumer cameras have a delay of 1/20thto ¼ second from the time they are triggered until the time the shutter opens. This time can be reduced by prefocusing or by using manual focus. An airframe can be very maneuverable and on the edge of stability like a jet fighter, or very stable and slow responsive like an airliner. More maneuverable airframes respond more quickly to changes in control. From the time the controls are adjusted, there will be a delay of tens of milliseconds until the airframe is in the new attitude.

For a given camera and airframe, the setup times can be measured and stored on autopilot14before the flight. The current location and groundspeed can be determined a) from GPS/GNSS or b) from recognizable features on the ground and the airspeed less the wind speed. Given the setup times, the current location, and the current groundspeed, setup for picture decision110can be made.

If airframe10is not close enough to begin setup for picture110, then it continues flight to waypoint process112. If airframe10is close enough then it has to determine object distance process114by one of the means described in the following three paragraphs. For an aerial survey with vertical images, as illustrated inFIGS. 1 and 2, the object distance is the height above ground. The ground elevation comes from a map or a digital elevation model (DEM), and the altitude of airframe10comes from an altimeter. Alternatively the distance to the object of interest can be measured with RADAR, LiDAR, SONAR, or other well-known distance measuring tools.

For the aerial inspection illustrated inFIGS. 3 and 4, the object distance is determined in the flight plan to be close enough to get the desired wire sample distance. For a single object of interest, the object distance is the distance from the camera to the object. For close distances the camera autofocus will measure the distance. If there are multiple objects of interest, e.g. wires60,62,64,65,66inFIG. 4, at different distances in one image, a virtual object location82may be calculated to reduce the forward motion blur over all the objects of interest. A first approximation is to simply average the distances. Then the radius used for the FMC arc is the distance from camera12to virtual object82.

For forward oblique imagingFIG. 6and side oblique imagingFIG. 7; use trigonometry with the height above ground, the camera angle of declination, and the lens angle of view to determine the furthest and closest object distances, at the far and near ends of the trapezoidal field of view assuming flat ground. To continue the earlier example, suppose camera12has 4800 pixels of dimension 0.005 mm in the vertical direction, i.e. the light sensitive sensor is 24 mm high. Then with a 25 mm focal length lens the vertical angle of view is 51 degrees. If inFIG. 6, forward oblique optical axis91is inclined 45 degrees up from vertical, then the furthest points of trapezoid90are 71 degrees up from vertical and the closest points of trapezoid90are 19 degrees up from vertical. For a flying height of 100 m and assuming flat terrain, the furthest points at the front of trapezoidal field of view90will be 302 m away and the closest points at the back of trapezoidal field of view90will be 106 m away. Arc92will start with radius of 302 m and decrease to radius 106 m as slit86is scanned from back to front across light sensitive array of camera12.

Trigger camera, adjust controls, set shutter timer, and optionally turn off propulsion process116are four steps to prepare for taking a photo. The order of triggering the camera or adjusting the controls depends on whether the camera or airframe has the greater delay. A shutter timer is set based on the camera delay plus the shutter time, to know when to stop flying FMC arc22. For an airframe that can glide for the duration of the shutter open time, the propulsion may be turned off to reduce blur due to vibration. This is easy for fixed wing airframes, less so for rotary wings. Once all four steps are completed, the airframe will fly arc process118. For a stable airframe control surfaces8may have to be feathered to maintain the arc, but for an airframe with neutral stability it may be sufficient to get the airframe into the correct configuration. For the reducing radii arcs for the forward obliqueFIG. 6and side obliqueFIG. 7imaging, the control surfaces8have to be adjusted to tighten the arc.

In the loop between fly arc process118and shutter closed decision120, the shutter timer is decremented. When the shutter is closed, the airframe will resume flight to next waypoint process122.

This section illustrated details of specific embodiments, but persons skilled in the art can readily make modifications and changes that are still within the scope. For example, the discussion and illustrations featured a fixed wing UAV airframe with an autopilot and propeller. The same concepts apply to manned airframes, jet engines, as well as to rotary wing airframes. For a helicopter, flight control is exercised through pitch angle of the main rotor and speed of the tail rotor. For a multicopter, flight control is through differential motor speeds. The power line inspection example was for a high voltage transmission line, but the same concepts apply to distribution lines, communication lines, electric railway lines, ski lift cables, suspension bridges, towers, and oblique aerial photography. Accordingly, the scope should be determined not by the embodiments illustrated, but by the appended claims and their legal equivalents.