Patent ID: 12236626

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

The following description uses an unmanned aerial vehicle (UAV) as an example of a movable object. UAVs include, e.g., fixed-wing aircrafts and rotary-wing aircrafts such as helicopters, quadcopters, and aircraft having other numbers and/or configurations of rotors. It will be apparent to those skilled in the art that other types of movable objects may be substituted for UAVs as described below in accordance with embodiments of the disclosure.

The present disclosure provides techniques related to target tracking by UAVs. In some embodiments, a UAV is configured to receive target information from a remote control unit, such as a user-operated device. The target information is related to a target to be tracked by an imaging device coupled to the UAV. The target information is used by the UAV to cause the imaging device to automatically track the target, e.g., to maintain a predetermined position and/or size of the target within one or more images captured by the imaging device. In some embodiments, tracking of the target is performed while the UAV is controlled by communications from a control unit, such as communications including user commands and/or predetermined navigation paths. In some embodiments, the control unit is configured to display images from the imaging device as well as allowing user input related to the target information.

In some embodiments, a user selects a target from an image displayed on a user interface of the control unit. For example, the image is displayed and the input is received via a touchscreen of the control unit. In some embodiments, when the target information is configured, the control unit and/or UAV manage operations associated with target tracking. Managing target tracking operations includes, e.g., adjusting motion of the UAV, adjusting the carrier and/or adjusting the imaging device. For example, the attitude, position, velocity, zoom, and/or other aspects of the UAV and/or imaging device are automatically adjusted to ensure that the target is maintained at a designated position and/or size within the images captured by the imaging device. In some embodiments, images captured during the tracking process (e.g., videos or pictures) are streamed to the control unit in real time or substantially real time for display, playback, storage, and/or other purposes. In this manner, a user is enabled to manage target tracking (e.g., by selecting a target for tracking) without the burden of managing operations involved in piloting the UAV to maintain a view of the target.

In accordance with various embodiments described herein, a UAV avoids obstacles detected while the UAV tracks a target. When an obstacle is detected, a distance between the UAV and the obstacle is determined. If the obstacle does not pose an immediate threat to the safety of the UAV (e.g., still far from the UAV), movement characteristics of the UAV are adjusted in a proactive manner to maintain a predetermined distance between the obstacle and the UAV. Proactive adjustment of movement characteristics may include selecting a set of potential motion adjustment options and determining route optimization scores for each option. Movement of the UAV is adjusted in accordance with the adjustment option that has the highest route optimization score or at least a route optimization score above a predefined threshold. If the obstacle may cause an immediate threat to the safety of the UAV (e.g., very close to the UAV), movement characteristics of the UAV are adjusted in a reactive manner to avoid collision of the UAV with the obstacle ranging from reducing acceleration or velocity of the UAV to reversing the motion of the UAV, depending on how close the UAV is to the obstacle.

FIG.1illustrates a target tracking system100, in accordance with various embodiments of the present disclosure. Target tracking system100includes a movable object102and a control unit104. In some embodiments, target tracking system100is used to track target106.

In some embodiments, target106includes natural and/or man-made objects such geographical landscapes (e.g., mountains, vegetation, valleys, lakes, and/or rivers), buildings, and/or vehicles (e.g., aircrafts, ships, cars, trucks, buses, vans, and/or motorcycles). In some embodiments, the target106includes live subjects such as people and/or animals. In some embodiments, target106is moving, e.g., moving relative to a reference frame (such as the Earth and/or movable object102). In some embodiments, target106is static. In some embodiments, target106includes an active target system that transmits information about target106, such as the target's GPS location, to movable object102, control unit104, and/or computing device126. For example, information is transmitted to movable object102via wireless communication from a communication unit of the active target to communication system120of movable object102. Active targets include, e.g., friendly vehicles, buildings, and/or troops. In some embodiments, target106includes a passive target (e.g., that does not transmit information about target106). Passive targets include, e.g., neutral or hostile vehicles, buildings, and/or troops.

In some embodiments, movable object102is configured to communicate with control unit104, e.g., via wireless communications124. For example, movable object102receives control instructions from control unit104and/or sends data (e.g., data from movable object sensing system122) to control unit104.

Control instructions include, e.g., navigation instructions for controlling navigational parameters of movable object102such as position, orientation, attitude, and/or one or more movement characteristics of movable object102, carrier108, and/or payload110. In some embodiments, control instructions include instructions directing movement of one or more of movement mechanisms114. For example, control instructions are used to control flight of a UAV. In some embodiments, control instructions include information for controlling operations (e.g., movement) of carrier108. For example, control instructions are used to control an actuation mechanism of carrier108so as to cause angular and/or linear movement of payload110relative to movable object102. In some embodiments, control instructions are used to adjust one or more operational parameters for payload110, such as instructions for capturing one or more images, capturing video, adjusting a zoom level, powering on or off, adjusting an imaging mode (e.g., capturing still images or capturing video), adjusting an image resolution, adjusting a focus, adjusting a viewing angle, adjusting a field of view, adjusting a depth of field, adjusting an exposure time, adjusting a shutter speed, adjusting a lens speed, adjusting an ISO, changing a lens and/or moving payload110(and/or a part of payload110, such as imaging device214). In some embodiments, the control instructions are used to control communication system120, sensing system122, and/or another component of movable object102.

In some embodiments, control instructions from control unit104include target information, as described further below with regard toFIG.7.

In some embodiments, movable object102is configured to communicate with computing device126. For example, movable object102receives control instructions from computing device126and/or sends data (e.g., data from movable object sensing system122) to computing device126. In some embodiments, communications from computing device126to movable object102are transmitted from computing device126to cell tower130(e.g., via internet128) and from cell tower130to movable object102(e.g., via RF signals). In some embodiments, a satellite is used in lieu of or in addition to cell tower130.

In some embodiments, target tracking system includes additional control units104and/or computing devices126configured to communicate with movable object102.

FIG.2Aillustrates an exemplary movable object102in a target tracking system100, in accordance with some embodiments. In some embodiments, one or more components of movable object, such as processor(s)116, memory118, communication system120, and sensing system122, are connected by data connections, such as a control bus112. A control bus optionally includes circuitry (sometimes called a chipset) that interconnects and controls communications between system components.

Movable object102typically includes one or more processing units116, memory118, one or more network or other communications interfaces120, sensing system112, and one or more communication buses112for interconnecting these components. In some embodiments, movable object102is a UAV. Although movable object102is depicted as an aircraft, this depiction is not intended to be limiting, and any suitable type of movable object can be used.

In some embodiments, movable object102includes movement mechanisms114(e.g., propulsion mechanisms). Although the plural term “movement mechanisms” is used herein for convenience of reference, “movement mechanisms114” refers to a single movement mechanism (e.g., a single propeller) or multiple movement mechanisms (e.g., multiple rotors). Movement mechanisms114include one or more movement mechanism types such as rotors, propellers, blades, engines, motors, wheels, axles, magnets, nozzles, animals, and/or human beings. Movement mechanisms114are coupled to movable object102at, e.g., the top, bottom, front, back, and/or sides. In some embodiments movement mechanisms114of a single movable object102include multiple movement mechanisms each having the same type. In some embodiments, movement mechanisms114of a single movable object102include multiple movement mechanisms having different movement mechanism types. Movement mechanisms114are coupled to movable object102(or vice-versa) using any suitable means, such as support elements (e.g., drive shafts) or other actuating elements (e.g., actuators132). For example, an actuator132receives control signals from processor(s)116(e.g., via control bus112) that activates the actuator to cause movement of a movement mechanism114. For example, processor(s)116include an electronic speed controller that provides control signals to actuators134.

In some embodiments, the movement mechanisms114enable movable object102to take off vertically from a surface or land vertically on a surface without requiring any horizontal movement of movable object102(e.g., without traveling down a runway). In some embodiments, movement mechanisms114are operable to permit movable object102to hover in the air at a specified position and/or orientation. In some embodiments, one or more of the movement mechanisms114are controllable independently of one or more of the other movement mechanisms114. For example, when movable object102is a quadcopter, each rotor of the quadcopter is controllable independently of the other rotors of the quadcopter. In some embodiments, multiple movement mechanisms114are configured for simultaneous movement.

In some embodiments, movement mechanisms114include multiple rotors that provide lift and/or thrust to movable object. The multiple rotors are actuated to provide, e.g., vertical takeoff, vertical landing, and hovering capabilities to movable object102. In some embodiments, one or more of the rotors spin in a clockwise direction, while one or more of the rotors spin in a counterclockwise direction. For example, the number of clockwise rotors is equal to the number of counterclockwise rotors. In some embodiments, the rotation rate of each of the rotors is independently variable, e.g., for controlling the lift and/or thrust produced by each rotor, and thereby adjusting the spatial disposition, velocity, and/or acceleration of movable object102(e.g., with respect to up to three degrees of translation and/or up to three degrees of rotation).

In some embodiments, carrier108is coupled to movable object102. A payload110is coupled to carrier108. In some embodiments, carrier108includes one or more mechanisms that enable payload110to move relative to movable object102, as described further with reference toFIG.2B. In some embodiments, payload110is rigidly coupled to movable object102such that payload110remains substantially stationary relative to movable object102. For example, carrier108is coupled to payload110such that payload is not movable relative to movable object102. In some embodiments, payload110is coupled to movable object102without requiring carrier108.

Communication system120enables communication with control unit104and/or computing device126, e.g., via wireless signals124. The communication system120includes, e.g., transmitters, receivers, and/or transceivers for wireless communication. In some embodiments, the communication is one-way communication, such that data is transmitted only from movable object102to control unit104, or vice-versa. In some embodiments, communication is two-way communication, such that data is transmitted in both directions between movable object102and control unit104.

In some embodiments, movable object102communicates with computing device126. In some embodiments, movable object102, control unit104, and/or the remote device are connected to the Internet or other telecommunications network, e.g., such that data generated by movable object102, control unit104, and/or computing device126is transmitted to a server for data storage and/or data retrieval (e.g., for display by a website).

In some embodiments, sensing system122of movable object102includes one or more sensors, as described further with reference toFIG.3. In some embodiments, movable object102and/or control unit104use sensing data generated by sensors of sensing system122to determine information such as a position of movable object102, an orientation of movable object102, movement characteristics of movable object102(e.g., angular velocity, angular acceleration, translational velocity, translational acceleration and/or direction of motion along one or more axes), proximity of movable object102to potential obstacles, weather conditions, locations of geographical features and/or locations of manmade structures.

FIG.2Billustrates an exemplary carrier108in a target tracking system100, in accordance with embodiments. In some embodiments, carrier108couples a payload110to a movable object102.

In some embodiments, carrier108includes a frame assembly including one or more frame members202. In some embodiments, frame member202is coupled with movable object102and payload110. In some embodiments, frame member202supports payload110.

In some embodiments, carrier108includes one or more mechanisms, such as one or more actuators204, to cause movement of carrier108and/or payload110. Actuator204is, e.g., a motor, such as a hydraulic, pneumatic, electric, thermal, magnetic, and/or mechanical motor. In some embodiments, actuator204causes movement of frame member202. In some embodiments, actuator204rotates payload110about one or more axes, such as three axes: X axis (“pitch axis”), Z axis (“roll axis”), and Y axis (“yaw axis”), relative to movable object102. In some embodiments, actuator204translates payload110along one or more axes relative to movable object102.

In some embodiments, carrier108includes one or more carrier sensing system206, e.g., for determining a state of carrier108or payload110. Carrier sensing system206includes, e.g., motion sensors (e.g., accelerometers), rotation sensors (e.g., gyroscopes), potentiometers, and/or inertial sensors. In some embodiments, carrier sensing system206includes one or more sensors of movable object sensing system122as described below with regard toFIG.3. Sensor data determined by carrier sensing system206includes, e.g., spatial disposition (e.g., position, orientation, or attitude) and/or movement information such as velocity (e.g., linear or angular velocity) and/or acceleration (e.g., linear or angular acceleration) of carrier108and/or payload110. In some embodiments, sensing data and/or state information calculated from the sensing data are used as feedback data to control the movement of one or more components (e.g., frame member202, actuator204, and/or damping element208) of carrier108. Carrier sensor206is coupled to, e.g., frame member202, actuator204, damping element208, and/or payload110. In an embodiment, a carrier sensor206(e.g., a potentiometer) measures movement of actuator204(e.g., the relative positions of a motor rotor and a motor stator) and generates a position signal representative of the movement of the actuator204(e.g., a position signal representative of relative positions of the motor rotor and the motor stator). In some embodiments, data generated by a carrier sensor206is received by processor(s)116and/or memory118of movable object102.

In some embodiments, the coupling of carrier108to movable object102includes one or more damping elements208. Damping elements208are configured to reduce or eliminate movement of the load (e.g., payload110and/or carrier108) caused by movement of movable object102. Damping elements208include, e.g., active damping elements, passive damping elements, and/or hybrid damping elements having both active and passive damping characteristics. The motion damped by the damping elements208can include one or more of vibrations, oscillations, shaking, or impacts. Such motions may originate from motions of movable object that are transmitted to the load. For example, the motion may include vibrations caused by the operation of a propulsion system and/or other components of a movable object101.

In some embodiments, a damping element208provides motion damping by isolating the load from the source of unwanted motion by dissipating or reducing the amount of motion transmitted to the load (e.g., vibration isolation). In some embodiments, damping element208reduces the magnitude (e.g., amplitude) of the motion that would otherwise be experienced by the load. In some embodiments the motion damping applied by a damping element208is used to stabilize the load, thereby improving the quality of images captured by the load (e.g., image capturing device), as well as reducing the computational complexity of image stitching steps required to generate a panoramic image based on the captured images.

Damping element208described herein can be formed from any suitable material or combination of materials, including solid, liquid, or gaseous materials. The materials used for the damping elements may be compressible and/or deformable. For example, the damping element208is made of, e.g. sponge, foam, rubber, gel, and the like. For example, damping element208includes rubber balls that are substantially spherical in shape. The damping element208is, e.g., substantially spherical, rectangular, and/or cylindrical. In some embodiments, damping element208includes piezoelectric materials or shape memory materials. In some embodiments, damping elements208include one or more mechanical elements, such as springs, pistons, hydraulics, pneumatics, dashpots, shock absorbers, isolators, and the like. In some embodiments, properties of the damping element208are selected so as to provide a predetermined amount of motion damping. In some instances, the damping element208has viscoelastic properties. The properties of damping element208are, e.g., isotropic or anisotropic. In some embodiments, damping element208provides motion damping equally along all directions of motion. In some embodiments, damping element208provides motion damping only along a subset of the directions of motion (e.g., along a single direction of motion). For example, the damping element208may provide damping primarily along the Y (yaw) axis. In this manner, the illustrated damping element208reduces vertical motions.

In some embodiments, carrier108includes controller210. Controller210includes, e.g., one or more controllers and/or processors. In some embodiments, controller210receives instructions from processor(s)116of movable object102. For example, controller210is connected to processor(s)116via control bus112. In some embodiments, controller210controls movement of actuator204, adjusts one or more parameters of carrier sensor206, receives data from carrier sensor206, and/or transmits data to processor116.

FIG.2Cillustrates an exemplary payload110in a target tracking system100, in accordance with some embodiments. In some embodiments, payload110includes a payload sensing system212and a controller218. In some embodiments, payload sensing system212includes an imaging device214, such as a camera. In some embodiments, payload sensing system212includes one or more sensors of movable object sensing system122as described below with regard toFIG.3.

Payload sensing system212generates static sensing data (e.g., a single image captured in response to a received instruction) and/or dynamic sensing data (e.g., a series of images captured at a periodic rate, such as a video). Imaging device214includes, e.g., an image sensor216to detect light (such as visible light, infrared light, and/or ultraviolet light. In some embodiments, imaging device214includes one or more optical devices (e.g., lenses) to focus or otherwise alter the light onto image sensor216.

In some embodiments, image sensors216includes, e.g., semiconductor charge-coupled devices (CCD), active pixel sensors using complementary metal-oxide-semiconductor (CMOS) or N-type metal-oxide-semiconductor (NMOS, Live MOS) technologies, or any other types of sensors. Image sensor216and/or imaging device214capture, e.g., images and/or image streams (e.g., videos). Adjustable parameters of imaging device214include, e.g., width, height, aspect ratio, pixel count, resolution, quality, imaging mode, focus distance, depth of field, exposure time, shutter speed and/or lens configuration. In some embodiments, imaging device214is configured to capture high-definition or ultra-high-definition videos (e.g., 720p, 1080i, 1080p, 1440p, 2000p, 2160p, 2540p, 4000p, 4320p, and so on).

In some embodiments, payload110includes controller218. Controller218includes, e.g., one or more controllers and/or processors. In some embodiments, controller218receives instructions from processor(s)116of movable object102. For example, controller218is connected to processor(s)116via control bus112. In some embodiments, controller218adjusts one or more parameters of one or more sensors of payload sensing system212; receives data from one or more sensors of payload sensing system212; and/or transmits data, such as image data from image sensor216, to processor116, memory118, and/or control unit104.

In some embodiments, data generated by one or more sensors of payload sensor system212is stored, e.g., by memory118. In some embodiments, data generated by payload sensor system212are transmitted to control unit104(e.g., via communication system120). For example, video is streamed from payload110(e.g., imaging device214) to control unit104. In this manner, control unit104displays, e.g., real-time (or slightly delayed) video received from imaging device214.

In some embodiments, adjustment to the orientation, position, attitude, and/or one or more movement characteristics of movable object102, carrier108, and/or payload110is generated based at least in part on configurations (e.g., preset and/or user configured in system configuration400) of movable object102, carrier108, and/or payload110. For example, adjustment that involves rotation around two axes (e.g., yaw and pitch) is achieved solely by corresponding rotation of movable object around the two axes if payload110including imaging device214is rigidly coupled to movable object102(and hence not movable relative to movable object102) and/or payload110is coupled to movable object102via a carrier108that does not permit relative movement between imaging device214and movable object102. The same two-axis adjustment is achieved by, e.g., combining adjustment to both movable object102and carrier108if carrier108permits imaging device214to rotate around at least one axis relative to movable object102. In this case, carrier108can be controlled to implement the rotation around one or two of the two axes required for the adjustment and movable object120can be controlled to implement the rotation around one or two of the two axes. For example, carrier108includes, e.g., a one-axis gimbal that allows imaging device214to rotate around one of the two axes required for adjustment while the rotation around the remaining axis is achieved by movable object102. In some embodiments, the same two-axis adjustment is achieved by carrier108alone when carrier108permits imaging device214to rotate around two or more axes relative to movable object102. For example, carrier108includes a two-axis or three-axis gimbal.

FIG.3illustrates an exemplary sensing system122of a movable object102, in accordance with some embodiments. In some embodiments, one or more sensors of movable object sensing system122are mounted to the exterior, located within, or otherwise coupled to movable object102. In some embodiments, one or more sensors of movable object sensing system are components of carrier sensing system206and/or payload sensing system212. Where sensing operations are described as being performed by movable object sensing system122herein, it will be recognized that such operations are optionally performed by carrier sensing system206and/or payload sensing system212.

Movable object sensing system122generates static sensing data (e.g., a single image captured in response to a received instruction) and/or dynamic sensing data (e.g., a series of images captured at a periodic rate, such as a video).

In some embodiments, movable object sensing system122includes one or more image sensors302, such as image sensor308(e.g., a left stereographic image sensor) and/or image sensor310(e.g., a right stereographic image sensor). Image sensors302capture, e.g., images, image streams (e.g., videos), stereographic images, and/or stereographic image streams (e.g., stereographic videos). Image sensors302detect light, such as visible light, infrared light, and/or ultraviolet light. In some embodiments, movable object sensing system122includes one or more optical devices (e.g., lenses) to focus or otherwise alter the light onto one or more image sensors302. In some embodiments, image sensors302include, e.g., semiconductor charge-coupled devices (CCD), active pixel sensors using complementary metal-oxide-semiconductor (CMOS) or N-type metal-oxide-semiconductor (NMOS, Live MOS) technologies, or any other types of sensors.

In some embodiments, movable object sensing system122includes one or more audio transducers304. For example, an audio detection system includes audio output transducer312(e.g., a speaker), and audio input transducer314(e.g. a microphone, such as a parabolic microphone). In some embodiments, microphone and a speaker are used as components of a sonar system. In some embodiments, a sonar system is used to detect current location information of an obstacle (e.g., obstacle1316shown inFIG.15).

In some embodiments, movable object sensing system122includes one or more infrared sensors306. In some embodiments, a distance measurement system includes a pair of infrared sensors e.g., infrared sensor316(such as a left infrared sensor) and infrared sensor318(such as a right infrared sensor) or another sensor or sensor pair. The distance measurement system is used to, e.g., measure a distance to a target106and/or an obstacle1316.

In some embodiments, a system to produce a depth map includes one or more sensors or sensor pairs of movable object sensing system122(such as left stereographic image sensor308and right stereographic image sensor310; audio output transducer312and audio input transducer314; and/or left infrared sensor316and right infrared sensor318. In some embodiments, a pair of sensors in a stereo data system (e.g., a stereographic imaging system) simultaneously captures data from different positions. In some embodiments, a depth map is generated by a stereo data system using the simultaneously captured data. In some embodiments, a depth map is used for positioning and/or detection operations, such as detecting an obstacle1316, detecting current location information of an obstacle1316, detecting a target106, and/or detecting current location information for a target106.

In some embodiments, movable object sensing system122includes one or more global positioning system (GPS) sensors, motion sensors (e.g., accelerometers), rotation sensors (e.g., gyroscopes), inertial sensors, proximity sensors (e.g., infrared sensors) and/or weather sensors (e.g., pressure sensor, temperature sensor, moisture sensor, and/or wind sensor).

In some embodiments, sensing data generated by one or more sensors of movable object sensing system122and/or information determined using sensing data from one or more sensors of movable object sensing system122are transmitted to control unit104(e.g., via communication system120). In some embodiments, data generated one or more sensors of movable object sensing system122and/or information determined using sensing data from one or more sensors of movable object sensing system122is stored by memory118.

FIG.4is a block diagram illustrating an implementation of memory118, in accordance with some embodiments. In some embodiments, one or more elements illustrated inFIG.4are located in control unit104, computing device126, and/or another device.

In some embodiments, memory118stores a system configuration400. System configuration400includes one or more system settings (e.g., as configured by a manufacturer, administrator, and/or user). For example, a constraint on one or more of orientation, position, attitude, and/or one or more movement characteristics of movable object102, carrier108, and/or payload110is stored as a system setting of system configuration400.

In some embodiments, memory118stores a motion control module402. Motion control module stores, e.g., control instructions, such as control instructions received from control module104and/or computing device126. Control instructions are used for, e.g., controlling operation of movement mechanisms114, carrier108, and/or payload110.

In some embodiments, memory118stores a tracking module404. In some embodiments, tracking module404generates tracking information for target106that is being tracked by movable object102. In some embodiments, tracking information is generated based on images captured by imaging device214and/or output from image analysis module406(e.g., after pre-processing and/or processing operations have been performed on one or more images). Tracking information generated by tracking module404includes, for example, location, size, or other characteristics of target106within one or more images. In some embodiments, tracking information generated by tracking module404is transmitted to control unit104and/or computing device126(e.g., augmenting or otherwise combined with images and/or output from image analysis module406). For example, tracking information is transmitted to control unit104in response to a request from control unit104and/or on a periodic basis.

In some embodiments, memory118includes an image analysis module406. Image analysis module406performs processing operations on images, such as images captured by imaging device214. In some embodiments, image analysis module performs pre-processing on raw image data, such as re-sampling to assure the correctness of the image coordinate system, noise reduction, contrast enhancement, and/or scale space representation. In some embodiments, processing operations performed on image data (including image data that has been pre-processed) include feature extraction, image segmentation, data verification, image recognition, image registration, and/or image matching. In some embodiments, output from image analysis module406after pre-processing and/or processing operations have been performed on one or more images is transmitted to control unit104.

In some embodiments, memory118stores target information408. In some embodiments, target information408is received by movable object102(e.g., via communication system120) from control unit104, computing device126, target106, and/or another movable object.

In some embodiments, target information408includes a time value and/or expiration time indicating a period of time during which the target106is to be tracked. In some embodiments, target information408includes a flag indicating whether a targeting information entry408includes specific target information412and/or target type information410.

In some embodiments, target information408includes target type information410such as color, texture, pattern, size, shape, and/or dimension. Target type information410is, e.g., provided by a user to a user input device, such as a user input device of control unit104. In some embodiments, the user may select a pre-existing target pattern or type (e.g., a black object or a round object with a radius greater or less than a certain value). In some embodiments, user input to provide target type information includes user selection of one or more targets106from within one or more images. In some embodiments, features or characteristics of the selected targets are extracted and/or generalized to produce target type information410, which is used, e.g., to identify targets with features or characteristics indicated by target type information410. In some embodiments, feature extraction is performed by control unit104, processor(s)116of movable object102, and/or computing device126.

In some embodiments, target information408includes specific target information412for a specific target106. Specific target information412includes, e.g., an image of target106, an initial position (e.g., location coordinates, such as pixel coordinates within an image) of target106, and/or a size of target106within one or more images (e.g., images captured by imaging device214of payload110). A size of target106is stored, e.g., as a length (e.g., mm or other length unit), an area (e.g., mm2or other area unit), a number of pixels in a line (e.g., indicating a length, width, and/or diameter), a ratio of a length of a representation of the target in an image relative to a total image length (e.g., a percentage), a ratio of an area of a representation of the target in an image relative to a total image area (e.g., a percentage), a number of pixels indicating an area of target106, and/or a corresponding distance of target106from movable object102(e.g., an area of target106changes based on a distance of target106from movable object102).

In some embodiments, target information408includes expected target information414. Expected target information414specifies one or more characteristics of target106, such as a size parameter (e.g., area, diameter, length and/or width), position (e.g., relative to an image center and/or image boundary), and/or shape. In some embodiments, one or more characteristics of target106are determined from an image of target106(e.g., using image analysis techniques on images captured by imaging device112). For example, one or more characteristics of target106are determined from an orientation and/or part or all of identified boundaries of target106. In some embodiments, expected target information includes pixel coordinates and/or pixel counts to indicate, e.g., a size parameter, position, and/or shape of a target106. In some embodiments, one or more characteristics of the expected target information414are to be maintained as movable object102tracks target106(e.g., the expected target information414are to be maintained as images of target106are captured by imaging device214). Expected target information414is used, e.g., to adjust movable object102, carrier108, and/or imaging device214, e.g., such that the specified characteristics of target106are substantially maintained. In some embodiments, expected target information414is determined based on one or more of target type410and/or specific target information412. For example, a size of a target is determined from specific target information412(e.g., an image of a target106) and a value representing an area of the target is stored as expected target information414.

In some embodiments, target information408(including, e.g., target type information410, information for a specific target412, and/or expected target information414) is generated based on user input, such as input received at user input device506of control unit104. Additionally or alternatively, target information is generated based on data from sources other than control unit104. For example, target type information may be based on stored previous images of target106(e.g., images captured by imaging device214and stored by memory118), other data stored by memory118, and/or data from data stores that are remote from control unit104and/or movable object102. In some embodiments, targeting information is generated using a computer-generated image of target106.

In some embodiments, target information408is used by movable object102to track target106. For example, target information408is used by tracking module404. In some embodiments, target information408is used by an image analysis module406to identify target106. In some cases, target identification involves image recognition and/or matching algorithms based on, e.g., CAD-like object models, appearance-based methods, feature-based methods, and/or genetic algorithms. In some embodiments, target identification includes comparing two or more images to determine, extract, and/or match features contained therein.

The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these modules may be combined or otherwise re-arranged in various embodiments. In some embodiments, memory118may store a subset of the modules and data structures identified above. Furthermore, memory118may store additional modules and data structures not described above. In some embodiments, the programs, modules, and data structures stored in memory118, or a non-transitory computer readable storage medium of memory118, provide instructions for implementing respective operations in the methods described below. In some embodiments, some or all of these modules may be implemented with specialized hardware circuits that subsume part or all of the module functionality. One or more of the above identified elements may be executed by one or more processors116of movable object102. In some embodiments, one or more of the above identified elements is executed by one or more processors of a device remote from movable object102, such as control unit104and/or computing device126.

FIG.5illustrates an exemplary control unit104of target tracking system100, in accordance with some embodiments. In some embodiments, control unit104communicates with movable object102via communication system120, e.g., to provide control instructions to movable object102. Although control unit104is typically a portable (e.g., handheld) device, control unit104need not be portable. In some embodiments, control unit104is a dedicated control device (e.g., dedicated to operation of movable object102), a laptop computer, a desktop computer, a tablet computer, a gaming system, a wearable device (e.g., glasses, gloves, and/or helmet), a microphone, and/or a combination thereof.

Control unit104typically includes one or more processing units502, a communication system510(e.g., including one or more network or other communications interfaces), memory504, one or more input/output (I/O) interfaces (e.g., display506and/or input device508) and one or more communication buses512for interconnecting these components.

In some embodiments, a touchscreen display includes display508and input device506. A touchscreen display optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. A touchscreen display and processor(s)502optionally detect contact and any movement or breaking thereof using any of a plurality of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touchscreen display.

In some embodiments, input device506includes, e.g., one or more joysticks, switches, knobs, slide switches, buttons, dials, keypads, keyboard, mouse, audio transducers (e.g., microphone for voice control system), motion sensor, and/or gesture controls. In some embodiments, an I/O interface of control unit104includes sensors (e.g., GPS sensors, and/or accelerometers), audio output transducers (e.g., speaker), and/or one or more tactile output generators for generating tactile outputs.

In some embodiments, input device506receives user input to control aspects of movable object102, carrier108, payload110, or a component thereof. Such aspects include, e.g., attitude, position, orientation, velocity, acceleration, navigation, and/or tracking. For example, input device506is manually set by a user to one or more positions, each of the positions corresponding to a predetermined input for controlling movable object102. In some embodiments, input device506is manipulated by a user to input control instructions for controlling the navigation of movable object102. In some embodiments, input device506is used to input a flight mode for movable object102, such as auto pilot or navigation according to a predetermined navigation path.

In some embodiments, input device506is used to input a target tracking mode for movable object102, such as a manual tracking mode or an automatic tracking mode. In some embodiments, the user controls movable object102, e.g., the position, attitude, and/or orientation of movable object102, by changing a position of control unit104(e.g., by tilting or otherwise moving control unit104). For example, a change in a position of control unit104is detected by, e.g., one or more inertial sensors and output of the one or more inertial sensors is used to generate command data. In some embodiments, input device506is used to adjust an operational parameter of the payload, such as a parameter of a payload sensing system212(e.g., to adjust a zoom parameter of imaging device214) and/or a position of payload110relative to carrier108and/or movable object102.

In some embodiments, input device506is used to indicate information about target106, e.g., to select a target106to track and/or to indicate target type information412. In some embodiments, input device506is used for interaction with augmented image data. For example, an image displayed by display508includes representations of one or more targets106. In some embodiments, representations of the one or more targets106are augmented to indicate identified objects for potential tracking and/or a target106that is currently being tracked. Augmentation includes, for example, a graphical tracking indicator (e.g., a box) adjacent to or surrounding a respective target106. In some embodiments, input device506is used to select a target106to track or to change from a target106being tracked to a different target for tracking. In some embodiments, a target106is selected when an area corresponding to a representation of target106is selected by e.g., a finger, stylus, mouse, joystick, or other component of input device506. In some embodiments, specific target information412is generated when a user selects a target106to track.

The control unit112may also be configured to allow a user to enter target information using any suitable method. In some embodiments, input device506receives a selection of a target106from one or more images (e.g., video or snapshot) displayed by display508. For example, input device506receives input including a selection performed by a gesture around target106and/or a contact at a location corresponding to target106in an image. In some embodiments, Computer vision or other techniques are used to determine a boundary of a target106. In some embodiments, input received at input device506defines a boundary of target106. In some embodiments, multiple targets are simultaneously selected. In some embodiments, a selected target is displayed with a selection indicator to indicate that the target is selected for tracking. In some other embodiments, input device506receives input indicating information such as color, texture, shape, dimension, and/or other characteristics associated with a target106. For example, input device506includes a keyboard to receive typed input indicating target information408.

In some embodiments, a control unit104provides an interface that enables a user to select (e.g., using input device506) between a manual tracking mode and an automatic tracking mode. When the manual tracking mode is selected, the interface enables the user to select a target106to track. For example, a user is enabled to manually select a representation of a target106from an image displayed by display508of control unit104. Specific target information412associated with the selected target106is transmitted to movable object102, e.g., as initial expected target information.

In some embodiments, when the automatic tracking mode is selected, the user does not provide input selecting a target106to track. In some embodiments, input device506receives target type information410from user input. In some embodiments, movable object102uses the target type information410, e.g., to automatically identify the target106to be tracked and/or to track the identified target106.

Typically, manual tracking requires more user control of the tracking of the target and less automated processing or computation (e.g., image or target recognition) by processor(s)116of movable object102, while automatic tracking requires less user control of the tracking process but more computation performed by processor(s)116of movable object102(e.g., by image analysis module406). In some embodiments, allocation of control over the tracking process between the user and the onboard processing system is adjusted, e.g., depending on factors such as the surroundings of movable object102, motion of movable object102, altitude of movable object102, system configuration400(e.g., user preferences), and/or available computing resources (e.g., CPU or memory) of movable object102, control unit104, and/or computing device126. For example, relatively more control is allocated to the user when movable object is navigating in a relatively complex environment (e.g., with numerous buildings or obstacles or indoor) than when movable object is navigating in a relatively simple environment (e.g., wide open space or outdoor). As another example, more control is allocated to the user when movable object102is at a lower altitude than when movable object102is at a higher altitude. As a further example, more control is allocated to movable object102if movable object is equipped with a high-speed processor adapted to perform complex computations relatively quickly. In some embodiments, the allocation of control over the tracking process between user and movable object102is dynamically adjusted based on one or more of the factors described herein.

In some embodiments, control unit104includes an electronic device (e.g., a portable electronic device) and an input device506that is a peripheral device that is communicatively coupled (e.g., via a wireless and/or wired connection) and/or mechanically coupled to the electronic device. For example, control unit104includes a portable electronic device (e.g., a smartphone) and a remote control device (e.g., a standard remote control with a joystick) coupled to the portable electronic device. In this example, an application executed by the smartphone generates control instructions based on input received at the remote control device.

In some embodiments, the display device508displays information about movable object102, carrier108, and/or payload110, such as position, attitude, orientation, movement characteristics of movable object102, and/or distance between movable object102and another object (e.g., target106and/or an obstacle). In some embodiments, information displayed by display device508includes images captured by imaging device214, tracking data (e.g., a graphical tracking indicator applied to a representation of target106, such as a box or other shape around target106shown to indicate that target106is currently being tracked), and/or indications of control data transmitted to movable object102. In some embodiments, the images including the representation of target106and the graphical tracking indicator are displayed in substantially real-time as the image data and tracking information are received from movable object102and/or as the image data is acquired.

The communication system510enables communication with communication system120of movable object102, communication system610of computing device126, and/or a base station (e.g., computing device126) via a wired or wireless communication connection. In some embodiments, the communication system510transmits control instructions (e.g., navigation control instructions, target information, and/or tracking instructions). In some embodiments, the communication system510receives data (e.g., tracking data from payload imaging device214, and/or data from movable object sensing system122). In some embodiments, control unit104receives tracking data (e.g., via wireless communications124) from movable object102. Tracking data is used by control unit104to, e.g., display target106as the target is being tracked. In some embodiments, data received by control unit104includes raw data (e.g., raw sensing data as acquired by one or more sensors) and/or processed data (e.g., raw data as processed by, e.g., tracking module404).

In some embodiments, memory504stores instructions for generating control instructions automatically and/or based on input received via input device506. The control instructions include, e.g., control instructions for operating movement mechanisms114of movable object102(e.g., to adjust the position, attitude, orientation, and/or movement characteristics of movable object102, such as by providing control instructions to actuators132). In some embodiments, the control instructions adjust movement of movable object102with up to six degrees of freedom. In some embodiments, the control instructions are generated to maintain tracking of a target106(e.g., to correct a detected deviation of target106from expected target information, as described further with regard toFIG.7). In some embodiments, control instructions include instructions for adjusting carrier108(e.g., instructions for adjusting damping element208, actuator204, and/or one or more sensors of carrier sensing system206of carrier108). In some embodiments, control instructions include instructions for adjusting payload110(e.g., instructions for adjusting one or more sensors of payload sensing system212). In some embodiments, control instructions include control instructions for adjusting the operations of one or more sensors of movable object sensing system122.

In some embodiments, input device506receives user input to control one aspect of movable object102(e.g., the zoom of the imaging device214) while a control application generates the control instructions for adjusting another aspect of movable object102(e.g., to control one or more movement characteristics of movable object102). The control application includes, e.g., control module402, tracking module404and/or a control application of control unit104and/or computing device126. For example, input device506receives user input to control one or more movement characteristics of movable object102while the control application generates the control instructions for adjusting a parameter of imaging device214. In this manner, a user is enabled to focus on controlling the navigation of movable object without having to provide input for tracking the target (e.g., tracking is performed automatically by the control application).

In some embodiments, allocation of tracking control between user input received at input device506and the control application varies depending on factors such as, e.g., surroundings of movable object102, motion of movable object102, altitude of movable object102, system configuration (e.g., user preferences), and/or available computing resources (e.g., CPU or memory) of movable object102, control unit104, and/or computing device126. For example, relatively more control is allocated to the user when movable object is navigating in a relatively complex environment (e.g., with numerous buildings or obstacles or indoor) than when movable object is navigating in a relatively simple environment (e.g., wide open space or outdoor). As another example, more control is allocated to the user when movable object102is at a lower altitude than when movable object102is at a higher altitude. As a further example, more control is allocated to movable object102if movable object102is equipped with a high-speed processor adapted to perform complex computations relatively quickly. In some embodiments, the allocation of control over the tracking process between user and movable object is dynamically adjusted based on one or more of the factors described herein.

FIG.6illustrates an exemplary computing device126for controlling movable object102, in accordance with embodiments. Computing device126is, e.g., a server computer, laptop computer, desktop computer, tablet, or phone. Computing device126typically includes one or more processing units602, memory604, communication system610and one or more communication buses612for interconnecting these components. In some embodiments, computing device126includes input/output (I/O) interfaces606, e.g., display614and/or input device616.

In some embodiments, computing device126is a base station that communicates (e.g., wirelessly) with movable object102and/or control unit104.

In some embodiments, computing device126provides data storage, data retrieval, and/or data processing operations, e.g., to reduce the processing power and/or data storage requirements of movable object102and/or control unit104. For example, computing device126is communicatively connected to a database614(e.g., via communication610) and/or computing device126includes database614(e.g., database614is connected to communication bus612).

Communication system610includes one or more network or other communications interfaces. In some embodiments, computing device126receives data from movable object102(e.g., from one or more sensors of movable object sensing system122) and/or control unit104. In some embodiments, computing device126transmits data to movable object102and/or control unit104. For example, computing device126provides control instructions to movable object102.

FIG.7is a flow diagram illustrating a method700for implementing target tracking, in accordance with some embodiments. The method700is performed at a device, such as moving object102, control unit104and/or computing device126. For example, instructions for performing the method700are stored in tracking module404of memory118and executed by processor(s)116.

The device obtains (702) target information408for one or more targets106. For example, target information408is obtained from memory118of movable object102, memory504of control unit104, and/or memory604of computing device126. In some embodiments, target information408obtained by the device is expected target information414.

The device identifies (704) a target106based on the target information408. For example, the device uses an image captured by imaging device214and/or one or more sensors of movable object sensing system122to identify target106. In some embodiments, target106is identified using image recognition or identification techniques (e.g., by image analysis module406).

The device determines (706) initial expected target information of the identified target106. For example, the device determines an initial position of identified target106within an initial image captured by imaging device214and/or one or more sensors of movable object sensing system122and stores the initial position as expected target information414.

The device determines (708) updated target information of the identified target106. For example, the device determines an updated position of target106, e.g., as identified within one or more subsequent images captured after the initial image. In some embodiments, the updated target information of the identified target106is stored as expected target information414(e.g., replacing previous expected target information414for target106).

In some embodiments, the device compares (710) the updated target information determined at operation708with expected target information414(e.g., to determine an extent to which target106has deviated from expected target information414). For example, the device determines a deviation of a position of a representation of target106, e.g., as identified within the one or more subsequent images captured after the initial image, from the position of target106within the initial image.

In some embodiments, a deviation of target106from the expected target information414includes a change in the position of target106. A change in the position of target106is detected by, e.g., comparing coordinates of a representation of target106(e.g., the coordinates of a center point of target106) within an image (e.g., within the one or more subsequent images captured after the initial image) to expected target information414(e.g., coordinates of an expected target position).

In some embodiments, a deviation of target106from the expected target information414includes a change in the size of target106. A change in the size of target106is determined by comparing a size parameter, such as an area (e.g., in pixels) of a representation of target106within an image (e.g., within the one or more subsequent images captured after the initial image) to expected target information414(e.g., the expected area of target106).

The device determines (712) whether deviation of target106from the expected target information414, e.g., as determined at operation710, requires corrective adjustment. In some embodiments, tolerance criteria are applied to determine whether the deviation of the target from the expected target information414requires corrective adjustment. In some embodiments, tolerance criteria are met when updated target information, such as a position of a representation of target106within an image, is within a predetermined minimum number of pixels of expected target information414, such as a position of a representation of target106in a prior image. For example, tolerance criteria are met when a position of a representation of target106within an image deviates from a position of target106in accordance with expected target information414by less than a predefined number of pixels. In some embodiments, the tolerance criteria are met when a size parameter of a representation of target106is above a minimum and/or below a maximum size parameter. In some embodiments, the tolerance criteria are met when a size parameter of a representation of target106deviates from expected target information by less than a predefined amount (e.g., by less than a predefined number of pixels). In some embodiments, tolerance criteria are defined by, e.g., system parameters (e.g., system configuration400), e.g., preset and/or adjustable parameters (e.g. adjustable by a device and/or by a user).

When a deviation of target106from the expected target information414requires corrective adjustment (e.g., the tolerance criteria are not met), flow proceeds to operation714. When deviation of target106from the expected target information414does not require corrective adjustment (e.g., the tolerance criteria are met), flow proceeds to operation708.

The device generates instructions (714) to substantially correct the deviation. In this manner, the device, e.g., substantially corrects the deviation and/or substantially maintains a representation of target106in accordance with expected target information414, for example, to facilitate ongoing tracking of the target106. In some embodiments, substantially correcting the deviation includes an adjustment to an orientation, position, attitude, and/or one or more movement characteristics of movable object102, carrier108, and/or payload110. In some embodiments, instructions to substantially correct the deviation changing a parameter of imaging device214and/or one or more sensors of movable object sensing system122, e.g., changing zoom, focus, or other characteristics associated with imaging device214.

In some embodiments, an adjustment to substantially correct a deviation includes adjusting a zoom level of imaging device214(e.g., if the imaging device supports the zoom level required), by adjustment to one or more movement characteristics of movable object102, or by a combination of adjusting a zoom level of imaging device214and adjustment to one or more movement characteristics of movable object102. In some embodiments, a control application (e.g., control module402, tracking module404and/or a control application of control unit104and/or computing device126) determines one or more adjustments. For example, if the imaging device214does not support a zoom level required to substantially correct a deviation, one or more movement characteristics of movable object102are adjusted instead of or in addition to adjusting the zoom level of imaging device214.

In some embodiments, the adjustment to the orientation, position, attitude, one or more movement characteristics, and/or another operation parameter of movable object102, carrier108, and/or payload110is limited by one or more constraints imposed by system configuration400(e.g., as configured by a manufacturer, administrator, or user), by control unit104(e.g., user control input received at control unit104), and/or by computing device126. Examples of constraints include limits (e.g., maximum and/or minimum limits) for a rotation angle, angular velocity, and/or linear velocity along one or more axes. For example, the angular velocity of movable object102, carrier108, and/or payload110around an axis is constrained by, e.g., a maximum angular velocity that is allowed for movable object102, carrier108, and/or payload110. In some embodiments, the linear velocity of movable object102, carrier108, and/or payload110is constrained by, e.g., a maximum linear velocity that is allowed for movable object102, carrier108, and/or payload110. In some embodiments, adjustment to the focal length of imaging device214is constrained by a maximum and/or minimum focal length for imaging device214.

In some embodiments, in cases where a navigation path of movable object102is predetermined, to the orientation, position, attitude, and/or one or more movement characteristics is implemented by carrier108and/or payload110without affecting the movement of movable object102. The navigation path of movable object102may be predetermined, for example, if a remote user is actively controlling the navigation of movable object via a control unit or if movable object is navigating (e.g., autonomously or semi-autonomously) according to a pre-stored navigation path.

In some embodiments, a warning indicator is provided when an adjustment to the orientation, position, attitude, and/or one or more movement characteristics is limited by a constraint as described above. In some embodiments, a warning indicator includes text, audio (e.g., siren or beeping sound), images or other visual indicators (e.g., changed user interface background color and/or flashing light), and/or haptic feedback. A warning indicator is provided at, e.g., movable object102, control unit104, and/or computing device126.

In some embodiments, the adjustment to the orientation, position, attitude, and/or one or more movement characteristics is performed in substantially real time as movable object102is executing user-provided navigation control instructions or a predetermined flight path.

In some embodiments, the instructions to substantially correct the deviation are generated using information, such as sensing data acquired by one or more sensors of movable object sensing system122(e.g., proximity sensor and/or GPS sensor) and/or position information transmitted by target106(e.g., GPS location).

In some embodiments, determining updated target information for the identified target (708) is performed periodically (e.g., every 0.01 second, 0.1 second, 0.2 second, 0.5 second, or 1 second) and/or in response to a received instruction from movable object102, carrier108, and/or payload110.

FIG.8illustrates an exemplary configuration800of a movable object102, carrier108, and payload110, in accordance with embodiments. The configuration800is used to illustrate exemplary adjustments to an orientation, position, attitude, and/or one or more movement characteristics of movable object102, carrier108, and/or payload110, e.g., as used to track target106.

In some embodiments, movable object102rotates around up to three orthogonal axes, such as X1(pitch)810, Y1(yaw)808and Z1(roll)812axes. Rotations around the three axes are referred to herein as pitch rotation822, yaw rotation820, and roll rotation824, respectively. Angular velocities of movable object102around the X1, Y1, and Z1axes are referred to herein as ωX1, ωY1, and ωZ1, respectively. In some embodiments, movable object102engages in translational movements828,826, and830along the X1, Y1, and Z1axes, respectively. Linear velocities of movable object102along the X1, Y1, and Z1axes are referred to herein as VX1, VY1, and VZ1, respectively.

In some embodiments, payload110is coupled to movable object102via carrier108. In some embodiments, payload110moves relative to movable object102(e.g., payload110is caused by actuator204of carrier108to move relative to movable object102).

In some embodiments, payload110moves around and/or along up to three orthogonal axes, X2(pitch)816, Y2(yaw)814and Z2(roll)818. The X2, Y2, and Z2axes are respectively parallel to the X1, Y1, and Z1axes. In some embodiments, where payload110includes imaging device214(e.g., including an optical module802), the roll axis Z2818is substantially parallel to an optical path or optical axis for optical module802. In some embodiments, optical module802is optically coupled to image sensor216(and/or one or more sensors of movable object sensing system122). In some embodiments, carrier108causes payload110to rotate around up to three orthogonal axes, X2(pitch)816, Y2(yaw)814and Z2(roll)818, e.g., based on control instructions provided to actuator204of carrier108. The rotations around the three axes are referred to herein as the pitch rotation834, yaw rotation832, and roll rotation836, respectively. The angular velocities of payload110around the X2, Y2, and Z2axes are referred to herein as ωX2, ωY2, and ωZ2, respectively. In some embodiments, carrier108causes payload110to engage in translational movements840,838, and842, along the X2, Y2, and Z2 axes, respectively, relative to movable object102. The linear velocity of payload110along the X2, Y2, and Z2 axes is referred to herein as VX2, VY2, and VZ2, respectively.

In some embodiments, the movement of payload110may be restricted (e.g., carrier108restricts movement of payload110, e.g., by constricting movement of actuator204and/or by lacking an actuator capable of causing a particular movement).

In some embodiments, the movement of payload110may be restricted to movement around and/or along a subset of the three axes X2, Y2, and Z2relative to movable object102. For example, payload110is rotatable around X2, Y2, Z2(movements832,834,836) or any combination thereof, payload110is not movable along any of the axes (e.g., carrier108does not permit payload110to engage in movements838,840,842). In some embodiments, payload110is restricted to rotation around one of the X2, Y2, and Z2axes. For example, payload110is only rotatable about the Y2axis (e.g., rotation832). In some embodiments, payload110is restricted to rotation around only two of the X2, Y2, and Z2axes. In some embodiments, payload110is rotatable around all three of the X2, Y2, and Z2axes.

In some embodiments, payload110is restricted to movement along X2, Y2, or Z2axis (movements838,840,842), or any combination thereof, and payload110is not rotatable around any of the axes (e.g., carrier108does not permit payload110to engage in movements832,834,836). In some embodiments, payload110is restricted to movement along only one of the X2, Y2, and Z2axes. For example, movement of payload110is restricted to movement840along the X2axis). In some embodiments, payload110is restricted to movement along only two of the X2, Y2, and Z2axes. In some embodiments, payload110is movable along all three of the X2, Y2, and Z2axes.

In some embodiments, payload110is able to perform both rotational and translational movement relative to movable object102. For example, payload110is able to move along and/or rotate around one, two, or three of the X2, Y2, and Z2axes.

In some embodiments, payload110is coupled to movable object102directly without a carrier108or carrier108does not permit payload110to move relative to movable object102. In some embodiments, the attitude, position and/or orientation of payload110is fixed relative to movable object102in such cases.

In some embodiments, adjustment to attitude, orientation, and/or position of payload110is performed by adjustment to movable object102, carrier108, and/or payload110, such as an adjustment to a combination of two or more of movable object102, carrier108, and/or payload110. For example, a rotation of 60 degrees around a given axis (e.g., yaw axis) for the payload is achieved by a 60-degree rotation by movable object alone, a 60-degree rotation by the payload relative to movable object as effectuated by the carrier, or a combination of 40-degree rotation by movable object and a 20-degree rotation by the payload relative to movable object.

In some embodiments, a translational movement for the payload is achieved via adjustment to movable object102, carrier108, and/or payload110such as an adjustment to a combination of two or more of movable object102, carrier108, and/or payload110. In some embodiments, a desired adjustment is achieved by adjustment to an operational parameter of the payload, such as an adjustment to a zoom level or a focal length of imaging device214.

FIG.9illustrates an exemplary tracking method for maintaining an expected position of a target106, in accordance with embodiments. An exemplary image900is e.g. an image captured by imaging device214. Assume that the image has a width of W pixels and a height of H pixels (where W and H are positive integers). A position within the image is defined by a pair of coordinates along an axis901(along the width of the image) and an axis903(along the height of the image), where the upper left corner of image has coordinates (0, 0) and the lower right corner of the image has coordinates (W, H).

Assume that a representation of target106, as captured in the image900, is located at position P (u, v)902, and the expected position of the target (e.g., as indicated by expected target information414) is P0(u0,v0)904that is different from P902. In some embodiments, the expected position of the target P0(u0,v0) may be near the center of the image, such that u0=W/2, and/or v0=H/2. In some embodiments, the expected position of the target is located at other locations within the image (e.g., off-center). In some embodiments, an expected position of the target may or may not be the same as an initial position of the target (e.g., as determined at706). Assuming that the deviation of current position P from the expected target information414(e.g., the expected position P0) requires corrective adjustment (e.g., as determined at710-712), instructions are generated (e.g., as generated at operation714) for an adjustment to bring the target position from P to close to the expected position P0. In some embodiments, a deviation is expressed as a Δx from u0, and a Δy from v0.

In some embodiments, the deviation from the expected target position is used to derive one or more angular velocities for rotating the field of view of the imaging device around one or more axes. For example, deviation along the axis901of the image (e.g., between u and u0) is used to determine an angular velocity ωY910for rotating the field of view of the imaging device214around the Y (yaw) axis906, as follows:
ωY=α*(u−u0), where α∈(real numbers)  (1)

In some embodiments, the rotation around the Y axis for the field of view of imaging device214is achieved by a rotation of movable object102, a rotation of payload110(e.g., via carrier108) relative to movable object102, or a combination of both. In some embodiments, payload110is adjusted when adjustment of movable object102is infeasible or otherwise undesirable, for example, when the navigation path of movable object is predetermined. In the equation (1), a is a constant that may be predefined and/or calibrated based on the configuration of the movable object (e.g., when the rotation is achieved by the movable object), the configuration of the carrier (e.g., when the rotation is achieved by the carrier), or both (e.g., when the rotation is achieved by a combination of the movable object and the carrier). In some embodiments, a is greater than zero (α>0). In other embodiments, a may be no greater than zero (α≤0). In some embodiments, α can be used to map a calculated pixel value to a corresponding control lever amount or sensitivity for controlling the angular velocity around a certain axis (e.g., yaw axis). In general, the control lever may be used to control the angular or linear movement of a controllable object (e.g., movable object102or carrier108). A greater control lever amount corresponds to greater sensitivity and greater speed (for angular or linear movement). In some embodiments, the control lever amount or a range thereof is determined by configuration parameters of the flight control system (e.g. stored by system configuration400and/or motion control module402) for a movable object102or configuration parameters of a control system for a carrier108. The upper and lower bounds of the range of the control lever amount may include any arbitrary numbers. For example, the range of the control lever amount may be (1000, −1000) for one flight control system and (−1000, 1000) for another flight control system.

For instance, assume that the images have a width of W=1024 pixels and a height of H=768 pixels. Thus, the size of the images is 1024*768. Further assume that the expected position of the target has a u0=512. Thus, (u−u0)∈(−512, 512). Assume that the range of the control lever amount around the yaw axis is (−1000, 1000), then the maximum control lever amount or maximum sensitivity is 1000 and α=1000/512. Thus, the value of a can be affected by image resolution or size provided by the imaging device, range of the control lever amount (e.g., around a certain rotation axis), the maximum control lever amount or maximum sensitivity, and/or other factors.

For instance, when the rotation is achieved by rotation of movable object102, the Y axis906ofFIG.9corresponds to the Y1axis808for the movable object as illustrated inFIG.8and the overall angular velocity of the field of view ωYis expressed as the angular velocity ωY1for the movable object:
ωY=ωY1=α1*(u−u0), where α1∈(2)

In the equation (2), α1is a constant that is defined based on the configuration of the movable object. In some embodiments, α1is greater than zero (α1>0). The α1can be defined similar to the α discussed above. For example, the value of α1may be defined based on image resolution or size and/or range of control lever amount for the movable object (e.g., around the yaw axis).

Similarly, when the rotation is achieved by the rotation of payload110relative to movable object102(e.g., via carrier108), the Y axis906ofFIG.9corresponds to the Y2axis814for the payload as illustrated inFIG.8and the overall angular velocity of the field of view coy is expressed as the angular velocity ωY2for the payload relative to the movable object:
ωY=ωY2=α2*(u−u0), where α2∈(3)

In the equation (3), α2is a constant that is defined based on the configuration of the carrier and/or payload. In some embodiments, α2is greater than zero (α2>0). The α2can be defined similar to the α discussed above. For example, the value of α2may be defined based on image resolution or size and/or range of control lever amount for the carrier108(e.g., around the yaw axis).

In general, the angular velocity of the field of view around the Y (yaw) axis906can be expressed as a combination of the angular velocity ωY1for the movable object and the angular velocity ωY2for the payload relative to the movable object, such as the following:
ωY=ωY1+ωY2(4)

In the equation (4), either ωY1or ωY2may be zero.

As illustrated herein, the direction of the rotation around the Y (yaw) axis may depend on the sign of u−u0. For instance, if the expected position is located to the right of the actual position (as illustrated inFIG.9), then u−u0<0, and the field of view needs to rotate in a counter-clockwise fashion around the yaw axis906(e.g., pan left) in order to bring the target to the expected position. On the other hand, if the expected position is located to the left of the actual position, then u−u0>0, and the field of view needs to rotate in a clockwise fashion around the yaw axis906(e.g., pan right) in order to bring the target to the expected position.

As illustrated herein, the velocity of rotation (e.g., absolute value of the angular velocity) around a given axis (e.g., the Y (yaw) axis) may depend on the distance between the expected and the actual position of the target along the axis (i.e., |u−u0|). The further the distance is, the greater the velocity of rotation. Likewise, the closer the distance is, the slower the velocity of rotation. When the expected position coincides with the position of the target along the axis (e.g., u=u0), then the velocity of rotation around the axis is zero and the rotation stops.

The method for adjusting the deviation from the expected target position and the actual target position along the axis901, as discussed above, can be applied in a similar fashion to correct the deviation of the target along a different axis903. For example, deviation along the axis903of the image (e.g., between v and v0) may be used to derive an angular velocity ωX914for the field of view of the imaging device around the X (pitch) axis908, as follows:
ωX=β*(v−v0) where β∈(5)

The rotation around the X axis for the field of view of an imaging device may be achieved by a rotation of the movable object, a rotation of the payload110(e.g. via carrier108) relative to the movable object102, or a combination of both. Hence, in the equation (5), β is a constant that may be predefined and/or calibrated based on the configuration of the movable object (e.g., when the rotation is achieved by the movable object), the configuration of the carrier (e.g., when the rotation is achieved by the carrier), or both (e.g., when the rotation is achieved by a combination of the movable object and the carrier). In some embodiments, β is greater than zero (β>0). In other embodiments, β may be no greater than zero (β≤0). In some embodiments, β can be used to map a calculated pixel value to a corresponding control lever amount for controlling the angular velocity around a certain axis (e.g., pitch axis). In general, the control lever may be used to control the angular or linear movement of a controllable object (e.g., movable object102or carrier108). Greater control lever amount corresponds to greater sensitivity and greater speed (for angular or linear movement). In some embodiments, the control lever amount or a range thereof may be determined by configuration parameters of the flight control system for a movable object102or configuration parameters of a carrier control system for a carrier108. The upper and lower bounds of the range of the control lever amount may include any arbitrary numbers. For example, the range of the control lever amount may be (1000, −1000) for one control system (e.g., flight control system or carrier control system) and (−1000, 1000) for another control system.

For instance, assume that the images have a width of W=1024 pixels and a height of H=768 pixels. Thus, the size of the images is 1024*768. Further assume that the expected position of the target has a v0=384. Thus, (v−v0)∈(−384, 384). Assume that the range of the control lever amount around the pitch axis is (−1000, 1000), then the maximum control lever amount or maximum sensitivity is 1000 and β=1000/384. Thus, the value of β can be affected by image resolution or size provided by the imaging device, range of the control lever amount (e.g., around a certain rotation axis), the maximum control lever amount or maximum sensitivity, and/or other factors.

For instance, when the rotation is achieved by rotation of the movable object, the X axis908ofFIG.9corresponds to the X1axis810for the movable object as illustrated inFIG.8and the angular velocity of the field of view ωXis expressed as the angular velocity ωX1for the movable object:
wX=ωX1=β1*(v−v0), where β1∈(6)

In the equation (6), β1is a constant that is defined based on the configuration of the movable object. In some embodiments, β1is greater than zero (β1>0). The β1can be defined similar to the β discussed above. For example, the value of β1may be defined based on image resolution or size and/or range of control lever amount for the movable object (e.g., around the pitch axis).

Similarly, when the rotation is achieved by the rotation of the payload relative to the movable object (e.g., via the carrier), the X axis908ofFIG.9corresponds to the X2axis816for the payload as illustrated inFIG.8and the angular velocity of the field of view ωXis expressed as the angular velocity ωX2for the payload relative to the movable object:
ωX=ωX2=β2*(v−v0), where β2∈(6)

In the equation (6), β2is a constant that is defined based on the configuration of the carrier and/or payload. In some embodiments, β2is greater than zero (β2>0). The β2can be defined similar to the β discussed above. For example, the value of β2may be defined based on image resolution or size and/or range of control lever amount for the movable object (e.g., around the pitch axis).

In general, the angular velocity of the field of view around the X (pitch) axis608can be expressed as a combination of the angular velocity ωX1for the movable object and the angular velocity ωX2for the payload relative to the movable object, such as the following:
ωX=ωX1+ωX2(7)

In the equation (7), either ωX1or ωX2may be zero.

As illustrated herein, the direction of the rotation around the X (yaw) axis may depend on the sign of v−v0. For instance, if the expected position is located above of the actual position (as illustrated inFIG.9), then v−v0>0, and the field of view needs to rotate in a clockwise fashion around the pitch axis908(e.g., pitch down) in order to bring the target to the expected position. On the other hand, if the expected position is located to below the actual position, then v−v0<0, and the field of view needs to rotate in a counter-clockwise fashion around the pitch axis608(e.g., pitch up) in order to bring the target to the expected position.

As illustrated herein, the speed of rotation (e.g., absolute value of the angular velocity) depends on the distance between the expected and the actual position of the target (i.e., |v−v0|) along a give axis (e.g., the X (pitch) axis). The further the distance is, the greater the speed of rotation. The closer the distance is, the slower the speed of rotation. When the expected position coincides with the position of the target (e.g., v=v0), then the speed of rotation is zero and the rotation stops.

In some embodiments, the values of the angular velocities as calculated above may be constrained or otherwise modified by various constraints of the system. Such constraints may include the maximum and/or minimum speed that may be achieved by the movable object and/or the imaging device, the range of control lever amount or the maximum control lever amount or maximum sensitivity of the control system for the movable object and/or the carrier, and the like. For example, the rotation speed may be the minimum of the calculated rotation speed and the maximum speed allowed.

In some embodiments, warning indicators are provided (e.g., displayed by display508or otherwise output by control unit104when the calculated angular velocities need to be modified according to the constraints described herein. Examples of such warning indicators may include textual, audio (e.g., siren or beeping sound), visual (e.g., certain color of light or flashing light), mechanical, any other suitable types of signals. Such warning indicators are provided, e.g., directly by the movable object102, carrier108, payload110, or a component thereof. In some embodiments, warning indicators are provided by the control unit104(e.g., via the display508). In some embodiments, control unit104provides warning indicators based on instructions received from movable object102.

FIG.10illustrates an exemplary tracking method for maintaining an expected size of a target106, in accordance with embodiments. An exemplary image1000is, e.g., an image captured by an imaging device214carried by movable object102. Image1000includes a representation1002of target106. In some embodiments, the current size s of a representation1002of target106within the image1000is indicated in pixels (such as calculated as the product of the width of representation1002and the height of representation1002). In some embodiments, the expected target size (e.g., as indicated by expected target information414) is smaller (e.g., the expected target may be represented by1004and S=s0) or larger (e.g., the expected target may be represented by1005and S=s1) than the current size s. In some embodiments, the expected target size is a range extending from s0to s1. The expected target size may or may not be the same as an initial size of the target (e.g., as indicated in specific target information412, such as specific target information412provided by control unit104to movable object102). In some embodiments, when a deviation of a current target size s (e.g., an area of a representation of target106within image1000) from the expected target size so or s1(or an expected target size that ranges from so to s1) requires corrective adjustment (e.g., as determined at operation710-712), instructions are generated (e.g., as generated at operation714) for an adjustment, e.g., to reduce the deviation of the target size from the expected size.

Although display area of the image1000and representation1002of target106are shown as rectangles, this is for illustrative purposes only and is not intended to be limiting. In some embodiments, an expected size of the target (e.g. stored as expected target information414) is indicated using representations such as a line (e.g., a radius or other dimension), circle, oval, polygon, sphere, rectangular prism, and/or polyhedron. Likewise, although the expected target size is expressed in pixels, this is for illustrative purposes only and not intended to be limiting. In some embodiments, the expected size of the target (e.g. stored as expected target information414) is expressed as, e.g., a length (e.g., mm or other length unit), an area (e.g., mm2or other area unit), a ratio of a length of a representation of the target in an image relative to a total image length (e.g., a percentage), a ratio of an area of a representation of the target in an image relative to a total image area (e.g., a percentage), a number of pixels in a line (e.g., corresponding to a diameter, length, and/or width of target106), and/or a number of pixels in an area.

In some embodiments, the deviation of target106from expected target information414(e.g., the expected target size) is used to derive one or more linear velocities for movable object102and/or payload110along one or more axes. For example, deviation in the target size between a current target size s and the expected target size S (e.g., S=so or s1) can be used to determine a linear velocity V for moving movable object along a Z (roll) axis1010, as follows:
V=δ(1−s/S), where δ∈(8)

In the equation (8), δ is a constant that is defined based on the configuration of movable object or any suitable controllable object (e.g., carrier) that may cause the field of view to move toward and/or away from the target. In some embodiments, δ is greater than zero (δ>0).

In other embodiments, δ may be no greater than zero (δ≤0). In some embodiments, δ can be used to map a calculated pixel value to a corresponding control lever amount or sensitivity for controlling the linear velocity.

In general, V represents the velocity of movable object102toward or away from the target106. The velocity vector points from the movable object102to the target106. If the current size s of the representation1002of target106is smaller than the expected size S, then V>0 and movable object moves towards the target so as to increase the size of the target as captured in the images. On the other hand, if the current size s of the target is larger than the expected size S, then V<0 and movable object moves away from the target so as to reduce the size of the target as captured in the images.

For instance, assume that the images have a width of W=1024 pixels and a height of H=768 pixels. Thus, the size of the images is 1024*768. Assume that the range of the control lever amount for controlling the linear velocity is (−1000, 1000). In an exemplary embodiment, δ=−1000 when s/S=3 and δ=1000 when s/S=1/3.

In some embodiments, the values of the velocities as calculated above are constrained or otherwise modified by various constraints of the system. Such constraints include, e.g., the maximum and/or minimum speed that may be achieved by movable object and/or the imaging device, the maximum sensitivity of the control system for movable object and/or the carrier, and the like. For example, the speed for movable object may be the minimum of the calculated speed and the maximum speed allowed.

Alternatively or additionally, the deviation between the actual target size and the expected target size can be used to derive adjustment to the operational parameters of the imaging device such as a zoom level or focal length in order to correct the deviation. Such adjustment to the imaging device may be necessary when adjustment to movable object is infeasible or otherwise undesirable, for example, when the navigation path of movable object is predetermined. An exemplary focal length adjustment F can be expressed as:
F=γ(1−s/S), where γ∈(9)

Where γ is a constant that is defined based on the configuration of the imaging device. In some embodiments, γ is greater than zero (γ>0). In other embodiments, γ is no greater than zero (γ≤0). The value of γ may be defined based on the types of lenses and/or imaging devices.

If the current size s of the representation1002of target106is smaller than the expected size S, then F>0 and the focal length increases by |F| so as to increase the size of the target as captured in the images. On the other hand, if the actual size s of the target is larger than the expected size S, then F<0 and the focal length decreases by |F| so as to reduce the current size s of the target106as captured in the images. For example, in an embodiment, γ=10. This means that, for example, when the actual size of the target is double the size of the expected size S, the focal length should be decreased by 10 mm accordingly (i.e., F=10*(1−2/1)=−10) and vice versa.

In some embodiments, the adjustment to the operational parameters of the imaging device such as focal length may be constrained or otherwise modified by various constraints of the system. Such constraints may include, for example, the maximum and/or minimum focal lengths that may be achieved by the imaging device214. As an example, assume the focal length range is (20 mm, 58 mm). Further assume that the initial focal length is 40 mm. Then when s>S, the focal length should be decreased according to equation (9); and when s<S, the focal length should be increased according to equation (9). However, such adjustment is limited by the lower and upper bounds of the focal length range (e.g., 20 mm to 58 mm). In other words, the post-adjustment focal length should be no less than the minimum focal length (e.g., 20 mm) and no more than the maximum focal length (e.g., 58 mm).

As discussed above inFIG.9, in some embodiments, warning indicators are provided (e.g., at control unit104) when the calculated adjustment (e.g., linear velocity of movable object or focal length) is modified according to the constraints described herein.

FIG.11illustrates an exemplary process1100for implementing target tracking, in accordance with some embodiments. The method1100is performed at a device, such as moving object102, control unit104and/or computing device126. For example, instructions for performing the method1100are stored in tracking module404of memory118and executed by processor(s)116of movable object102.

The device obtains (1102) user control instructions such as navigation control instructions, for example, from control unit104and/or computing device126. In some embodiments, the navigation control instructions are used for controlling navigational parameters of movable object102such as the position, speed, orientation, attitude, or one or more movement characteristics of movable object102. In some cases, the navigation control instructions include instructions for movable object102to execute pre-stored navigation control instructions (e.g., stored by motion control module402) such as control instructions corresponding to a predetermined navigation path. The navigation control instructions are used, e.g., to control movable object to navigate according to a user-specified or pre-stored navigation path.

The device obtains (1104) target information408, for example, from control unit104and/or computing device126. In some embodiments, some or all of target information408is obtained from memory118(e.g., in lieu of receiving targeting information from control unit104and/or computing device126). In some embodiments, some or all of target information408is obtained from memory504, memory604, and/or database614. The target information408includes, e.g., specific target information412, target type information410, and/or expected target information414.

In some embodiments, target information408is generated at least in part based on input received at input device506of control unit104. In some embodiments, target information408is generated at least in part using data from memory118, memory504, memory604, and/or database614. For example, target type information410is derived based on e.g., stored images (e.g., images previously captured by imaging device214).

In some embodiments, the device generates instructions (1106) for adjusting an orientation, position, attitude, and/or one or more movement characteristics of movable object102, in response to the navigation control instructions obtained at operation1102. In some embodiments, the generated instructions are used for navigation of movable object102according to a user-specified and/or pre-stored navigation path.

In some embodiments, the device generates instructions (1108) for adjusting an orientation, position, attitude, and/or one or more movement characteristics of movable object102, carrier108, and/or payload110to track target106according to the target information408(e.g., using operations discussed with regard toFIG.7).

FIG.12illustrates an exemplary user interface1200for selecting and/or tracking a target106, in accordance with some embodiments. In some embodiments, user interface1200is displayed by a control unit104and/or a computing device126. In some embodiments, the user interface is displayed by display508of control terminal104. The user interface includes one or more objects, such as objects1202,1204, and1206. In some embodiments, one or more of objects1202,1204,1206is a representation of a target106. In some embodiments, user interface1200displays an image captured by imaging device214and the image includes the one or more objects1202,1204,1206. In some embodiments, graphical tracking indicator1208is displayed in user interface1200, e.g., adjacent to or surrounding a tracked target106(e.g., object1202). In some embodiments, the position of graphical tracking indicator1208changes as the position of object1202changes, e.g., such that graphical tracking indicator1208remains associated with (e.g., adjacent to or surrounding) object1202while object1202is a tracked target106.

In some embodiments, control unit104includes one or more input devices506for receiving user input. In some embodiments, input received by input devices506is used to provide input indicating an object1202,1204,1206with which graphical tracking indicator1208is to be associated. In this way, a user indicates a target106to be tracked, in accordance with some embodiments. In some embodiments, target information408is generated based on received input associating graphical tracking indicator1208with the object1202(e.g., to designate object1202as target106). In some embodiments, user input received at input device506to associate a graphical tracking indicator1208with an object1202includes an input gesture received at a point that corresponds to an object (e.g.,1202). In some embodiments, an input gesture is provided by a contact (e.g., by a finger and/or stylus) at display508(e.g., a touchscreen display). In some embodiments, a selection of an object1202is provided by user-manipulated input device506(such as a mouse, button, joystick, keyboard, etc.).

FIG.13illustrates controlling a movable object102to avoid an obstacle, in accordance with some embodiments.

Movable object102moves along a path1302. In some embodiments, path1302is a predetermined navigation path and/or a path along which movable object102moves in response to navigation control instructions (e.g., navigation control instructions received from control unit104and/or computing device126). In some embodiments, the path1302is determined at least in part in response to instructions generated for tracking target106(e.g., instructions generated as described with respect toFIG.7).

Movable object102moves along path1302from a first position1304at an initial time to, to a subsequent position1306at a second time t1that is later than to, and so on to positions1308,1310at times t2, t3, respectively. Movable object102is depicted with broken lines herein to indicate a prior location of movable object102at a time prior to a current time (at which movable object102is shown with solid lines) or a later location of movable object102at a time after a current time.

An obstacle1316is located on path1302, such that movable object102would eventually collide with obstacle1316if the movable object102continued along path1302after time t3. In some embodiments, obstacle1316is a substantially static object, such as a manmade and/or natural structure, e.g., a traffic sign, radio tower, building, bridge, or geological feature. In some embodiments, obstacle1316is a dynamic object, such as a vehicle, tree, human, animal, or another movable object (e.g., a UAV).

In some embodiments, movable object102is diverted from path1302to an alternate path1318, e.g., such that movable object does avoids collision with obstacle1316. For example, at time t5, movable object102has moved along alternate path1318, e.g., to avoid collision with obstacle1316and/or to maintain a predetermined distance from obstacle1316.

In some embodiments, different approaches (e.g., a “reactive” approach or a “proactive approach”) are taken to controlling movable object102to avoid collision depending on a distance between movable object102and obstacle1316. In some embodiments, a threshold distance used to determine whether a reactive approach or a proactive approach will be used is referred to as a “reactive region.” A reactive region is typically defined relative to movable object102. In some embodiments, when obstacle1316is beyond a reactive region of movable object102(e.g., obstacle1316is located at a relatively large distance from movable object102, such as a distance exceeding 10 meters), one or more movement characteristics of movable object102are adjusted in a proactive manner, as described further below with reference toFIGS.14and19-22.

FIG.14illustrates adjusting a movement characteristic of movable object102in a proactive manner, in accordance with some embodiments. As movable object102tracks target106, a navigation path1402is determined for movable object102(e.g., instructions are generated for movable object102, such as by target tracking process700). In some embodiments, in response to detecting obstacle1316, one or more movement characteristics of movable object102are adjusted, e.g., such that movable object moves along alternate path1404. For example, one or more movement characteristics of movable object102are adjusted in a proactive manner. In some embodiments, adjusting one or more movement characteristics of movable object102in a proactive manner includes adjusting one or more movement characteristics of movable object102such that a distance between movable object102and the obstacle exceeds a predefined distance1406(e.g., the distance between movable object102and obstacle1316is maintained at and/or or beyond a predefined distance1406as movable object102moves relative to obstacle1316. Predefined distance1406is e.g., a distance between 5 and 20 meters, such as 10 meters.

In some embodiments, after movable object102moves along alternate path1404, movable object102resumes movement along a navigation path1408for tracking target106. In some embodiments, movable object102tracks target106continuously as one or more movement characteristics of movable object102are adjusted in a proactive manner. In some embodiments, movable object102ceases to track target106when one or more movement characteristics of movable object102are adjusted in a proactive manner and/or when obstacle1316is detected. In some embodiments, after ceasing to track target106, movable object102resumes tracking of target106when obstacle1316is avoided (for example, when, motion of movable object102is along a vector that points away from obstacle1316) and/or when obstacle1316is no longer detected.

FIG.15illustrates adjusting a movement characteristic of movable object102in a reactive manner, in accordance with some embodiments. For example, a movement characteristic of movable object102is adjusted in a reactive manner, e.g., such that a collision between obstacle1316and movable object102is avoided. As movable object102tracks target106, a navigation path1502is determined for movable object102(e.g., instructions are generated for movable object102, such as by target tracking process700). In some embodiments, in response to detecting obstacle1316, and in response to determining that the location of the obstacle1316corresponds to a reactive region (e.g., obstacle1316is located within 10 meters of movable object102), one or more movement characteristics of movable object102are adjusted, e.g., such that movable object102ceases to move or moves along reverse path1504. For example, one or more movement characteristics of movable object102are adjusted in a reactive manner. Adjusting one or more movement characteristics of movable object102in a reactive manner includes adjusting one or more movement characteristics to reduce the acceleration of movable object102, reduce the velocity of movable object102, cease the motion of movable object102, and/or to reverse the motion of movable object102. For example, one or more movement characteristics of movable object102are adjusted such that movable object102moves along an alternate path1504(e.g., in a direction that increases distance between movable object102and obstacle1316, such as a direction that is opposite to navigation path1502).

FIG.16illustrates a reactive region1602, in accordance with some embodiments. Reactive region1602is a region in which one or more movement characteristics of movable object102are adjusted in a reactive manner, e.g., to avoid collision of movable object102with obstacle1316. Typically, reactive region1602is a region defined relative to movable object102, such a region centered on movable object102and/or surrounding movable object102. For example, reactive region1602is, e.g., a circle or a sphere (e.g., centered on movable object102and/or a defined point relative to movable object102, such as a center of mass of movable object102). In some embodiments, reactive region1602extends from movable object102in a direction of movement of a movable object102. For example, reactive region1602is, e.g., a cone or triangle (e.g., with a vertex on movable object102and/or a defined point relative to movable object102, such as a center of mass of movable object102). In some embodiments, reactive region1602is defined by a radius1604. In some embodiments, the length of radius1604, is, e.g., a distance between 5 and 20 meters, such as 10 meters. In some embodiments, reactive region1602is a distance from movable object102along an axis that is a line defined by movable object102(and/or a defined point relative to movable object102) and obstacle1316(e.g. a point determined from and/or indicated by current location information for obstacle1316). In some embodiments, reactive region1602is a predefined distance from movable object102along path1302.

Obstacle1316is discovered, e.g., in response to a periodic scan, in response to a user- or device-initiated scan, based on navigation information, or in response to a determination that obstacle intersects a path1302of movable object102. In some embodiments, obstacle1316is discovered using a stored depth map and/or a depth map generated in real time (using one or more sensors of movable object sensing system122). Obstacle1316is discovered by movable object102, control unit104, computing device126, and/or based on received user input indicating the presence of an obstacle. In some embodiments, movable object102, control unit104, and/or computing device126determines, based on current location information for obstacle1316, whether a location of obstacle1316corresponds to a reactive region1602. In some embodiments, detecting an obstacle includes obtaining current location information of an obstacle1316. In some embodiments, current location information is obtained for an obstacle1316in response to detection of the obstacle.

Obstacle1316-A is presented as an example of an obstacle1316that does not correspond to reactive region1602(e.g., because obstacle1316-A is outside of reactive region1602). For example, a point, dimension, outline, area, and/or volume of obstacle1316-A (e.g., as determined by image analysis module406) or a point defined with respect to obstacle1316-A (e.g., a centroid of obstacle1316-A) is partially (e.g., at least 90%) or fully outside of and/or beyond reactive region1602. In some embodiments, in response to a determination that the location of obstacle1316-A does not correspond to the reactive region1602, one or more movement characteristics of movable object102are adjusted in a proactive manner (e.g., such that a distance between the movable object102and the obstacle1316-A exceeds a first predefined distance1406). In some embodiments, first predefined distance1406is greater than or equal to the length of radius1604.

Obstacle1316-B is presented as an example of an obstacle1316that corresponds to reactive region1602. Obstacle1316-B is shown within (e.g., at least partially within and/or overlapping) reactive region1602. For example, a point, dimension, outline, area, and/or volume of obstacle1316-B (e.g., as determined by image analysis module406) or a point defined with respect to obstacle1316-B (e.g., a centroid of obstacle1316-B) is within (e.g., at least partially within, such as at least 10% within, and/or overlapping) reactive region1602. In some embodiments, in response to a determination that the location of obstacle1316-B corresponds to the reactive region1602, one or more movement characteristics of movable object102are adjusted in a reactive manner (e.g., such that a collision between movable object102and the obstacle1316-B is avoided).

FIG.17illustrates sub-regions of reactive region1602, in accordance with some embodiments. In some embodiments, reactive region1602includes two or more sub-regions. For example, reactive region1602as shown inFIG.17includes a first sub-region1702, a second sub-region1704, and a third sub-region1706.

First sub-region1702of reactive region1602is, e.g., a region defined by first boundary1714(e.g., a sphere with a radius as indicated at1708) and second boundary1708(e.g., a sphere with a radius as indicated at1710), such as a volume between first boundary1714and second boundary1716.

Second sub-region1704of reactive region1602is, e.g., a region defined by second boundary1716(e.g., a sphere with a radius as indicated at1710), and third boundary1718(e.g., a sphere with a radius as indicated at1712), such as a volume between second boundary1716and third boundary1718.

Third sub-region1706of reactive region1602is, e.g., a region defined by third boundary1718, such as a spherical volume inside boundary1718.

In some embodiments, the lengths of radii1712,1710, and1708are, e.g., 2 meters, 5 meters, and 10 meters, respectively.

In some embodiments, one or more of first sub-region1702, second sub-region1704, and third sub-region1706is defined by a distance from movable object102along an axis that is a line from movable object102to obstacle1316. In some embodiments, one or more of first sub-region1702, second sub-region1704, and third sub-region1706is defined by one or more circles within a particular plane, and/or other geometric shapes, volumetric shapes, and or irregular shapes.

In some embodiments, when a determined location of an obstacle1316corresponds to first sub-region1702, one or more movement characteristics of the movable object102are adjusted to, e.g., reduce an acceleration of the movable object102(such as acceleration in the direction of obstacle1316-C). InFIG.17, obstacle1316-C corresponds to first sub-region1702because obstacle1316-C is located within (e.g., at least partially within) first sub-region1702. For example, a location of obstacle1316-C corresponds to first sub-region1702when a point, dimension, outline, area, and/or volume of obstacle1316-C (e.g., as determined by image analysis module406) or a point defined with respect to obstacle1316-C (e.g., a centroid of obstacle1316-C) is partially (e.g., at least 10%) within first sub-region1702.

In some embodiments, when a determined location of an obstacle1316corresponds to second sub-region1704, one or more movement characteristics of the movable object102are adjusted to, e.g., reduce a velocity of the movable object102(such as velocity in the direction of obstacle1316-D). InFIG.17, obstacle1316-D corresponds to second sub-region1704(e.g., obstacle1316-D is located within (e.g., at least partially within) second sub-region1702). For example, a location of obstacle1316-D corresponds to second sub-region1704when a point, dimension, outline, area, and/or volume of obstacle1316-D (e.g., as determined by image analysis module406) or a point defined with respect to obstacle1316-D (e.g., a centroid of obstacle1316-D) is partially (e.g., at least 10%) within second sub-region1704.

In some embodiments, when a determined location of an obstacle1316corresponds to a third sub-region1706, one or more movement characteristics of the movable object102are adjusted to, e.g., reverse movement direction of the movable object102and/or cease movement of the movable object102. For example movement of movable object102toward obstacle1316-E is reversed such that movable object102ceases to move toward obstacle1316-E and begins to move away from obstacle1316-E. InFIG.17, obstacle1316-E corresponds to third sub-region1706because obstacle1316-E is located within (e.g., at least partially within) third sub-region1706. For example, a location of obstacle1316-E corresponds to third sub-region1706when a point, dimension, outline, area, and/or volume of obstacle1316-E (e.g., as determined by image analysis module406) or a point defined with respect to obstacle1316-E (e.g., a centroid of obstacle1316-E) is partially (e.g., at least 10%) within third sub-region1706.

FIGS.18A-18Billustrate exemplary adjustments made to user interface1200in response to received adjusted target tracking information, in accordance with some embodiments.

InFIG.18A, display508displays a first state1800of a user interface1200for selecting and/or tracking a target106, e.g., as described with reference toFIG.12. For example, in the first state1800, selection and/or ongoing tracking of target106is indicated by graphical tracking indicator1208.

InFIG.18B, display508displays a second state1850of user interface1200, e.g., a second state1850of the user interface1200presented in response to received adjusted target tracking information. In some embodiments, adjusted target tracking information is received by control unit104from computing device126and/or movable object102. For example, updated target tracking is generated in response to determining that the location of obstacle1316corresponds to the reactive region1602, first sub-region1702, second sub-region1704and/or third sub-region1706. In some embodiments, adjusted target tracking information is generated in response to adjusting one or more movement characteristics of the movable object102in a reactive manner.

In some embodiments, in response to receiving adjusted target tracking information (e.g., when one or more movement characteristics of the movable object102are updated in a reactive manner) at least one aspect of the appearance of graphical tracking indicator1208is changed. For example, a color of an outline of graphical tracking indicator1208is changed, part or all of the area of graphical tracking indicator1208becomes shaded, a color of shading of part or all of graphical tracking indicator1208is changed, and/or a size of graphical tracking indicator1208is changed. In some embodiments, graphical tracking indicator overlaps (e.g., partially overlaps) obstacle1316and/or target106. In some embodiments a shading of at least part of graphical tracking indicator1208is transparent (e.g., partially transparent) such that obstacle1316and/or target106is visible through the shading of graphical tracking indicator1208. In some embodiments, in response to receiving adjusted target tracking information, graphical tracking indicator1208ceases to be displayed. Changing at least one aspect of the appearance of graphical tracking indicator1208and/or ceasing to display graphical tracking indicator1208occurs, e.g., to indicate that tracking of target106has been disabled (e.g., temporarily disabled).

In some embodiments, in response to receiving adjusted target tracking information (e.g., when one or more movement characteristics of the movable object102are adjusted in a reactive manner) a user is prevented from selecting a target106.

In some embodiments, in response to receiving adjusted target tracking information (e.g., when one or more movement characteristics of the movable object102are adjusted in a reactive manner), warning indicator1852is presented. In some embodiments, warning indicator1852includes text, images or other visual indicators (e.g., changed user interface background color and/or flashing light) displayed by display508, audio (e.g., siren or beeping sound) output by control unit104, and/or haptic feedback output by control unit104.

FIG.19illustrates a frame of reference used for adjusting one or more movement characteristics of the movable object102in a proactive manner, in accordance with some embodiments. In some embodiments, a frame of reference1900is defined relative to a point corresponding to movable object102, such as a center of gravity or central point of a dimension (e.g., length, width, or height) an area, or a volume movable object102. In some embodiments, movable object102as shown inFIG.19is moving along a path1302as shown inFIG.13. Frame of reference1900includes velocity vector components VX, VY, VZalong an x-axis, y-axis, and z-axis respectively. In some embodiments, VX1902is oriented along a vector of movement of movable object102(e.g., a vector pointing along path1302, such as a vector along an axis that is a line defined by a point corresponding to movable object102and a point corresponding to target106). In some embodiments, VY1904and VZ1906are orthogonal to VX1902. Angular velocity ωZ1908(yaw) indicates a velocity of rotation about a z-axis. In some embodiments, adjusting one or more movement characteristics of the movable object in a proactive manner includes adjusting movement characteristics along axes VY1904and axis VZ1906(e.g., preferentially adjusting movement characteristics along axes VY1904and axis VZ1906), for example, to allow movable object102to maintain a predetermined distance from obstacle1316while maintaining and/or minimally adjusting angular velocity ωZ1908and/or movement along VX1902.

FIG.20illustrates sets of candidate movement characteristics for determining a (VY,VZ) motion adjustment, in accordance with some embodiments. In some embodiments, in response to determining that the location of the obstacle does not correspond to reactive region1602, one or more movement characteristics of the movable object102are adjusted in a proactive manner (e.g., such that a distance between the movable object102and the obstacle1316exceeds a first predefined distance). In some embodiments, adjusting one or more movement characteristics of the movable object102in a proactive manner includes determining route optimization scores for multiple sets of candidate movement characteristics. A set of candidate movement characteristics is, e.g., a set of (VY,VZ) coordinates (e.g., set2002) in frame of reference1900. Plot2000includes multiple sets (e.g., sets2002,2004,2006, and so on) of candidate movement characteristics. In some embodiments, a route optimization score is determined for each set of (VY,VZ) coordinates. In some embodiments, the route optimization score is an indication of factors such as predicted amount of time before movable object102will collide with obstacle1316, differences between the set of candidate movement characteristics and the set of current movement characteristics of movable object102, and/or a distance between movable object102and target106(e.g., a target being tracked) at a predetermined future time. In some embodiments, a set of (VY,VZ) coordinates that has a highest route optimization score is selected and one or more movement characteristics of movable object102are adjusted based on the selected set.

For example, sets2002,2004,2006, correspond to (VY,VZ) coordinates (24, 21), (26, 21), (28, 21) respectively, where each value indicates a component of a velocity vector along respective axes VY,VZ(e.g., in meters per second). One or more rules are applied to determine route optimization scores for each set. For example, route optimization scores of 0.4, 0.62, and 0.51 are determined for sets2002,2004, and2006, respectively. In this example (in which only three sets are evaluated for ease of description), set2004is selected because set2004has the highest route optimization score. Movement of movable object102is adjusted, e.g., to adjust its (VY,VZ) motion to (26, 21) meters per second.

FIG.21illustrates sets of candidate movement characteristics for determining a (VX, ωZ) motion adjustment, in accordance with some embodiments. In some embodiments, in response to determining that the location of the obstacle does not correspond to reactive region1602, one or more movement characteristics of the movable object102are adjusted in a proactive manner (e.g., such that a distance between the movable object102and the obstacle1316exceeds a first predefined distance). In some embodiments, (VX, ωZ) adjustment criteria are applied to determine whether a (VY,VZ) adjustment or a (VX, ωZ) motion adjustment is to be used. For example, in some embodiments, when a (VY,VZ) adjustment will be insufficient to adjust movement characteristics of the movable object102such that a distance between the movable object102and the obstacle1316exceeds a first predefined distance, a (VX, ωZ) motion adjustment is used. In some embodiments, (VX, ωZ) adjustment criteria include a determination of whether a size of obstacle1316exceeds a threshold size (e.g., a threshold size that varies depending on motion of movable object102and/or obstacle1316), such that a (VX, ωZ) motion adjustment is used when the size of obstacle1316exceeds the threshold size. In some embodiments, adjusting one or more movement characteristics of the movable object102in a proactive manner includes a (VY,VZ) adjustment and a (VX, ωZ) motion adjustment.

In some embodiments, adjusting one or more movement characteristics of the movable object102in a proactive manner includes determining route optimization scores for multiple sets of candidate (VX, ωZ) movement characteristics. A set of candidate movement characteristics is, e.g., a set of (VX, ωZ) coordinates (e.g., set2102) in frame of reference1900. Plot2100includes multiple sets (e.g., sets2102,2104,2106, and so on) of candidate movement characteristics. In some embodiments, a route optimization score is determined for each set of (VX, ωZ) coordinates. In some embodiments, the route optimization score is an indication of factors such as predicted amount of time before movable object102will collide with obstacle1316, differences between the set of candidate movement characteristics and the set of current movement characteristics of movable object102, and/or a distance between movable object102and target106(e.g., a target being tracked) at a predetermined future time. In some embodiments, a set of (VX, ωZ) coordinates that has a highest route optimization score is selected and one or more movement characteristics of movable object102are adjusted based on the selected set.

FIGS.22A-22Billustrate (VX, ωZ) adjustment criteria (e.g. obstacle size criteria) applied to determine whether a (VY,VZ) adjustment or a (VX, ωZ) motion adjustment is to be used, in accordance with some embodiments. In some embodiments obstacle size criteria are met when a size of obstacle1316exceeds a threshold size. InFIG.22A, obstacle1316-F exceeds a threshold size. For example, the size of1316-F is sufficiently large that a (VY,VZ) adjustment will be insufficient, so a (VX, ωZ) adjustment is needed. In some embodiments, because obstacle1316-F exceeds the threshold size, obstacle size criteria are met, and a (VX, ωZ) set is selected from plot2100for adjusting one or more movement characteristics of movable object102.

InFIG.22B, obstacle1316-G does not exceed a threshold size. For example, the size of1316-G is sufficiently small that a (VY,VZ) adjustment will be sufficient. In some embodiments, because obstacle1316-G does not exceed the threshold size, (VX, ωZ) obstacle size criteria are not met, and a (VY,VZ) set is selected from plot2000for adjusting one or more movement characteristics of movable object102.

FIGS.23A-23Fare a flow diagram illustrating a method2300for controlling a movable object, in accordance with some embodiments. The method2300is performed at a device, such as moving object102, control unit104and/or computing device126. For example, instructions for performing the method2300are stored in motion control module402of memory118and executed by processor(s)116.

The device obtains (2212) current location information of an obstacle1316while movable object102tracks a target106. Current location information for the obstacle1316includes, e.g., an absolute position of the obstacle1316(such as GPS coordinates of obstacle1316), a relative position of the obstacle (e.g., a vector from movable object102to the obstacle1316), a scalar distance from movable object102to the obstacle1316, and/or one or more motion attributes of the obstacle, such as a velocity, acceleration and/or direction of movement of the obstacle1316.

The device determines (2214), based on the current location information of the obstacle1316, whether a location of the obstacle1316corresponds to (e.g., is within) a reactive region1602(e.g., reactive obstacle avoidance region) relative to movable object102.

In response to determining that the location of the obstacle1316corresponds to (e.g., is within) the reactive region1602, the device adjusts (2216) one or more movement characteristics of movable object102in a reactive manner (e.g., reducing an acceleration of movable object102along one or more axes, reducing a velocity of movable object102along one or more axes, and/or reversing a direction of movement of movable object102) such that collision of movable object102with the obstacle1316is avoided.

In response to determining that the location of the obstacle1316does not correspond to (e.g., is not within) the reactive region1602, the device adjusts (2308) one or more movement characteristics of movable object102in a proactive manner such that a distance between movable object102and the obstacle1316exceeds a first predefined distance.

In some embodiments, the current location information of the obstacle1316includes information obtained (2310) using one or more depth maps. In some embodiments, multiple depth maps are used, e.g., to determine motion attributes of obstacle1316, such as velocity of obstacle1316, acceleration of obstacle1316and/or direction of movement of obstacle1316along one or more axes.

In some embodiments, a depth map is a set of points including an indication of a distance to a surface from a viewpoint (e.g., a distance to a surface that is closest to the viewpoint) corresponding to each point. For example, a depth map is an image in which each pixel includes an indication of a distance to a surface (e.g., the surface nearest to the viewpoint) from a viewpoint. In some embodiments, the viewpoint is movable object102and/or a part thereof.

In some embodiments, a respective depth map is obtained (2312) using one or more sensors of movable object sensing system122. In some embodiments, the at least one sensor is a light sensor (e.g., left stereographic image sensor308, right stereographic image sensor310, left infrared sensor316, and/or right infrared sensor318). In some embodiments, the at least one sensor is a pressure sensor (such as a sound pressure level sensor, e.g., one or more audio transducers314). In some embodiments, a depth map is obtained suing a sonar system including audio output transducer312and audio input transducer314.

In some embodiments, the at least one sensor of movable object102includes (2314) a pair of sensors (left stereographic image sensor308, right stereographic image sensor310) for depth mapping.

In some embodiments, the current location information of the obstacle1316includes (2316) information obtained using at least one sensor of movable object102(e.g., one or more sensors of movable object sensing system122). In some embodiments, the at least one sensor is used to determine a distance to an obstacle1316(e.g., in lieu of or in addition to using depth map to determine current location information of the obstacle1316).

In some embodiments, the current location information of the obstacle1316includes a position (2318) of the obstacle1316, such as GPS coordinates of the obstacle1316.

In some embodiments, the current location information includes (2320) one or more movement characteristics (e.g. velocity (e.g., velocity along one or more axes), acceleration (e.g., acceleration along one or more axes) and/or a vector indicating a direction of movement of the obstacle1316.

In some embodiments, the current location information of the obstacle1316is transmitted (2322) from a computing device126to movable object102(e.g., communicated via a wireless communication channel and received by a communication system of movable object).

In some embodiments, the current location information of the obstacle1316is received (2324) by movable object102from a computing device126.

In some embodiments, the reactive region1602is defined (2326) at least in part based on a determined distance from movable object102(e.g., the reactive region1602is a circular area and/or spherical volume corresponding to movable object102(e.g., centered on movable object102), and the circular area and/or spherical volume has a radius1604equal to a predetermined distance from movable object102. In some embodiments, the determined distance is based on one or more current movement characteristics of movable object102. For example, as the velocity of movable object102increases, the determined distance between movable object102and the obstacle1316increases, e.g., because movable object102requires more time for adjustment to avoid obstacle1316.

In some embodiments, the determined distance is further based (2328) on one or more current movement characteristics of the obstacle1316. For example, if obstacle1316is moving toward movable object102, the determined distance between movable object102and the obstacle1316increases, e.g., because movable object102requires more time for adjustment to avoid obstacle1316.

In some embodiments, in response to determining that the location of the obstacle1316does not correspond to the reactive region, the device (2330): selects multiple sets of candidate movement characteristics (e.g., plot2000, plot2100) based on the one or more movement characteristics of movable object102; obtains a route optimization score for each set of candidate movement characteristics of the multiple sets of candidate movement characteristics in accordance with a first set of rules; selects the candidate movement characteristics that have a highest route optimization score; and adjusts the one or more movement characteristics of movable object102based on the selected candidate movement characteristics.

In some embodiments, the device predicts (2332), for a set of candidate movement characteristics, a time at which movable object102will collide with the obstacle1316; and determines a route optimization score for the set of candidate movement characteristics based on a difference between a current time and the predicted time at which movable object102will collide with the obstacle1316. For example, a route optimization score assigned to the set of candidate movement characteristics increases as the difference between the current time and the predicted time increases.

In some embodiments, the predicted amount of time before movable object102will collide with the obstacle1316is determined (2334) using one or more movement characteristics of the obstacle1316.

In some embodiments, the route optimization score for a set of candidate movement characteristics depends (2336), at least in part, upon one or more differences between the set of candidate movement characteristics and the set of current movement characteristics of movable object102. For example, as the differences between a set of candidate movement characteristics (VY,VZ) and/or (VX, ωZ) and a current (VY,VZ) and/or (VX, ωZ) increases (e.g., increases with a higher ΔVY, ΔVZ, ΔVXand/or ΔωZ).

In some embodiments, the device predicts (2338), for a set of candidate movement characteristics, a distance between movable object102and target106at a predetermined future time. A predetermined time is, e.g., a time that is a predetermined amount of time from a current time (e.g., a time between 0.1 and 10 seconds, such as 3 seconds). The predetermined time varies based on, e.g., one or more movement characteristics of movable object102and/or target106. In some embodiments, the route optimization score for a set of candidate movement characteristics depends, at least in part, upon the predicted distance between movable object102and the target106at the predetermined future time. For example, as the distance between movable object102and the target106decreases, the route optimization score increases.

In some embodiments, a respective set of candidate movement characteristics includes (2340) a first movement characteristic in a first direction that is perpendicular to movement of the movable object102(e.g., along a path1302and/or in a direction of movable object102as it tracks target106) and a second movement characteristic in a second direction that is perpendicular to the movement of movable object102. For example, the first direction is perpendicular to the second direction, and both the first direction and the second direction are perpendicular to an axis aligned along a path of movement of movable object102.

In some embodiments, a respective set of candidate movement characteristics includes (2342) a y-axis movement characteristic VYand a z-axis movement characteristic VZ(e.g., as described with regard toFIGS.19and20).

In some embodiments, a respective set of candidate movement characteristics includes (2344) a movement characteristic in a direction of movement of movable object102(e.g., along path1302and/or in a direction of movement of movable object102as it tracks target106) and an angular velocity.

In some embodiments, a respective set of candidate movement characteristics includes (2346) an x-axis movement characteristic VXand an angular velocity ωX(e.g., as described with regard toFIGS.19and21).

In some embodiments, the device determines (2348) a size of the obstacle1316; determines whether the size of the obstacle1316meets first obstacle size criteria, wherein: in response to determining that the size of the obstacle1316meets first obstacle size criteria, the multiple sets of candidate movement characteristics include movement of movable object102along a y-axis and a z-axis relative to movable object102(e.g., (VY,VZ)), and in response to determining that the size of the obstacle does not meet first obstacle size criteria, a respective set of candidate movement characteristics of the multiple sets of candidate movement characteristics include: movement of movable object102along an x-axis relative to movable object102and angular velocity of movable object102(e.g., (VX, ωX)).

In some embodiments, in response to determining that the location of the obstacle1316corresponds to the reactive region1602, adjusting the one or more movement characteristics includes (2350): determining, based on the current location information of the obstacle1316, whether the location of the obstacle1316corresponds to the first sub-region1702of the reactive region relative to movable object102; and in response to determining that the location of the obstacle1316corresponds to the first sub-region1702of the reactive region relative to movable object102, reducing an acceleration (e.g., along one or more axes) of movable object102. For example, an acceleration of movable object102toward obstacle1316is reduced.

In some embodiments, in response to determining that the location of the obstacle1316corresponds to the reactive region1602, adjusting the one or more movement characteristics includes (2352): determining, based on the current location information of the obstacle1316, whether the location of the obstacle1316corresponds to a second sub-region of the reactive region relative to movable object102; and in response to determining that the location of the obstacle1316corresponds to the second sub-region1704of the reactive region1602relative to movable object102, reducing a velocity (e.g., along one or more axes) of movable object102.

In some embodiments, an area (and/or volume) of the second sub-region1704of the reactive region relative to movable object102is smaller (2354) than an area (and/or volume) of the first sub-region1702of the reactive region1602relative to movable object102.

In some embodiments, in response to determining that the location of the obstacle1316corresponds to the reactive region1602, adjusting the one or more movement characteristics includes (2356): determining, based on the current location information of the obstacle1316, whether the location of the obstacle1316corresponds to a third sub-region1706of the reactive region1602relative to movable object102; and, in response to determining that the location of the obstacle1316corresponds to the third sub-region1706of the reactive region relative to movable object102, reversing a direction of movement (e.g., along one or more axes) of movable object102.

In some embodiments, an area (and/or volume) of the third sub-region1706of the reactive region1602relative to movable object102is smaller (2358) than the area (and/or volume) of the second sub-region1706of the reactive region relative to movable object102.

In some embodiments, in response to determining that the location of the obstacle1316corresponds to the reactive region1602, the device transmits (2360) updated targeting information (e.g., an indication and/or instruction that tracking of target106is to be suspended) to a control unit104(e.g., as described with regard toFIGS.18A-18B).

In some embodiments, the adjusted target tracking information includes (2362) information about the obstacle1316(e.g., an image of obstacle1316, one or more movement characteristics of obstacle1316, and/or current location information of obstacle1316).

FIGS.24A-24Gare a flow diagram illustrating a method2400for controlling a movable object102, in accordance with some embodiments. The method2400is performed at a device, such as moving object102, control unit104and/or computing device126. For example, instructions for performing the method2400are stored in motion control module402of memory118and executed by processor(s)116.

The device obtains (2402) current location information of an obstacle1316while the movable object102tracks a target106.

The device determines (2406), based on the current location information of the obstacle1316, whether a location of the obstacle1316corresponds to a reactive region1602relative to the movable object102.

In response to determining that the location of the obstacle1316corresponds to the reactive region, the device (2408): adjusts one or more movement characteristics of the movable object102, adjusts target tracking information based on a distance between the obstacle1316and the movable object102, and sends the adjusted target tracking information to a control unit104. The control unit104is configured to update a displayed user interface1200in accordance with the adjusted target tracking information.

In some embodiments, the distance between the obstacle1316and the movable object102is determined (2410) based on the current location information of the obstacle1316.

In some embodiments, the distance between the obstacle1316and the movable object102is determined (2412) based on output of one or more sensors of movable object sensing system122.

In some embodiments, adjusting the one or more movement characteristics of the movable object102includes (2414) adjusting the one or more movement characteristics of the movable object102based on: the distance between the obstacle1316and the movable object102; and one or more current movement characteristics of the movable object102.

In some embodiments, adjusting the one or more movement characteristics of the movable object102further includes (2416): determining one or more movement characteristics of the obstacle1316. The one or more movement characteristics of the movable object102are adjusted based on motion of the obstacle1316.

In some embodiments, the motion of the obstacle1316is determined (2418) based on the current location information of the obstacle1316.

In some embodiments, the motion of the obstacle1316is determined (2420) based on output of one or more sensors of movable object sensing system122.

In some embodiments, adjusting the one or more movement characteristics of the movable object102further includes (2422) applying a first movement adjustment to movement of the movable object102(e.g., towards the obstacle1316) when the distance between the obstacle1316and the movable object102meets first distance criteria (e.g., when the obstacle1316is located within a first region1702surrounding the movable object102). In some embodiments, when the distance between the obstacle1316and the movable object102meets second distance criteria (e.g., the obstacle1316is located within a second region1704surrounding the movable object102, the second region1704smaller than the first region1702), the device applies a second movement adjustment to movement of the movable object102(e.g., towards the obstacle1316).

In some embodiments, the first distance criteria are met (2424) when the distance between the obstacle1316and the moving object120exceeds a first distance (e.g., a distance of radius1710) and the second distance criteria are met when the distance between the obstacle1316and the moving object102exceeds a second distance (e.g., a distance of radius1712) that is smaller than the first distance.

In some embodiments, the first distance criteria are met (2426) when the location of the obstacle1316corresponds to a first sub-region1702of the reactive region1602relative to the movable object102and the second distance criteria are met when the location of the obstacle1316corresponds to a second sub-region1704of the reactive region1602relative to the movable object102. For example, the first distance criteria are met when obstacle1316is located within first sub-region1702(for example, beyond second sub-region1704) and the second distance criteria are met when the obstacle1316is located within a second sub-region1704(e.g., beyond a third sub-region1706).

In some embodiments, applying the first movement adjustment includes (2428) reducing an acceleration of the movable object. In some embodiments, applying the second movement adjustment includes (2428) reducing a velocity of the movable object.

In some embodiments, adjusting the one or more movement characteristics of the movable object102includes (2430), when the distance between the obstacle1316and the movable object102meets third distance criteria, applying a third movement adjustment to movement of movable object102. In some embodiments, the third distance criteria are met when the distance between the obstacle1316and the moving object102is a third distance that is smaller than the second distance (e.g., the distance between the obstacle1316and the moving object102is a distance less than or equal to the distance of radius1712).

In some embodiments, applying the third movement adjustment includes reversing motion (2432) of the movable object.

In some embodiments, applying the third movement adjustment includes ceasing motion (2434) of the movable object.

In some embodiments, adjusting the one or more movement characteristics of the movable object102includes (2436): when the distance between the obstacle1316and the movable object102meets third distance criteria, applying a third movement adjustment to movement of movable object102(e.g., reversing a direction of movement of movable object102), wherein the third distance criteria are met when the location of the obstacle1316corresponds to a third sub-region1706of the reactive region1602relative to the movable object102. For example, the third distance criteria are met when obstacle1316is located within third sub-region1706.

In some embodiments, adjusting the one or more movement characteristics of the movable object102includes (2438): when the distance between the obstacle1316and the movable object102meets third distance criteria, determining whether a velocity of the movable object102meets velocity criteria; in response to determining that the velocity of the movable object102meets velocity criteria, ceasing motion of the movable object102; and, in response to determining that the velocity of the movable object102does not meet velocity criteria, reversing a direction of movement of the movable object102.

In some embodiments, the velocity criteria are met (2440) when the velocity of the movable object102exceeds a threshold velocity (e.g., a threshold velocity between 5 meters per second and 60 meters per second, such as a threshold velocity of 25 meters per second).

In some embodiments, the adjusted target tracking information includes (2442) information for altering a graphical tracking indicator1208corresponding to the target106.

In some embodiments, the information for altering the graphical tracking indicator1208corresponding to the target106includes (2444) information for changing a color of the graphical tracking indicator1208.

In some embodiments, the information for altering the graphical tracking indicator1208corresponding to the target106includes (2446) information for ceasing to display the graphical tracking indicator1208.

In some embodiments, the device determines (2448) an updated location of the obstacle1316. For example, the device determines an updated location of obstacle1316after altering graphical tracking indicator1208and/or after adjusting one or more movement characteristics of movable object102. The updated location of obstacle1316depends on, e.g., movement of movable object012and/or movement of obstacle1316. In some embodiments, the device determines (2450) whether the updated location of the obstacle1316corresponds to the reactive region1602relative to the movable object102(e.g., the device determines whether the obstacle is at least partially within reactive region1602). In response to determining that the updated location of the obstacle1316does not correspond to the reactive region1602relative to the movable object102, the device further adjusts (2452) the target tracking information. In some embodiments, the device sends the further adjusted target tracking information to the control unit104. In some embodiments, the further adjusted target tracking information includes information for ceasing to alter the graphical tracking indicator1208. For example, when the movable object102is no longer at risk of (imminent) collision with obstacle1316, graphical tracking indicator is re-displayed and/or displayed with its previous color.

In some embodiments, the adjusted target tracking information includes (2454) information for altering (e.g., temporarily altering) a response to a predefined user input received by the control unit104.

In some embodiments, the predefined user input includes (2456) control input for controlling at least one movement characteristic of the movable object102(e.g., as described with regard toFIG.5andFIG.12).

In some embodiments, the predefined user input includes (2458) selection input for selecting a target106to track (e.g., as described with regard toFIG.12).

In some embodiments, altering the response to the predefined user input includes (2460) ceasing to respond to the predefined user input.

In some embodiments, the device determines (2462) an updated location of the obstacle1316. In some embodiments, the device determines (2464) whether the updated location of the obstacle1316corresponds to the reactive region1602relative to the movable object102. In some embodiments, in response to determining that the updated location of the obstacle1316does not correspond to the reactive region1602relative to the movable object102, the device further adjusts (2466) the target tracking information and sends the further adjusted target tracking information to the control unit104. In some embodiments, the further adjusted target tracking information includes information for ceasing to alter the response to the predefined user input received by the control unit104. For example, when the movable object102is no longer at risk of (imminent) collision with obstacle1316, user ability to control movable object102using control device104resumes.

In some embodiments, the current location information of the obstacle1316includes (2468) information obtained using one or more depth maps.

In some embodiments, the current location information of the obstacle1316includes (2470) information obtained using one or more sensors of movable object sensing system122.

In some embodiments, the reactive region1602is defined (2472) at least in part based on a determined distance (e.g.1604) from the movable object102. In some embodiments, the determined distance is based on one or more current movement characteristics of the movable object102.

In some embodiments, the determined distance is further based on (2474) one or more current movement characteristics of the obstacle1316.

FIGS.25A-25Gare a flow diagram illustrating a method2500for controlling a movable object102, in accordance with some embodiments. The method2500is performed at a device, such as moving object102, control unit104and/or computing device126. For example, instructions for performing the method2500are stored in motion control module402of memory118and executed by processor(s)116.

The device obtains (2502) current location information of an obstacle1316while the movable object102tracks a target106.

The device generates (2504) a plurality of sets of candidate movement characteristics (e.g., sets2002,2004,2006of plot2000; sets2102,2104,2106of plot2001) for the movable object102based on the current location information of the obstacle1316and a set of current movement characteristics of the movable object102.

The device selects (2506), from the plurality of sets of candidate movement characteristics for the movable object102, a set of movement characteristics for the movable object102.

The device adjusts (2508) one or more movement characteristics of the movable object102based on the selected set of movement characteristics for the movable object102.

In some embodiments, after adjusting (2510) the one or more movement characteristics of the movable object102based on the selected set of movement characteristics for the movable object102(e.g., after adjusting the one or more movement characteristics for a predefined time period, such as 1 second), the device repeats the obtaining, generating, selecting, and adjusting operations.

In some embodiments, the device assigns (2512) a route optimization score to each set of candidate movement characteristics of the plurality of sets of candidate movement characteristics and determines a set of movement characteristics for the movable object102that has a highest route optimization score. In some embodiments, selecting the set of movement characteristics for the movable object102includes selecting the set of candidate movement characteristics for the movable object102that has the highest route optimization score.

In some embodiments, after adjusting (2514) the one or more movement characteristics of the movable object102based on the selected set of movement characteristics for the movable object102(e.g., after adjusting the one or more movement characteristics for a predefined time period, such as 1 second), the device repeats the obtaining, generating, assigning, determining, selecting, and adjusting operations.

In some embodiments, the device predicts (2516), for a set of candidate movement characteristics, a time at which the movable object102will collide with the obstacle1316and determines a route optimization score to assign to the set of candidate movement characteristics based at least in part on a difference between a current time and the predicted time at which the movable object102will collide with the obstacle1316.

In some embodiments, the device determines (2518), for a set of candidate movement characteristics, whether the movable object102will collide with the obstacle1316. in response to determining that the movable object102will collide with the obstacle1316, the device predicts, for the set of candidate movement characteristics, a time at which the movable object102will collide with the obstacle1316, and determines a route optimization score to assign to the set of candidate movement characteristics based at least in part on a difference between a current time and the predicted time at which the movable object102will collide with the obstacle1316. In response to determining that the movable object102will not collide with the obstacle1316, the device determines a route optimization score to assign to the set of candidate movement characteristics based at least in part on a default value corresponding to a determination that the movable object102will not collide with the obstacle1316.

In some embodiments, the predicted amount of time before the movable object102will collide with the obstacle1316is determined (2520) using one or more current movement characteristics of the obstacle1316.

In some embodiments, the current location information of the obstacle1316includes (2522) the one or more current movement characteristics of the obstacle1316.

In some embodiments, the one or more current movement characteristics of the obstacle1316are determined (2524) using one or more sensors of movable object sensing system122.

In some embodiments, the device determines (2526) a route optimization score to assign to a respective set of candidate movement characteristics based at least in part on differences between the set of candidate movement characteristics and the set of current movement characteristics of the movable object102.

In some embodiments, the device predicts (2528), for a set of candidate movement characteristics, a distance between the movable object102and the target106at a predetermined future time; and the device determines a route optimization score to assign to the set of candidate movement characteristics based at least in part on the predicted distance between the movable object102and the target106at the predetermined future time.

In some embodiments, predicting the distance between the movable object102and the target106at a predetermined future time includes (2530) obtaining at least one current movement characteristic of the target106. The device predicts the distance using the obtained at least one current movement characteristic of the target106.

In some embodiments, the device assigns a route optimization score to each set of candidate movement characteristics of the plurality of sets of candidate movement characteristics. The device determines a set of movement characteristics for the movable object102that has a highest route optimization score. The device determines whether the set of movement characteristics for the movable object102that has the highest route optimization score complies with (2532) constraint criteria. In response to determining that the set of candidate movement characteristics complies with the constraint criteria, the device selects the set of movement characteristics for the movable object102that has the highest route optimization score. In response to determining that the set of candidate movement characteristics does not comply with the constraint criteria, the device selects an alternative set of movement characteristics for the movable object102.

For example, in some embodiments, one or more movement characteristics of movable object102, such as a linear velocity, an angular velocity, a linear acceleration, an angular acceleration, and/or an altitude are constrained. For example, a constraint exists due to, e.g., a mechanical limit (e.g., a mechanical limit of an actuator controlling a movement mechanism114) and/or a policy limit (e.g., a law limiting allowable velocity, acceleration, and/or elevation). In some embodiments, selection of a set of candidate movement characteristics and/or generation of candidate movement characteristics is constrained based on the constraints.

In some embodiments, the constraint criteria include (2534) a maximum linear velocity of the movable object102.

In some embodiments, the constraint criteria include a (2536) maximum angular velocity of the movable object102.

In some embodiments, the plurality of sets of candidate movement characteristics are generated (2538) subject to constraint criteria.

In some embodiments, the constraint criteria include (2540) a maximum linear velocity of the movable object102.

In some embodiments, the constraint criteria include (2542) a maximum angular velocity of the movable object102.

In some embodiments, a respective set of candidate movement characteristics includes (2544) a y-axis movement characteristic and a z-axis movement characteristic (e.g., VZ). A movement characteristic is, e.g., an x-axis velocity VX, an x-axis acceleration, a y-axis velocity VY, a y-axis acceleration, a z-axis velocity VZ, and/or z-axis acceleration.

In some embodiments, the y-axis movement characteristic and the z-axis movement characteristic are determined (2546) in a frame of reference1900of the movable object102(e.g., as described with regard toFIG.19).

In some embodiments, the respective set of candidate movement characteristics includes (2548) a y-axis velocity VYand a z-axis velocity VZ.

In some embodiments, the plurality of sets of candidate movement characteristics include (2550): a respective set of candidate movement characteristics including a negative y-axis velocity value and a negative z-axis velocity value; a set of candidate movement characteristics including a negative y-axis velocity value and a positive z-axis velocity value; a set of candidate movement characteristics including a positive y-axis velocity value and a negative z-axis velocity value; and a set of candidate movement characteristics including a positive y-axis velocity value and a positive z-axis velocity value. For example, candidate movement characteristics inFIG.20are shown in each quadrant of the coordinate plot2000.

In some embodiments, a respective set of candidate movement characteristics includes (2552) an x-axis movement characteristic and an angular velocity (e.g., ωX). An x-axis movement characteristic is, e.g., an x-axis velocity VXand/or an x-axis acceleration.

In some embodiments, the x-axis movement characteristic and the angular velocity are determined (2554) in a frame of reference1900of the movable object102.

In some embodiments, the x-axis movement characteristic (2556) is a velocity VX.

In some embodiments, the plurality of sets of candidate movement characteristics include (2558): a set of candidate movement characteristics including a negative x-axis velocity value and a negative angular velocity value; a set of candidate movement characteristics including a negative x-axis velocity value and a positive angular velocity value; a set of candidate movement characteristics including a positive x-axis velocity value and a negative angular velocity value; and a set of candidate movement characteristics including a positive x-axis velocity value and a positive angular velocity value. For example, candidate movement characteristics inFIG.22are shown in each quadrant of the coordinate plot2100.

In some embodiments, the device determines (2560) a size of the obstacle1316. The device determines whether the size of the obstacle meets first obstacle size criteria. In response to determining that the size of the obstacle meets first obstacle size criteria, the multiple sets of candidate movement characteristics include movement of the movable object along a y-axis and a z-axis relative to the movable object. In response to determining that the size of the obstacle does not meet first obstacle size criteria, a respective set of candidate movement characteristics of the multiple sets of candidate movement characteristics include movement of the movable object along an x-axis relative to the movable object, and angular velocity of the movable object.

Many features of the present disclosure can be performed in, using, or with the assistance of hardware, software, firmware, or combinations thereof. Consequently, features of the present disclosure may be implemented using a processing system. Exemplary processing systems (e.g., processor(s)116, controller210, controller218, processor(s)502and/or processor(s)602) include, without limitation, one or more general purpose microprocessors (for example, single or multi-core processors), application-specific integrated circuits, application-specific instruction-set processors, field-programmable gate arrays, graphics processing units, physics processing units, digital signal processing units, coprocessors, network processing units, audio processing units, encryption processing units, and the like.

Features of the present disclosure can be implemented in, using, or with the assistance of a computer program product, such as a storage medium (media) or computer readable medium (media) having instructions stored thereon/in which can be used to program a processing system to perform any of the features presented herein. The storage medium (e.g., (e.g. memory118,504,604) can include, but is not limited to, any type of disk including floppy disks, optical discs, DVD, CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, DDR RAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data.

Stored on any one of the machine readable medium (media), features of the present disclosure can be incorporated in software and/or firmware for controlling the hardware of a processing system, and for enabling a processing system to interact with other mechanism utilizing the results of the present disclosure. Such software or firmware may include, but is not limited to, application code, device drivers, operating systems and execution environments/containers.

Communication systems as referred to herein (e.g., communication systems120,510,610) optionally communicate via wired and/or wireless communication connections. For example, communication systems optionally receive and send RF signals, also called electromagnetic signals. RF circuitry of the communication systems convert electrical signals to/from electromagnetic signals and communicate with communications networks and other communications devices via the electromagnetic signals. RF circuitry optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and so forth. Communication systems optionally communicate with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet and/or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and/or a metropolitan area network (MAN), and other devices by wireless communication. Wireless communication connections optionally use any of a plurality of communications standards, protocols and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 102.11a, IEEE 102.11ac, IEEE 102.11ax, IEEE 102.11b, IEEE 102.11g and/or IEEE 102.11n), voice over Internet Protocol (VoIP), Wi-MAX, a protocol for e-mail (e.g., Internet message access protocol (IMAP) and/or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and/or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

While various embodiments of the present disclosure have been described above, it should be understood that they have been presented by way of example, and not limitation. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made therein without departing from the spirit and scope of the disclosure.

The present disclosure has been described above with the aid of functional building blocks illustrating the performance of specified functions and relationships thereof. The boundaries of these functional building blocks have often been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Any such alternate boundaries are thus within the scope and spirit of the disclosure.

The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting,” that a stated condition precedent is true, depending on the context. Similarly, the phrase “if it is determined [that a stated condition precedent is true]” or “if [a stated condition precedent is true]” or “when [a stated condition precedent is true]” may be construed to mean “upon determining” or “in response to determining” or “in accordance with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.

The foregoing description of the present disclosure has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments. Many modifications and variations will be apparent to the practitioner skilled in the art. The modifications and variations include any relevant combination of the disclosed features. The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical application, thereby enabling others skilled in the art to understand the disclosure for various embodiments and with various modifications that are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalence.