Patent Description:
Highly skilled human operators are typically used to guide complex, highspeed docking operations, such as air-to-air refueling and spacecraft docking operations. As such, the operations rely heavily on human judgment, which is sometimes supplemented by computer vision techniques. To illustrate, complex stereoscopic vision systems may be used to aid the human operator in mating connectors (e.g., a receiver and refueling boom or docking connectors).

Document <CIT>, with its abstract, discloses aerial refueling systems and associated methods. A system includes an operator input device configured to receive operator inputs and direct a first input signal corresponding to a target position for an aerial refueling device. A sensor can be positioned to detect a location of at least one of the aerial refueling device and a receiver aircraft, and can be configured to direct a second input signal. A controller can be operatively coupled to the operator input device and the sensor to receive the first and second input signals and direct a command signal to adjust the position of the aerial refueling device in response to both the first and second input signals, unless either or both of the input signals are absent or below a threshold value. Accordingly, the system can respond to both automatically generated sensor data and data input by an operator. A fully automated version of the system can be installed on an unmanned aircraft having additional capabilities, for example, electronic surveillance and/or jamming capabilities.

The present disclosure provides a method comprising the steps described at claim <NUM>. The dependent claims outline advantageous ways of carrying out the method. Additionally, the present disclosure provides a device having the features described at claim <NUM>. The dependent claims outline advantageous forms of embodiment of the device.

The features, functions, and advantages described herein can be achieved independently in various implementations or may be combined in yet other implementations, further details of which can be found with reference to the following description and drawings.

Aspects disclosed herein present systems and methods that use a trained autonomous agent to generate proposed maneuvers for mating connectors of two different devices. For example, the autonomous agent may include one or more neural networks or other machine-learning models that are trained to generate maneuvering recommendations or commands to guide a refueling connector to a refueling port of an aircraft during air-to-air refueling operations. As another example, the autonomous agent may be trained to generate maneuvering recommendations or commands to guide docking of one spacecraft to another spacecraft. In some implementations, the trained autonomous agent may be used to assist a human operator to improve reliability and to standardize operations during maneuvering to mate connectors. In other implementations, the trained autonomous agent may be used instead of a human operator to reduce costs, such as costs associated with training human operators and costs associated with operations to mate connectors.

In some contexts, the two devices performing docking include a primary device and a secondary device. Although the terms may be arbitrarily assigned in some contexts (such as where two peer devices are docking), generally, the primary device refers to a device that is servicing the secondary device, or the primary device refers to the device, onboard which the trained autonomous agent resides. To illustrate, in an air-to-air refueling context, the primary device is the tanker aircraft. Likewise, the secondary device refers to the other device of a pair of devices. To illustrate, in the air-to-air refueling context, the secondary device is the receiving aircraft (e.g., the aircraft that is to be refueled). Further, the term device is used broadly to include an object, system, or assembly of components that is/are operated upon as a unit (e.g., in the case of the secondary device) or that operate cooperatively to achieve a task (e.g., in the case of the primary device).

In a particular aspect, a system uses a camera (e.g., a single camera) to capture monocular video of at least a portion of each of the devices that are undergoing docking operations. For example, the camera may capture images of a portion of a refueling boom and of a receiving aircraft. As another example, the camera may capture images of a portion of docking ports of two spacecraft. The autonomous agent is trained to output a real-time directional indicator of a maneuver to position couplers of the devices undergoing docking. This directional indicator can be output in a manner that can be interpreted and executed by either a human operator or an automated system. In some implementations, the autonomous agent is also, or alternatively, trained to limit maneuvers that can be performed to reduce the likelihood of unintended contact between the devices.

To illustrate, one form of aerial refueling uses a complex targeting operation combined with a controlled docking of a boom from the tanker aircraft to a receptacle on the receiving aircraft. During this operation, an operator interprets images from a camera and guides the boom to dock with the receptacle on the receiving aircraft while both aircraft are in motion. The operator controls the relative angle of the boom as well as the deployed length of the boom. The operation can be complicated due to relative motion of the aircraft, poor visibility, poor lighting conditions, etc. Additional complications can arise when the operator controls the boom in three-dimensional (3D) space based on two-dimensional (2D) information from one or more cameras. For example, interpreting the 2D information from the images complicates the operator's depth perception, the operator's ability to predict boom contact point with receiver, and the operator's evaluation and response to boom jitter due to turbulence. Inaccurate interpretation of the 2D information can result in unsuccessful refueling since the receiving aircraft or the boom may be damaged if the boom contacts portions of the receiving aircraft other than the receptacle. The same or similar challenges are present when two spacecraft are docking.

In a particular aspect disclosed herein, machine learning is used to train an autonomous agent to replace or augment a human operator. For example, the autonomous agent may process available information in a manner that enables precisely locating two devices in 3D space based on 2D image data and sensor data (e.g., position encoder data). As another example, the autonomous agent may determine an optimal policy (under particular circumstances) to mate connectors of the two devices.

In a particular implementation, a U-Net convolutional neural network (CNN) is trained as a feature extractor to map an image of a target to a binary mask (or masks) with the locations of the features denoted by a Gaussian distribution centered on the features. In this implementation, reliability of feature detection is improved by training the feature extractor using images with different levels of detail. For example, an original or source image may be downsampled one or more times to generate one or more downsampled images, and the feature extractor may be trained using the source image and the one or more downsampled images. Training the feature extractor using multiple images with different levels of detail reduces the chance that the feature detector is detecting specific details of features rather than the general attributes, which in turn, improves reliability of the feature detector. To illustrate, the feature extractor is able to detect general attributes even when specific detailed features are obscured.

Downsampling the image removes or reduces higher frequency content representing feature details, which encourages the feature extractor to learn more general features, such as general shapes and geometry. The U-net CNN architecture enables generating feature data at various levels of downsampling and merging the feature data to form feature data output. Forming merged feature data in this manner improves reliability (due the use of low resolution images) while retaining accuracy (due the use of high resolution images). A feature extractor can be trained in this manner to have a lower error rate, by orders of magnitude, than a feature extractor that uses a single image resolution.

The feature data generated by the feature extractor identifies coordinates of key points of the devices in an image. In a particular aspect, an imputation network is trained, based on known geometry of the devices, to recognize and repair outliers in the feature data. For example, the imputation network may compare known geometries of a refueling boom and a particular receiving aircraft to the key points and remove or reposition erroneous key points. In some aspects, a Kalman filter is used to temporally filter the images and/or feature data to further improve accuracy. In some implementations, in addition to image data, the feature extractor may generate the feature data using supplemental sensor data, such as data from one or more of a lidar system, a radar system, etc..

In a particular aspect, the feature data is provided as input to the trained autonomous agent. In some aspects, the trained autonomous agent also receives as input position data indicating a position in 3D space of a first connector of the primary device (e.g., the fueling coupler of the boom in an air-to-air refueling use case). The trained autonomous agent is configured to generate a recommended maneuver based on the feature data and the position data. The recommended maneuver indicates a motion of the connector of the primary device to connect to the connector of the secondary device.

In a particular aspect, the autonomous agent is trained using reinforcement learning techniques. For example, in reinforcement learning, a reward is determined based on how well the autonomous agent performs a desired action. Additionally, or in the alternative, a penalty is determined based on how poorly the autonomous agent performs the desired action. In a particular aspect, the reinforcement learning is used to train the autonomous agent to determine an optimum maneuver, such as a shortest or least cost maneuver to mate the connectors. In another particular aspect, the reinforcement learning is used to train the autonomous agent to mimic one or more highly skilled human operators. Rewards may be applied if the autonomous agent successfully mates connectors of the two devices without the connector of the primary device contacting any surface of the secondary device except the connector of the secondary device. Additionally, or in the alternative, a penalty may be applied if the autonomous agent causes any undesired contact between portions of the primary and secondary devices.

One benefit of the disclosed systems and methods is that machine learning based docking processes can be parallelized for execution on one or more graphical processing units (GPU) to operate more quickly than computer vision techniques that rely on pattern matching. For example, pattern matching techniques generally operate at about one frame every few seconds, whereas the disclosed techniques can operate in excess of <NUM> frames per second. Additionally, the autonomous agent is capable of mimicking a human operator by learning patterns used by highly skilled human operators for mating connectors of the devices and can improve on these patterns to eliminate suboptimal actions or to combine the best maneuvers of multiple different skilled operators.

The figures and the following description illustrate specific exemplary aspects.

Particular implementations are described herein with reference to the drawings. In the description, common features are designated by common reference numbers throughout the drawings. In some drawings, multiple instances of a particular type of feature are used. Although these features are physically and/or logically distinct, the same reference number is used for each, and the different instances are distinguished by addition of a letter to the reference number. When the features as a group or a type are referred to herein (e.g., when no particular one of the features is being referenced), the reference number is used without a distinguishing letter. However, when one particular feature of multiple features of the same type is referred to herein, the reference number is used with the distinguishing letter. Referring to <FIG>, illustrates an example that is generic to any primary device (e.g., first device <NUM>) and any secondary device (e.g., second device <NUM>), whereas <FIG>, <FIG> illustrate specific examples of the primary device (e.g., tanker aircraft 102A or first spacecraft 102B) and specific examples of the secondary device (e.g., receiving aircraft 112A or second spacecraft 112B). When referring to a particular one of the specific examples, such as the example illustrated in <FIG>, the primary device is referred to as the tanker aircraft 102A. However, when referring to any arbitrary example or the generic example of <FIG>, the first device <NUM> is used without a distinguishing letter "A". Unless otherwise indicated in a specific context, each generic description (e.g., a description of the first device <NUM>) is also a description of each of the specific examples (e.g., the tanker aircraft 102A).

As used herein, various terminology is used for the purpose of describing particular implementations only and is not intended to be limiting. For example, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, some features described herein are singular in some implementations and plural in other implementations. To illustrate, <FIG> depicts a system <NUM> including one or more processors ("processor(s)" <NUM> in <FIG>), which indicates that in some implementations the system <NUM> includes a single processor <NUM> and in other implementations the system <NUM> includes multiple processors <NUM>. For ease of reference herein, such features may be introduced as "one or more" features and subsequently referred to in the singular unless aspects related to multiple of the features are being described.

<FIG> is a diagram that illustrates a system <NUM> including several devices including a first device <NUM> that is configured to generate a proposed maneuver to mate a first connector <NUM> of the first device <NUM> with a second connector <NUM> of a second device <NUM> based on image data and position data. In the example illustrated in <FIG>, the first device <NUM> includes or corresponds to a primary device, as described above, and the second device <NUM> includes or corresponds to a secondary device, as described above. The first device <NUM> includes a trained autonomous agent <NUM> and may be configured to service or support the second device <NUM>. The second device <NUM> includes a device or system configured to couple to the first device <NUM> and possibly to be serviced by or supported by the first device <NUM>.

The first device <NUM> includes a moveable coupling system <NUM> configured to move the first connector <NUM> relative to the second connector <NUM> of the second device <NUM>. For example, the moveable coupling system <NUM> may include a steerable boom of a refueling system, as described in further detail in <FIG>. As another example, the moveable coupling system <NUM> may include a steerable docking arm of a docking system, as described in further detail in <FIG>. The above referenced examples are merely illustrative and are not limiting.

The first device <NUM> also includes a camera <NUM>, one or more sensors <NUM>, one or more processors <NUM>, and a memory <NUM>. In the example illustrated in <FIG>, the first device <NUM> also includes one or more image processors <NUM>. In some implementations, the image processor(s) <NUM> and the processor(s) <NUM> are combined. To illustrate, one or more GPUs, one or more central processing units (CPUs), or one or more other multi-core or multi-thread processing units may serve as both the image processor(s) <NUM> and the processor(s) <NUM>.

In <FIG>, the sensor(s) <NUM> include position encoders <NUM> and one or more supplemental sensors <NUM>. The position encoders <NUM> are coupled to the moveable coupling system <NUM> and configured to generate position data <NUM> indicating a position in 3D space <NUM> of the first connector <NUM>. For example, the position encoders <NUM> may generate data that indicates an angle of one or more joints of the moveable coupling system <NUM>, a deployment length of the moveable coupling system <NUM>, other 3D position data, or a combination thereof. In this example, the position data <NUM> indicates the position in a reference frame that is associated with the first device <NUM> and is independent of the second device <NUM>. To illustrate, an angle of a joint of the moveable coupling system <NUM> may be indicated relative to a reference frame of the joint or a reference frame of the first device <NUM>, which angle, by itself, does not indicate a position of the first connector <NUM> or the joint relative to the second device <NUM> because the second device <NUM> is moveable relative to the first device <NUM>.

The supplemental sensor(s) <NUM>, when present, are configured to generate supplemental sensor data (e.g., additional position data) indicative of relative positions of the first device <NUM> and the second device <NUM> in the 3D space <NUM>. For example, the supplemental sensor(s) <NUM> may include a range finder (e.g., a laser range finder), and the supplemental sensor data may include range data (e.g., a distance from the range finder to the second device <NUM>). Additionally, or in the alternative, the supplemental sensor(s) <NUM> may include a radar system, and the supplemental sensor data may include radar data (e.g., radar returns indicating a distance to the second device <NUM>, a direction to the second device <NUM>, or both). Additionally, or in the alternative, the supplemental sensor(s) <NUM> may include a lidar system, and the supplemental sensor data may include lidar data (e.g., lidar returns indicating a distance to the second device <NUM>, a direction to the second device <NUM>, or both). Additionally, or in the alternative, the supplemental sensor(s) <NUM> may include a sonar system, and the supplemental sensor data may include sonar data (e.g., sonar returns indicating a distance to the second device <NUM>, a direction to the second device <NUM>, or both). Additionally, or in the alternative, the supplemental sensor(s) <NUM> may include one or more additional cameras (e.g., in addition to a camera <NUM>), and the supplemental sensor data may include stereoscopic image data.

The camera <NUM> of the first device <NUM> is configured to generate image data (e.g., image(s) <NUM>) that depict at least a portion of the moveable coupling system <NUM> and at least a portion of the second device <NUM>. In some implementations, the image(s) <NUM> include a stream of real-time (e.g., subject to only minor video front-end processing delays and buffering) video frames that represent relative positions of the moveable coupling system <NUM> and the second device <NUM>.

In <FIG>, the image(s) <NUM> are processed by the image processor(s) <NUM> to generate processed image data. The image processor(s) <NUM> of <FIG> include a downsampler <NUM> that is configured to generate one or more downsampled images <NUM> based on the image(s) <NUM>. In some implementations, the downsampler <NUM> can include multiple downsampling stages that generate different levels of downsampled image(s) <NUM>. To illustrate, a source image from the camera <NUM> can be downsampled a first time to generate a first downsampled image, and the first downsampled image can be downsampled one or more additional times to generate a second downsampled image. The downsampled image(s) <NUM> may include the first downsampled image, the second downsampled image, one or more additional images, or a combination thereof. Thus, in some implementations, multiple downsampled images <NUM> may be generated for each source image <NUM> from the camera <NUM>.

In <FIG>, the image processor(s) <NUM> also include a segmentation model <NUM> that is configured to generate one or more segmentation maps <NUM> ("seg. map(s)" in <FIG>) based on the image(s) <NUM>. The segmentation map(s) <NUM> associated with a first image of the image(s) <NUM> represent distinct regions of the first image. The segmentation map <NUM> distinguishes boundaries between various portions of the second device <NUM>; boundaries between the second device <NUM> and portion of the first device <NUM> represented in the first image; boundaries between the second device <NUM>, the first device <NUM>, or both, and a background region; or a combination thereof.

The image(s) <NUM>, the downsampled images(s) <NUM>, or any combination thereof, and the segmentation map(s) <NUM>, are provided as input to a feature extraction model <NUM> to generate feature data <NUM>. In a particular aspect, the feature data <NUM> includes first coordinates representing key points of the moveable coupling system <NUM> and includes second coordinates representing key points of the second device <NUM>. For example, a representative display <NUM> illustrating an image <NUM> of the image(s) <NUM> with at least a portion of the feature data <NUM> is shown in <FIG>. In this example, the image <NUM> depicts a first portion <NUM> of the first device <NUM> (such as a portion of the moveable coupling system <NUM>) and a second portion <NUM> of the second device <NUM>. The first portion <NUM> includes key points <NUM> representing or represented by first coordinates <NUM>, and the second portion <NUM> includes key points <NUM> representing or represented by second coordinates <NUM>. The key points <NUM>, <NUM> and the coordinates <NUM>, <NUM> are determined by the feature extraction model <NUM> and are indicated in the feature data <NUM>.

In some implementations, the feature extraction model <NUM> includes or corresponds to a machine-learning model. To illustrate, the feature extraction model <NUM> may include or correspond to a neural network that is trained to detect the key points <NUM>, <NUM> in the image <NUM> and to determine coordinate locations within the image <NUM> that are associated with each key point. In some implementations, the feature extraction model <NUM> includes or corresponds to one or more convolutional neural networks, such as a U-CNN. In such implementations, the downsampler <NUM> and the feature extraction model <NUM> may be combined in the U-CNN architecture such that a particular image of the image(s) <NUM> is evaluated by CNNs at multiple distinct resolutions to generate the feature data <NUM>. Additionally, or in the alternative, the downsampler <NUM> and the segmentation model <NUM> are combined in a U-CNN architecture, which enables the segmentation map to be formed based on multiple resolutions of the images <NUM>.

In a particular aspect, the feature extraction model <NUM> is configured to generate the feature data <NUM> based, at least in part, on a known geometry <NUM> of the second device <NUM>. For example, the memory <NUM> of <FIG> stores data representing the known geometry <NUM> of the second device <NUM>, and the feature extraction model <NUM> compares key points detected in the image(s) <NUM> with the known geometry <NUM> to detect and correct misplaced key points. As a specific example, the feature extraction model <NUM> may include an imputation network to compare the known geometry <NUM> to the key points and to remove or reposition erroneous key points. In some aspects, the feature extraction model <NUM> also includes a Kalman filter to temporally filter the image(s) <NUM> and/or feature data <NUM> to improve accuracy of the feature data <NUM>.

The feature data <NUM> and the position data <NUM> are provided as input to a trained autonomous agent <NUM>. In a particular aspect, the trained autonomous agent <NUM> is configured to generate a proposed maneuver <NUM> to mate the first connector <NUM> with the second connector <NUM> based on the feature data <NUM> and the position data <NUM>. In a particular implementation, the trained autonomous agent <NUM> includes or corresponds to a neural network. As an example, the neural network of the trained autonomous agent <NUM> is trained using one or more reinforcement learning techniques. To illustrate, during a training phase, the reinforcement learning techniques may train the neural network based on in part on a reward that is determined by comparing a proposed maneuver <NUM> output by the neural network to an optimum or target maneuver in particular circumstances (e.g., for a particular set of input feature data <NUM> and a particular set of position data <NUM>). In this context, the optimum or target maneuver may include, for example, a shortest or least cost maneuver to mate the connectors <NUM>, <NUM>; a maneuver that mimics a maneuver performed by one or more skilled human operators under similar circumstances; a maneuver that satisfies a set of safety conditions, such as not causing any undesired contact between portions of the devices <NUM>, <NUM>; a maneuver that corresponds to maneuvering characteristics specified during or before training; or a combination thereof.

The proposed maneuver <NUM> generated by the trained autonomous agent <NUM> may be output to a maneuvering system <NUM>, to a graphical user interface (GUI) engine <NUM>, or both. The maneuvering system <NUM> is configured to generate and/or execute commands to reposition the first device <NUM>, the moveable coupling system <NUM>, or both. To illustrate, the maneuvering system <NUM> may include or correspond to a control system configured to command repositioning of the moveable coupling system <NUM> based on the proposed maneuver <NUM>. As one specific example, when the first device <NUM> includes a tanker aircraft, the maneuvering system <NUM> includes a flight control system of the tanker aircraft, a boom control system of a refueling boom, or both.

In some implementations, the maneuvering system <NUM> includes or is coupled to one or more instruments <NUM>, a maneuver limiter <NUM>, or both. The instrument(s) <NUM> include or are coupled to control and/or safety sensors associated with the first device <NUM>. In a particular implementations, data from the instrument(s) <NUM> is provided to the trained autonomous agent <NUM> to generate the proposed maneuver <NUM>. For example, when the first device <NUM> corresponds to a tanker aircraft, the instrument(s) <NUM> may include flight instruments (e.g., an altimeter, an angle of attack indicator, a heading indicator, an airspeed indicator, etc.) of the tanker aircraft. In this example, flight data generated by the instrument(s) <NUM> may be provided to the trained autonomous agent <NUM> (along with the feature data <NUM> and the position data <NUM>), to generate the proposed maneuver <NUM>.

The maneuver limiter <NUM> performs checks associated with the maneuvering system <NUM> to determine whether a maneuver limit criterion is satisfied. If the maneuver limiter <NUM> determines that the maneuver limit criterion is satisfied, the maneuver limiter <NUM> causes the maneuvering system <NUM> to limit the maneuvering operations that can be performed. As one example, the maneuver limiter <NUM> may provide data to the trained autonomous agent <NUM> to limit a set of maneuvers that can be selected as the proposed maneuver <NUM>. To illustrate, the maneuver limit criterion may indicate a minimum altitude for air-to-air refueling, and the maneuver limiter <NUM> may prevent the trained autonomous agent <NUM> from proposing a maneuver to mate a boom refueling connector (e.g., the first connector <NUM>) to a fuel receptacle (e.g., the second connector <NUM>) when an altimeter (e.g., one of the instruments <NUM>) indicates that the altitude of a tanker aircraft (e.g., the first device <NUM>) is below the minimum altitude. Additionally, or in the alternative, the maneuver limiter <NUM> may limit maneuvering operations that can be executed by the first device <NUM> or the moveable coupling system <NUM> based on the proposed maneuver <NUM>.

The GUI engine <NUM> is configured to generate the display <NUM> and to provide the display <NUM> to a display device onboard or offboard the first device <NUM>. The display <NUM> includes the image <NUM> depicting the first portion <NUM> of the first device <NUM>, the second portion <NUM> of the second device <NUM>, or both. In some implementations, the key points <NUM>, <NUM>; the coordinates <NUM>, <NUM>; or both are depicted in the display <NUM>. In other implementations, the key points <NUM>, <NUM> and the coordinates <NUM>, <NUM> are determined as part of the feature data <NUM> but are not depicted in the display <NUM>. In some implementations, the display <NUM> includes one or more graphical elements <NUM> overlaying at least a portion of the image <NUM> and indicating the proposed maneuver <NUM>. For example, the graphical element(s) <NUM> may indicate a direction to steer the moveable coupling system <NUM>, an amount (e.g., angular displacement or distance) to steer the moveable coupling system <NUM>, or both.

The trained autonomous agent <NUM>, in conjunction with other features of the first device <NUM>, improves efficiency (e.g., by reducing training costs), reliability, and repeatability of operations to mate the first connector <NUM> and the second connector <NUM>. For example, the trained autonomous agent <NUM> can mimic maneuvers performed by highly skilled human operators without the time and cost required to train the operators. Further, the trained autonomous agent <NUM> can improve on maneuvers performed by the skilled human operators by determining more optimal maneuvers than those executed by the skilled human operators.

Although <FIG> depicts the first device <NUM> including the supplemental sensor(s) <NUM>, in some implementations the supplemental sensors <NUM> are omitted or are not used to generate input to the trained autonomous agent <NUM>. For example, the position data <NUM> may be determined solely from output of the position encoders <NUM>.

Although <FIG> depicts the first device <NUM> including the image processor(s) <NUM>, in other implementations, the image(s) <NUM> are not pre-processed by the image processor(s) <NUM> before they are provided as input to the feature extraction model <NUM>.

Although <FIG> depicts the maneuvering system <NUM>, the GUI engine <NUM>, and the display <NUM> offboard the first device <NUM>, in other implementation, one or more of the maneuvering system <NUM>, the GUI engine <NUM>, or the display <NUM> are onboard the first device <NUM>.

Although the segmentation model <NUM>, the downsampler <NUM>, and the feature extraction model <NUM> are depicted as separate components in <FIG>, in other implementations the described functionality of two or more of the segmentation model <NUM>, the downsampler <NUM>, and the feature extraction model <NUM> can be performed by a single component. In some implementations, one or more of the segmentation model <NUM>, the downsampler <NUM>, and the feature extraction model <NUM> can be represented in hardware, such as via an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). In some implementations, the operations described with reference to one or more of the segmentation model <NUM>, the downsampler <NUM>, and the feature extraction model <NUM> are performed by a single processor (e.g., the processor(s) <NUM> or the image processor(s) <NUM>) or by multiple processors using serial operations, parallel operations, or combinations thereof.

<FIG> is a diagram that illustrates a system <NUM> configured to generate a proposed maneuver to mate a first connector with a second connector based on image and position data. The system <NUM> is a specific, non-limiting example of the system <NUM>. In the example of <FIG>, the first device <NUM> corresponds to a tanker aircraft 102A, and the moveable coupling system <NUM> corresponds to a refueling boom 104A. In this example, the first connector <NUM> corresponds to a refueling connector 106A of the refueling boom 104A. Further, the second device <NUM> corresponds to a receiving aircraft 112A, and the second connector <NUM> corresponds to the receiving receptacle 116A of the receiving aircraft 112A.

In <FIG>, the tanker aircraft 102A includes the sensor(s) <NUM> and the camera <NUM>, each of which operates as described with reference to <FIG>. To illustrate, the sensor(s) <NUM> may include the position encoders <NUM> that generate the position data <NUM> indicating a position of the refueling boom 104A or of the refueling connector 106A in a frame of reference of the tanker aircraft 102A.

Additionally, the tanker aircraft 102A includes a fuel tank <NUM> to supply fuel, via the refueling boom 104A, to the receiving aircraft 112A. The tanker aircraft 102A also includes a flight control system <NUM> coupled to one or more flight instruments <NUM>. The flight control system <NUM> is configured to control or facilitate control of flight operations of the tanker aircraft 102A. The tanker aircraft 102A also includes a boom controller <NUM> to control or facilitate control of the refueling boom 104A. The flight instrument(s) <NUM>, the flight control system <NUM>, the boom controller <NUM>, or a combination thereof, are coupled to aircraft sensors <NUM> to receive flight data, which may be used by the trained autonomous agent <NUM> to generate the proposed maneuver <NUM> or may be used by the maneuver limiter <NUM> to limit the propose maneuver <NUM>. In some implementations, the flight control system <NUM>, the boom controller <NUM>, or both, include, correspond to, or are included within the maneuvering system <NUM> of <FIG>.

The camera <NUM> of the tanker aircraft 102A is positioned to capture the image(s) <NUM> depicting at least a portion of the refueling boom 104A and the receiving aircraft 112A during an air-to-air refueling operation. <FIG> is a diagram that illustrates a particular example of a display 152A generated based on an image from the camera <NUM> of <FIG>. In <FIG>, the display 152A is a particular example of the display <NUM> of <FIG>. For example, the display 152A depicts a portion of the receiving aircraft 112A and a portion of the refueling boom 104A. In this specific example, the display 152A also depicts the refueling connector 106A and the refueling receptacle 116A of the receiving aircraft 112A. Also, in the example illustrated in <FIG>, the display 152A includes a graphical element <NUM> indicating a proposed maneuver <NUM>. In the example illustrated in <FIG>, the proposed maneuver <NUM> suggests moving the refueling connector 106A to the left (in the frame of reference illustrated).

<FIG> are diagrams that illustrate various aspects of image processing and feature detection by the system <NUM> of <FIG> according to some implementations. In particular, <FIG> illustrates an example of a source image <NUM> that depicts the receiving aircraft 112A of <FIG>. <FIG> depicts an example <NUM> of key points, such as representative key points <NUM> and <NUM>, detected by the feature extraction model <NUM> of <FIG>. In the example <NUM> of <FIG>, the key point <NUM> is an example of a misplaced key point which should be placed at a location <NUM> based on the known geometry <NUM> of the receiving aircraft 112A. <FIG> depicts an example <NUM> in which the key point <NUM> of <FIG> has been repositioned based on the known geometry <NUM> of the receiving aircraft 112A. <FIG> illustrates an example <NUM> in which the source image <NUM> has been processed by the segmentation model <NUM> to generate a segmentation map <NUM>. For example, in <FIG>, various segments, such as a representative segment <NUM>, of the source image <NUM> are differentiated by different fill patterns.

<FIG> is a diagram that illustrates a system <NUM> configured to generate a proposed maneuver to mate a first connector with a second connector based on image data and position data. The system <NUM> is a specific, non-limiting example of the system <NUM>. In the example of <FIG>, the first device <NUM> corresponds to a first spacecraft 102B, and the moveable coupling system <NUM> corresponds to a docking arm 104B. In this example, the first connector <NUM> corresponds to a docking connector 106B of the docking arm 104B. Further, the second device <NUM> corresponds to a second spacecraft 112B, and the second connector <NUM> corresponds to a docking connector 116B of the second spacecraft 112B.

In <FIG>, the first spacecraft 102B includes the sensor(s) <NUM> and the camera <NUM>, each of which operates as described with reference to <FIG>. To illustrate, the sensor(s) <NUM> may include the position encoders <NUM> that generate the position data <NUM> indicating a position of the docking arm 104B or of the docking connector 106B relative to a frame of reference of the first spacecraft 102B.

The first spacecraft 102B includes a control system <NUM> coupled to one or more instruments <NUM>. The control system <NUM> is configured to control or facilitate control of operations of the first spacecraft 102B. The first spacecraft 102B also includes a docking controller <NUM> to control or facilitate control of the docking arm 104B. The control system <NUM>, the docking arm 104B, or both, are coupled to sensors <NUM> to receive data, which may be used by the trained autonomous agent <NUM> to generate the proposed maneuver <NUM> or may be used by the maneuver limiter <NUM> to limit the proposed maneuver <NUM>. In some implementations, the control system <NUM>, the docking controller <NUM>, or both, include, correspond to, or are included within the maneuvering system <NUM> of <FIG>.

The camera <NUM> of the first spacecraft 102B is positioned to capture the image(s) <NUM> depicting at least a portion of the docking arm 104B and the second spacecraft 112B during a docking operation. <FIG> is a diagram that illustrates a particular example of a display 152B generated based on an image from the camera <NUM> of <FIG>. In <FIG>, the display 152B is a particular example of the display <NUM> of <FIG>. For example, the display 152B depicts a portion of the second spacecraft 112B and a portion of the docking arm 104B. In this specific example, the display 152B also depicts the docking connector 106B and the docking connector 116B of the second spacecraft 112B. Also, in the example illustrated in <FIG>, the display 152B includes a graphical element <NUM> indicating a proposed maneuver <NUM>. In the example illustrated in <FIG>, the proposed maneuver <NUM> suggests moving the docking connector 106B to the left (in the frame of reference illustrated).

<FIG> are diagrams that illustrate various aspects of image processing and feature detection by the system <NUM> of <FIG> in accordance with some implementations. In particular, <FIG> illustrates an example of a source image <NUM> that depicts the second spacecraft 112B of <FIG>. <FIG> depicts an example <NUM> of key points, including representative key points <NUM> and <NUM>, detected by the feature extraction model <NUM> of <FIG>. In the example <NUM> of <FIG>, the key point <NUM> is an example of a misplaced key point which should be placed at a location <NUM> based on the known geometry <NUM> of the second spacecraft 112B. <FIG> depicts an example <NUM> in which the key point <NUM> of <FIG> has been repositioned based on the known geometry <NUM> of the second spacecraft 112B. <FIG> illustrates an example <NUM> in which the source image <NUM> has been processed by the segmentation model <NUM> to generate a segmentation map <NUM>. For example, in <FIG>, various segments of the source image <NUM>, such as a representative segment <NUM>, are differentiated by different fill patterns.

<FIG> is a flowchart of an example of a method <NUM> of generating a proposed maneuver to mate a first connector with a second connector based on image data and position data. The method <NUM> may be performed by the first device <NUM> or one or more components thereof, such as by the image processor(s) <NUM>, the processor(s) <NUM>, the downsampler <NUM>, the segmentation model <NUM>, the feature extraction model <NUM>, the trained autonomous agent <NUM>, or a combination thereof.

The method <NUM> includes, at <NUM>, providing a first image as input to a feature extraction model to generate feature data. The first image depicts a first portion of a first device and a second portion of a second device. For example, one or more of the image(s) <NUM>, one or more of the downsampled image(s) <NUM>, or both, depict the first portion <NUM> and the second portion <NUM> and may be provided as input to the feature extraction model <NUM> of <FIG>. The feature data generated by the feature extraction model includes first coordinates representing key points of the first device depicted in the first image and includes second coordinates representing key points of the second device depicted in the first image. For example, the feature data <NUM> may indicate the first coordinates <NUM> representing the key points <NUM> depicted in the image <NUM> and may indicate the second coordinates <NUM> representing the key points <NUM> depicted in the image <NUM>.

The method <NUM> also includes, at <NUM>, obtaining, from one or more sensors onboard the first device, position data indicating a position in 3D space of a first connector of the first device, where the first connector is disposed on the first portion of the first device. For example, the sensor(s) <NUM> generate the position data <NUM>, which indicates the position of the first connector <NUM> in a frame of reference of the first device <NUM>.

The method <NUM> further includes, at <NUM>, providing the feature data and the position data as input to a trained autonomous agent to generate a proposed maneuver to mate the first connector with a second connector of the second device. For example, the processor(s) <NUM> provide the feature data <NUM> and the position data <NUM> as input to the trained autonomous agent <NUM> to generate the proposed maneuver <NUM> to mate the first connector <NUM> and the second connector <NUM>.

<FIG> is a flowchart of another example of a method <NUM> of generating a proposed maneuver to mate a first connector with a second connector based on image data and position data. The method <NUM> may be performed by the first device <NUM> or one or more components thereof, such as by the image processor(s) <NUM>, the processor(s) <NUM>, the downsampler <NUM>, the segmentation model <NUM>, the feature extraction model <NUM>, the trained autonomous agent <NUM>, or a combination thereof.

The method <NUM> includes, at <NUM>, receiving a source image. For example, the source image may include or correspond to one of the image(s) <NUM> captured by the camera <NUM>.

The method <NUM> includes, at <NUM>, downsampling the source image one or more times to generate a first image. For example, the downsampler <NUM> may downsample one of the image(s) <NUM> to generate one of the downsampled image(s) <NUM> as the first image. In some examples, the downsampler <NUM> downsamples the source image more than one time, such as two or more times, to generate the first image. To illustrate, the downsampler <NUM> may downsample the source image one or more times to generate the first image and may further downsample the first image one or more additional times to generate a second image.

The method <NUM> includes, at <NUM>, providing at least one of the images (e.g., the source image, the first image, the second image, or a combination thereof), as input to a feature extraction model to generate feature data. In some examples, the method <NUM> includes generating a segmentation map based on at least one of the images and providing the segmentation map as input to the feature extraction model as well. For example, one or more of the image(s) <NUM>, one or more of the downsampled image(s) <NUM>, the segmentation map <NUM>, or combination thereof, are provided as input to the feature extraction model <NUM> of <FIG>. In this example, the feature data generated by the feature extraction model <NUM> may indicate the first coordinates <NUM> representing the key points <NUM> depicted in the image <NUM> and the second coordinates <NUM> representing the key points <NUM> depicted in the image <NUM>.

The method <NUM> includes, at <NUM>, obtaining position data indicating a position in 3D space of a first connector of the first device. For example, the sensor(s) <NUM> may generate the position data <NUM>, which indicates the position of the first connector <NUM> in a frame of reference of the first device <NUM>.

The method <NUM> includes, at <NUM>, performing a comparison of the coordinates representing the key points of the second device to known geometry of the second device. For example, the second coordinates <NUM> of the key points <NUM> may be compared to the known geometry <NUM> of the second device <NUM> to determine whether any key points are misplaced.

The method <NUM> includes, at <NUM>, modifying the feature data based on the comparison. For example, the feature data <NUM> may be modified in response to determining, based on the comparison of the known geometry <NUM> and the second coordinates <NUM>, that one or more of the key points <NUM> is misplaced. In this example, the misplaced key points <NUM> may be repositioned in the feature data <NUM> or omitted from the feature data <NUM>.

The method <NUM> includes, at <NUM>, providing the feature data and the position data as input to a trained autonomous agent to generate a proposed maneuver to mate the first connector with a second connector of the second device. For example, the processor(s) <NUM> may provide the feature data <NUM> and the position data <NUM> as input to the trained autonomous agent <NUM> to generate the proposed maneuver <NUM> to mate the first connector <NUM> and the second connector <NUM>. In some examples, the position data <NUM> includes supplemental sensor data from the supplemental sensors <NUM>, which is also provided as input to the trained autonomous agent <NUM>.

The method <NUM> includes, at <NUM>, generating a maneuver limit output if a maneuver limit criterion is satisfied. For example, the maneuver limiter <NUM> of <FIG> may determine whether the maneuver limit criterion is satisfied and may generate the maneuver limit output in response to the maneuver limit criterion being satisfied. In some examples, the maneuvering limit output is also provided as input to the trained autonomous agent to limit the maneuvers that can be selected as the proposed maneuver.

The method <NUM> includes, at <NUM>, generating a GUI with a graphical element indicating the proposed maneuver, commanding repositioning of the first device based on the proposed maneuver, or both. For example, the proposed maneuver <NUM> of <FIG> may be provided to the GUI engine <NUM> which generates the display <NUM>. In this example, the display <NUM> includes the graphical element(s) representing the proposed maneuver <NUM>. As another example, the proposed maneuver <NUM> may be provided to the maneuvering system <NUM> which may cause the first device <NUM> or a portion thereof (such as the moveable coupling system <NUM>) to execute the proposed maneuver <NUM>.

<FIG> is a flowchart of an example of a method <NUM> of training an autonomous agent to generate a proposed maneuver to mate a first connector with a second connector based on image data and position data. The method <NUM> may be performed by one or more processors onboard the first device <NUM> of <FIG> or offboard the first device <NUM>. For example, the method <NUM> may be performed at the image processor(s) <NUM>, the processor(s) <NUM>, or the processor(s) <NUM> of <FIG>.

The method <NUM> further includes, at <NUM>, providing the feature data and the position data as input to a machine-learning model to generate a proposed maneuver to mate the first connector with a second connector of the second device. For example, the feature data <NUM> and the position data <NUM> may be provided as input to the machine-learning model to generate the proposed maneuver <NUM> to mate the first connector <NUM> and the second connector <NUM>.

The method <NUM> further includes, at <NUM>, generating a reward valued based on the proposed maneuver. For example, the reward value may be determined using a value function of a reinforcement learning technique.

The method <NUM> further includes, at <NUM>, modifying a decision policy of the machine-learning model based on the reward value to generate a trained autonomous agent. For example, a decision policy that generates the proposed maneuver may be updated.

In some implementations, the image data, the position data, or other data used for training the machine-learning model to generate the trained autonomous agent is simulated or prerecorded. For example, a simulator of the system <NUM> may be used to generate the image data, the position data, or both.

<FIG> is a flowchart of an example of a lifecycle <NUM> of an aircraft configured to generate a proposed maneuver to mate a first connector with a second connector based on image data and position data. During pre-production, the exemplary lifecycle <NUM> includes, at <NUM>, specification and design of an aircraft, such as the tanker aircraft 102A described with reference to <FIG>. During specification and design of the aircraft, the lifecycle <NUM> may include specification and design of the feature extraction model <NUM>, the trained autonomous agent <NUM>, or both. At <NUM>, the lifecycle <NUM> includes material procurement, which may include procuring materials for the feature extraction model <NUM>, the trained autonomous agent <NUM>, or both.

During production, the lifecycle <NUM> includes, at <NUM>, component and subassembly manufacturing and, at <NUM>, system integration of the aircraft. For example, the lifecycle <NUM> may include component and subassembly manufacturing of the feature extraction model <NUM>, the trained autonomous agent <NUM>, or both, and system integration of the feature extraction model <NUM>, the trained autonomous agent <NUM>, or both. At <NUM>, the lifecycle <NUM> includes certification and delivery of the aircraft and, at <NUM>, placing the aircraft in service. Certification and delivery may include certification of the feature extraction model <NUM>, the trained autonomous agent <NUM>, or both, to place the feature extraction model <NUM>, the trained autonomous agent <NUM>, or both, in service. While in service by a customer, the aircraft may be scheduled for routine maintenance and service (which may also include modification, reconfiguration, refurbishment, and so on). At <NUM>, the lifecycle <NUM> includes performing maintenance and service on the aircraft, which may include performing maintenance and service on the feature extraction model <NUM>, the trained autonomous agent <NUM>, or both.

Each of the processes of the lifecycle <NUM> may be performed or carried out by a system integrator, a third party, and/or an operator (e.g., a customer). For the purposes of this description, a system integrator may include without limitation any number of aircraft manufacturers and major-system subcontractors; a third party may include without limitation any number of venders, subcontractors, and suppliers; and an operator may be an airline, leasing company, military entity, service organization, and so on.

Aspects of the disclosure can be described in the context of an example of a vehicle, such as a spacecraft or an aircraft. A particular example of a vehicle is an aircraft <NUM> as shown in <FIG>.

In the example of <FIG>, the aircraft <NUM> includes an airframe <NUM> with a plurality of systems <NUM> and an interior <NUM>. Examples of the plurality of systems <NUM> include one or more of a propulsion system <NUM>, an electrical system <NUM>, an environmental system <NUM>, and a hydraulic system <NUM>. Any number of other systems may be included. The systems <NUM> of <FIG> also include the processor(s) <NUM>, the feature extraction model <NUM>, and the trained autonomous agent <NUM>.

<FIG> is a block diagram of a computing environment <NUM> including a computing device <NUM> configured to support aspects of computer-implemented methods and computer-executable program instructions (or code) according to the present disclosure. For example, the computing device <NUM>, or portions thereof, is configured to execute instructions to initiate, perform, or control one or more operations described with reference to <FIG>.

The computing device <NUM> includes one or more processors <NUM>. The processor(s) <NUM> are configured to communicate with system memory <NUM>, one or more storage devices <NUM>, one or more input/output interfaces <NUM>, one or more communications interfaces <NUM>, or any combination thereof. In some implementations, the processor(s) <NUM> correspond to, include, or are included within the image processor(s) <NUM> or the processor(s) <NUM> of <FIG>.

The system memory <NUM> includes volatile memory devices (e.g., random access memory (RAM) devices), nonvolatile memory devices (e.g., read-only memory (ROM) devices, programmable read-only memory, and flash memory), or both. The system memory <NUM> stores an operating system <NUM>, which may include a basic input/output system for booting the computing device <NUM> as well as a full operating system to enable the computing device <NUM> to interact with users, other programs, and other devices. The system memory <NUM> stores system (program) data <NUM>, such as data representing the known geometry <NUM> of <FIG>.

The system memory <NUM> includes one or more applications <NUM> (e.g., sets of instructions) executable by the processor(s) <NUM>. As an example, the one or more applications <NUM> include instructions executable by the processor(s) <NUM> to initiate, control, or perform one or more operations described with reference to <FIG>. To illustrate, the one or more applications <NUM> include instructions executable by the processor(s) <NUM> to initiate, control, or perform one or more operations described with reference to the feature extraction model <NUM>, the trained autonomous agent <NUM>, or a combination thereof.

In a particular implementation, the system memory <NUM> includes a non-transitory, computer readable medium storing the instructions that, when executed by the processor(s) <NUM>, cause the processor(s) <NUM> to initiate, perform, or control operations to generate a proposed maneuver based on image data and position data. For example, the operations include providing a first image as input to a feature extraction model to generate feature data; obtaining position data indicating a position in three-dimensional space of a first connector of the first device; and providing the feature data and the position data as input to a trained autonomous agent to generate a proposed maneuver to mate a first connector with a second connector.

The one or more storage devices <NUM> include nonvolatile storage devices, such as magnetic disks, optical disks, or flash memory devices. In a particular example, the storage devices <NUM> include both removable and non-removable memory devices. The storage devices <NUM> are configured to store an operating system, images of operating systems, applications (e.g., one or more of the applications <NUM>), and program data (e.g., the program data <NUM>). In a particular aspect, the system memory <NUM>, the storage devices <NUM>, or both, include tangible computer-readable media. In a particular aspect, one or more of the storage devices <NUM> are external to the computing device <NUM>.

The one or more input/output interfaces <NUM> enable the computing device <NUM> to communicate with one or more input/output devices <NUM> to facilitate user interaction. For example, the one or more input/output interfaces <NUM> can include the GUI engine <NUM> of <FIG>, a display interface, an input interface, or both. For example, the input/output interface <NUM> is adapted to receive input from a user, to receive input from another computing device, or a combination thereof. In some implementations, the input/output interface <NUM> conforms to one or more standard interface protocols, including serial interfaces (e.g., universal serial bus (USB) interfaces or Institute of Electrical and Electronics Engineers (IEEE) interface standards), parallel interfaces, display adapters, audio adapters, or custom interfaces ("IEEE" is a registered trademark of The Institute of Electrical and Electronics Engineers, Inc. of Piscataway, New Jersey). In some implementations, the input/output device <NUM> includes one or more user interface devices and displays, including some combination of buttons, keyboards, pointing devices, displays, speakers, microphones, touch screens, and other devices.

The processor(s) <NUM> are configured to communicate with devices or controllers <NUM> via the one or more communications interfaces <NUM>. For example, the one or more communications interfaces <NUM> can include a network interface. The devices or controllers <NUM> can include, for example, the maneuvering system <NUM> of <FIG>, one or more other devices, or any combination thereof.

In conjunction with the described systems and methods, an apparatus includes means for providing a first image as input to a feature extraction model to generate feature data. In some implementations, the means for providing a first image as input to a feature extraction model corresponds to the first device <NUM>, the camera <NUM>, the image processor(s) <NUM>, the downsampler <NUM>, the processor(s) <NUM>, one or more other circuits or devices configured to provide an image as input to a feature extraction model, or a combination thereof.

The apparatus also includes means for obtaining position data indicating a position in three-dimensional space of a first connector of the first device. For example, the means for obtaining position data can correspond to the first device <NUM>, the sensor(s) <NUM>, the position encoders <NUM>, the processor(s) <NUM>, one or more other devices configured to obtain position data, or a combination thereof.

The apparatus also includes means for providing the feature data and the position data as input to a trained autonomous agent to generate a proposed maneuver to mate the first connector with a second connector of a second device. For example, the means for providing the feature data and the position data as input to a trained autonomous agent can correspond to the first device <NUM>, the feature extraction model <NUM>, the processor(s) <NUM>, one or more other devices configured to provide the feature data and the position data as input to a trained autonomous agent, or a combination thereof.

Claim 1:
A method for generating maneuvers for mating connectors of two different devices. (<NUM>, <NUM>) comprising
receiving a source image and downsampling (<NUM>) the source image one or more times to generate a first image (<NUM>);
providing the first image to a segmentation model (<NUM>) to generate a segmentation map (<NUM>) representing distinct regions (<NUM>, <NUM>) of the first image;
providing (<NUM>, <NUM>) the first image (<NUM>) and the segmentation map as inputs to the feature extraction model (<NUM>) to generate feature data (<NUM>), wherein the first image depicts a first portion (<NUM>) of a first device (<NUM>) and a second portion (<NUM>) of a second device (<NUM>), wherein the feature data includes first coordinates (<NUM>) representing key points (<NUM>) of the first device depicted in the first image and includes second coordinates (<NUM>) representing key points (<NUM>) of the second device depicted in the first image;
obtaining (<NUM>, <NUM>), from one or more sensors (<NUM>) onboard the first device, position data (<NUM>) indicating a position in three-dimensional space (<NUM>) of a first connector (<NUM>) of the first device, wherein the first connector is disposed on the first portion of the first device, wherein the position data indicates the position in a reference frame that is associated with the first device and is independent of the second device;
performing (<NUM>) a comparison of the second coordinates representing the key points of the second device to known geometry (<NUM>) of the second device;
modifying (<NUM>) the feature data based on the comparison;
providing (<NUM>, <NUM>) the feature data and the position data as input to a trained autonomous agent (<NUM>) to generate a proposed maneuver (<NUM>) to mate the first connector with a second connector of the second device; and
commanding (<NUM>) repositioning of the first portion of the first device based on the proposed maneuver.