Patent Description:
Some arrangements use stereoscopic vision with dual cameras, in which the human operator wears goggles that provide a three-dimensional (3D) view based on the views from the dual cameras. Some other arrangements use light detection and ranging (LIDAR) or radar to provide supplemental range measurements for the human operator. These latter types of arrangements require additional expensive components.

<CIT>, in accordance with its abstract, states: A system is provided. The system includes a camera on a refueling aircraft to obtain an image of a receiver aircraft having a receptacle for refueling the receiver aircraft. The system also includes a processor on the refueling aircraft to determine an orientation and position of the receiver aircraft relative to the camera, a separation distance and respective positions of a refueling boom of the refueling aircraft and the receptacle. The processor also generates a display of a side perspective view of 3D models of the refueling boom and the receiver aircraft having the receptacle. The display of the side perspective view illustrates a current position of the refueling boom relative to a. current position of the receptacle and enables an operator on the refueling aircraft to observe and guide the refueling boom to the receptacle.

<CIT>, in accordance with its abstract, states: In various examples, live perception from sensors of a vehicle may be leveraged to generate potential paths for the vehicle to navigate an intersection in real-time or near real-time. For example, a deep neural network (DNN) may be trained to compute various outputs-such as heat maps corresponding to key points associated with the intersection, vector fields corresponding to directionality, heading, and offsets with respect to lanes, intensity maps corresponding to widths of lanes, and/or classifications corresponding to line segments of the intersection. The outputs may be decoded and/or otherwise post-processed to reconstruct an intersection-or key points corresponding thereto-and to determine proposed or potential paths for navigating the vehicle through the intersection.

The paper <NPL>, presents results of using visual information to estimate the relative motion of the aircrafts in Autonomous Air-to-Air Refuelling tasks for unmanned aerial vehicles.

According to the present disclosure, a method and a system as defined in the independent claims are provided. Further embodiments of the invention are defined in the dependent claims. Although the invention is only defined by the claims, the below embodiments, examples, and aspects are present for aiding in understanding the background and advantages of the invention. The following summary is provided to illustrate examples or implementations disclosed herein.

Aspects of the disclosure provide solutions for automated air-to-air refueling (A3R) and assisted air-to-air refueling. Examples include: receiving a video frame; generating, from the video frame, a plurality of images having differing decreasing resolutions; detecting, within each of the plurality of images, a set of aircraft keypoints for an aircraft to be refueled; merging the sets of aircraft keypoints into a set of merged aircraft keypoints; based on at least the merged aircraft keypoints, determining a position of a fuel receptacle on the aircraft; and determining a position of a boom tip of an aerial refueling boom. Some examples include, based on at least the position of the fuel receptacle and the position of the boom tip, controlling the aerial refueling boom to engage the fuel receptacle, and for some examples, the video frame is monocular (e.g., provided by a single camera).

Corresponding reference characters indicate corresponding parts throughout the drawings in accordance with an example.

The various examples will be described in detail with reference to the accompanying drawings. References made throughout this disclosure relating to specific examples and implementations are provided solely for illustrative purposes but, unless indicated to the contrary, are not meant to limit all implementations.

The foregoing summary, as well as the following detailed description of certain implementations will be better understood when read in conjunction with the appended drawings. Further, references to an implementation or an example are not intended to be interpreted as excluding the existence of additional examples that also incorporate the recited features. Moreover, unless explicitly stated to the contrary, examples "comprising" or "having" an element or a plurality of elements having a particular property could include additional elements not having that property.

Aspects of the disclosure have a technical effect of improved operation of a computer, for example by reducing distance calculations, in image processing, to thereby reduce computation time and processing expense. Examples herein improve the efficiency of computational hardware, and provide better allocation of resources, as compared to traditional systems that rely on, for example processing many different measurement inputs.

Aspects of the disclosure are able to estimate the position of a three-dimensional object (e.g., an aircraft fuel receptacle) in a video stream collected by a single camera, such as in support of autonomous aerial refueling operations and/or human-assisted aerial refueling operations. For example, aspects of the disclosure locate the relative positions of an aircraft fuel receptacle and a refueling platform's refueling boom in order to automate control of the refueling boom during refueling. In some examples, position and pose information is represented as six degrees-of-freedom (6DoF) including the three-dimensional (3D) position (x, y, and z coordinates) and orientation (roll, pitch, and yaw).

The location occurs in stages, such as by generating a pyramid representation from a two-dimensional (2D) video frame to produce a plurality of images having differing decreasing resolutions, detecting a set of aircraft keypoints (for the aircraft to be refueled) within each of the plurality of images, merging the sets of aircraft keypoints, and determining a three-dimensional (3D) position of a fuel receptacle on the aircraft based using the merged aircraft keypoints. The detection of sets of aircraft keypoints at differing resolutions, followed by merging those results, provides for higher accuracy. Multi-stage pose estimation pipelines use real-time deep learning-based detection algorithms, for example, a neural network (NN) such as a deep convolutional neural network (CNN), which may be a residual neural network (ResNet). This provides accurate detection and tracking under adverse weather and lighting conditions which can be used for autonomous aerial (air-to-air) refueling, and/or an operator feedback loop. The use of a single camera can reduce component failures and be more easily integrated into existing systems.

Referring more particularly to the drawings, <FIG> illustrates an arrangement <NUM> that includes a refueling platform <NUM> and an aircraft <NUM> to be refueled. Each of refueling platform <NUM> and aircraft <NUM> may be an example of a flying apparatus <NUM>, described in further retail in relation to <FIG> and <FIG>. In the arrangement <NUM>, the refueling platform <NUM> uses an aerial refueling boom <NUM> to refuel the aircraft <NUM>.

A camera <NUM> provides a video stream 202a (shown in <FIG>) for use in determining a positions of a boom tip <NUM> (shown in <FIG>) of the aerial refueling boom <NUM> and a fuel receptacle <NUM> (shown in <FIG>) of the aircraft <NUM>. A proximity sensor <NUM> (e.g., a light detection and ranging (lidar) or radar) is also shown. For some examples of the arrangement <NUM>, a computer vision (CV) architecture <NUM> (shown in <FIG>) fuses proximity sensor measurements from the proximity sensor <NUM> with extracted features of the aircraft <NUM> to determining a position of the fuel receptacle <NUM> on the aircraft <NUM>.

<FIG> illustrates a representative video frame <NUM>, which is a frame from the video stream 202a, captured by the camera <NUM>. For clarity, <FIG> shows only a clean version of the video frame <NUM>. <FIG> provides an annotated version of the video frame <NUM>, identifying various elements such as the aircraft <NUM> and the aerial refueling boom <NUM>. An expended view section identifies the boom tip <NUM>, the fuel receptacle <NUM>, and a fiducial marker <NUM> that outlines the fuel receptacle <NUM>. In operation, the aerial refueling boom <NUM> delivers fuel to the aircraft <NUM> by the boom tip <NUM> engaging the fuel receptacle <NUM>.

The fiducial marker <NUM> has a defined location relative to the fuel receptacle <NUM> that facilitates location of the fuel receptacle <NUM> on the aircraft <NUM>. In some examples, the fiducial marker <NUM> may be used by elements of the CV architecture <NUM>, specifically a CNN within an aircraft position estimation pipeline <NUM> (shown in <FIG> and <FIG>) as a reference aid for locating the fuel receptacle <NUM>. The video frame <NUM> also shows a boom tip bounding box <NUM>, which is used to crop the video frame <NUM> to the area around the boom tip <NUM> for image-based location of the boom tip <NUM>. A bounding box may also be used for cropping the video frame <NUM> to the area around the aircraft <NUM> in an early stage of the aircraft position estimation pipeline <NUM>.

<FIG> illustrates the CV architecture <NUM> that determines a position of the fuel receptacle <NUM> on the aircraft <NUM> for the arrangement <NUM>. Various components of the CV architecture <NUM> are shown with further detail in <FIG>, <FIG>, and <FIG>, and the operation of CV architecture <NUM> is described in further detail in relation to <FIG> and <FIG>, using flowcharts <NUM> and <NUM>, respectively. In some examples, the entirety of the CV architecture <NUM> resides onboard the refueling platform <NUM>. In some examples, the portions of the CV architecture <NUM> operate remotely, off of the refueling platform <NUM>. The CV architecture <NUM> receives the video stream 202a of the aircraft <NUM> from the camera <NUM>. The video stream 202a includes the video frame <NUM> and a plurality of additional video frames 202b. Operation of the CV architecture <NUM> is described in relation to processing the video frame <NUM>. Processing of each of the plurality of additional video frames 200b is similar to that for the video frame <NUM>.

The CV architecture <NUM> includes the aircraft position estimation pipeline <NUM> and a boom tip position estimation pipeline <NUM>. The aircraft position estimation pipeline <NUM> is shown and described in further detail in relation to <FIG>. The boom tip position estimation pipeline <NUM> is shown and described in further detail in relation to <FIG>. The aircraft position estimation pipeline <NUM> receives the video stream 202a and outputs a fuel receptacle position <NUM>. In some examples, the fuel receptacle position <NUM> is provided as 6DoF. In some examples, the aircraft position estimation pipeline <NUM> also outputs an aircraft position <NUM>, which may also be provided as 6DoF. The fuel receptacle position <NUM> is derivable from the aircraft position <NUM>, because the position of the fuel receptacle <NUM> on the aircraft <NUM> is fixed and known. The boom tip position estimation pipeline <NUM> outputs a boom tip position <NUM>, which may be provided as 6DoF.

The fuel receptacle position <NUM> and the boom tip position <NUM> are provided to a tracking logic <NUM> that determines a distance <NUM> between the boom tip <NUM> and the fuel receptacle <NUM>, which are both shown in <FIG>. The tracking logic <NUM> determines boom control parameters <NUM>, which are provided to a boom control <NUM> that autonomously moves the aerial refueling boom <NUM> to position the boom tip <NUM> to engage the fuel receptacle <NUM>. That is, boom control <NUM> controls the aerial refueling boom <NUM> to engage the fuel receptacle <NUM>. In some examples, the tracking logic <NUM> also determines whether controlling the aerial refueling boom <NUM> to engage the fuel receptacle <NUM> is within safety parameters <NUM>, and if not, generates an alert <NUM>. In some examples, the tracking logic <NUM> also generates and provides, to the aircraft <NUM> (e.g., for a pilot or to an unmanned aerial vehicle (UAV)), maneuvering information <NUM> to facilitate engaging the fuel receptacle <NUM> with the aerial refueling boom <NUM> and/or to avoid an unsafe condition.

Safety parameters <NUM> includes a set of rules, conditions, and/or measurement values that provide boundaries for safe operation, such as to reduce risk of damage to the aircraft <NUM>, the aerial refueling boom <NUM>, and/or the refueling platform <NUM>. Examples of safety parameters <NUM> include limits on closing rates that vary by distance (e.g., when the boom tip <NUM> is close to the fuel receptacle <NUM>, the closing rate must be slower than when the boom tip <NUM> is further from the fuel receptacle <NUM>), the boom tip <NUM> must not be closer than some minimum distance to any part of the aircraft <NUM> except for the fuel receptacle <NUM>, and the angles at which the aerial refueling boom approaches the aircraft must be within some defined range. Other parameters may also be used.

Boom control parameters <NUM> include information (e.g., variables) that describe how the aerial refueling boom <NUM> may move (e.g., roll, pitch, yaw, translate, telescope, extend, retract, pivot, rotate, and the like) and may include limits and rates of such movement. The boom control parameters <NUM> may control the aerial refueling boom <NUM> given constraints of the boom pivot position and camera intrinsic and extrinsic parameters (e.g., camera parameters <NUM>, shown in <FIG>), for example, how to rotate the aerial refueling boom <NUM> (roll and pitch) and telescopically extend the aerial refueling boom <NUM> so that the 3D position of the boom tip <NUM> will be projected onto the video frame <NUM> where the boom tip keypoint <NUM> (also shown in <FIG>) is detected.

In some examples, a video compilation <NUM> overlays an aircraft model projection <NUM> and/or a boom model projection onto the video frame <NUM> to produce an overlaid video frame <NUM>. An example video frame <NUM> is shown in <FIG>. In some examples, the video frame <NUM> and/or the alert <NUM> are provided to a human operator <NUM> over presentation components <NUM> (e.g., by displaying the video frame <NUM> on a video monitor screen). In some examples, the human operator <NUM> uses input/output (I/O) components <NUM> (e.g., a joystick, mouse, keyboard, touchscreen, keypad, and/or other input devices) to provide the boom control parameters <NUM> to control the aerial refueling boom <NUM> to position the boom tip <NUM> to engage the fuel receptacle <NUM>.

<FIG> illustrates further detail for the aircraft position estimation pipeline <NUM>, showing the aircraft position estimation pipeline <NUM> as comprising four stages: a stage <NUM>,a stage <NUM>, a stage <NUM>, and a stage <NUM>. In the stage <NUM>, the video frame <NUM> is provided (as part of the video stream 202a) to a feature extraction function <NUM>, which is illustrated and described in further detail in relation to <FIG>.

The feature extraction function <NUM> outputs an aircraft keypoint heatmap <NUM> containing a set of aircraft keypoints <NUM>. A heatmap is a graphical representation of data that uses a system of color-coding to represent different values. Heatmap pixel values indicate, for each keypoint, the likelihood of a 3D object's keypoint being found at each pixel location of the image. In some examples, the keypoints are not represented as binary points, but rather as probabilistic distributions. That is, each of the keypoints corresponds to a region of pixels, with the values of the pixels dropping according to a probability density function (pdf), with increasing distance from the center of the region. In some examples, the maximum value of a pixel, in a keypoint region of pixels, reflects a confidence level of that keypoint.

The set of aircraft keypoints <NUM> is provided to a correction and imputation <NUM> in the stage <NUM>. The correction and imputation <NUM> produces an adjusted version of the set of aircraft keypoints <NUM> shown in an adjusted version of the aircraft keypoint heatmap 312a. For example, the correction and imputation <NUM> determines whether an aircraft keypoint is missing from the set of aircraft keypoints <NUM>, and based on at least determining that an aircraft keypoint is missing, inserts an additional aircraft keypoint <NUM> into the aircraft keypoint heatmap <NUM>.

The correction and imputation <NUM> also determines whether an aircraft keypoint <NUM> requires correction (e.g., is in the wrong position and should be moved), and based on at least determining that the aircraft keypoint <NUM> requires correction, correcting the aircraft keypoint <NUM>. As illustrated, the aircraft keypoint <NUM> is shifted to the left in the aircraft keypoint heatmap <NUM>. In some examples, the correction and imputation <NUM> uses a machine learning (ML) component, such as a Neural Network (NN) to recognize when aircraft keypoint are missing or require correction. Aircraft keypoints may be missing due to obscuration by portions of the aircraft <NUM>, due to the viewing angle of the aircraft <NUM> by the camera <NUM>, or due to other poor visibility conditions. Aircraft keypoints may be shifted (e.g., require correction), due to glare, bright reflections from the aircraft <NUM>, or other perturbations to the view of the camera <NUM> that are manifest in the video frame <NUM>.

The stage <NUM> uses a filter <NUM> to performing temporal filtering of the set of set of aircraft keypoints <NUM>. In some examples, the temporal filtering comprises Kalman filtering that performs time-domain filtering across a time-series set of aircraft keypoint heatmaps 314a. Kalman filtering uses a series of measurements observed over time, containing statistical noise and other inaccuracies, and produces estimates of unknown variables that tend to be more accurate than those based on a single measurement alone, by estimating a joint probability distribution over the variables for each timeframe. In some examples, the filter <NUM> operates across video frames (e.g., the video frame <NUM> and the plurality of additional video frames 202b). In some examples, a threshold is applied to eliminate aircraft keypoints having a low confidence level.

In some examples, the filter <NUM> also fuses proximity sensor measurements <NUM> of the aircraft <NUM>, received from the proximity sensor <NUM>, with the set of aircraft keypoints <NUM>. In such examples, determining the fuel receptacle position <NUM> on the aircraft <NUM> comprises fusing the proximity sensor measurements <NUM> of the aircraft <NUM> with the set of aircraft keypoints <NUM>. To accomplish this, in some examples, the filter <NUM> uses a trained network (e.g., an NN) to incorporate the proximity information into the filtering process.

The adjusted and filtered aircraft keypoint heatmap <NUM> is provided to an aircraft 2D to 3D transform <NUM>. In some examples, the aircraft 2D to 3D transform <NUM> uses a perspective-n-point (PnP) algorithm. PnP algorithms estimate the pose of a calibrated camera relative to an object, given a set of N 3D points on the object and their corresponding 2D projections in an image collected by the camera. The PnP algorithm used leverages the correspondences between the 2D pixel locations of detected keypoints and 3D keypoint locations on an object model to rotate and position the object in space such that the camera's view of the 3D keypoints matches the 2D pixel locations.

The aircraft 2D to 3D transform <NUM> determines the aircraft position <NUM> of the aircraft <NUM> and, from that, the fuel receptacle position <NUM>. That is, once the aircraft position <NUM> is known, the fuel receptacle position <NUM>, which is in a predetermined location on the aircraft <NUM>, can be determined using a refine algorithm <NUM> that uses the known position of the fuel receptacle <NUM> on the aircraft <NUM>. In some examples, the fuel receptacle position <NUM> is filtered with a temporal filter (which may be a Kalman filter). In some examples, the aircraft 2D to 3D transform <NUM> also generates the aircraft model projection <NUM> that is used by the video compilation <NUM>. The aircraft model projection <NUM> is determined by rendering a 3D aircraft model according to the aircraft position <NUM>.

The feature extraction function <NUM> and the correction and imputation <NUM> are both initially trained offline, although some examples may employ on-going training during deployment. In some examples, the training uses approximately <NUM>,<NUM> labeled training images, generated by sweeping across ranges of 6DoF variations for a 3D aircraft model <NUM>. To increase variations in the datasets used for NN training, data augmentations may be randomly applied to each image passed to the NN. These augmentations include: brightness scaling, contrast scaling, image size scaling, and image translation, among others. Such augmentations may result in dropped and/or displaced keypoints that are beneficial for training the correction and imputation <NUM>.

<FIG> illustrates further detail for the feature extraction function <NUM>. The video frame <NUM> is iteratively blurred by a blur function <NUM> and downsampled by a downsampling <NUM> to generate a plurality of images <NUM> having differing decreasing resolutions. That is, the plurality of images <NUM> comprises multiple reduced-resolution images, each iteration resulting in a lower resolution. In some examples, image 408a has half the resolution of image 408b, which has half the resolution of image 408c. This results in a pyramid representation (e.g., a plurality of images <NUM> comprises a pyramid representation), which may be a Gaussian pyramid representation. In some examples, a different downsampling rate may be used, including non-integer reduction ratios. In some examples, a different number of downsampled images may be used.

A pyramid representation, is a type of multi-scale signal representation in which a signal or an image is subject to repeated smoothing and subsampling. A low pass pyramid is made by smoothing the image with an appropriate smoothing filter and then subsampling the smoothed image, often by a factor of <NUM> along each coordinate direction. The resulting image is then subjected to the same procedure, and the cycle is repeated multiple times. Each cycle of this process results in a smaller image with increased smoothing, but with decreased spatial sampling density (e.g., decreased image resolution).

When illustrated graphically, with each cycle's resulting smaller image stacked one atop the other, the multi-scale representation appears in the shape of a pyramid. In a Gaussian pyramid, subsequent images are weighted down using a Gaussian average (e.g., a Gaussian blur is used in the blur function <NUM>) and scaled down. Each pixel containing a local average corresponds to a neighborhood pixel on a lower level of the pyramid. In some examples, the video frame <NUM> is cropped to an aircraft bounding box, surrounding the aircraft <NUM>, prior to the generation of the plurality of images <NUM>, to exclude unnecessary sections of the video frame <NUM> from the keypoint detection process. This decreases computational time and allows the use of more computationally intensive algorithms.

A keypoint detector <NUM> detects, within each of the plurality of images <NUM>, the set of aircraft keypoints <NUM> for the aircraft <NUM>. This is illustrated as the keypoint detector <NUM> comprising a keypoint detector 410a that detects a set of aircraft keypoints 314a for the image 408a, a keypoint detector 410b that detects a set of aircraft keypoints 314b for the image 408b, and a keypoint detector 410c that detects a set of aircraft keypoints 314c for the image 408c. The keypoint detector <NUM> may be implemented using one or more NNs, such as CNNs, and in some examples, with a ResNet. Although the keypoint detector <NUM> is illustrated as comprising one keypoint detection function for each image, it should be understood that some NNs may perform keypoint detection on multiple images simultaneously.

The separate sets of aircraft keypoints 314a, 314b, and 314c are merged, by a merge function <NUM>, into a merged (composite) set of aircraft keypoints, which becomes the set of aircraft keypoints <NUM>. (That is, the set of aircraft keypoints <NUM> is a set of merged aircraft keypoints). One of the benefits of this approach is improved accuracy. In some scenarios, when an NN is trained on only a full-scale, high resolution image, the NN learns specific details of features rather than general attributes. Then, if these specific features are not present in use of the NN (e.g., during deployment), because of less-than-ideal image collection situations (e.g., glare, blur, flashed and bright reflections of sunlight), the NN underperforms because it is unable to locate the features it learned. However, by degrading the image via downsampling, the higher frequency content containing these specific feature details is lost, forcing the NN to learn the features more generally. Merging results from NNs (e.g., CNNs or even other types of networks), that had been trained with a set of reduced-resolutions images (even if accomplished prior to time-domain filtering) may significantly improve keypoint detection (e.g., feature extraction) performance.

<FIG> illustrates a video frame <NUM>, in which the aircraft model projection <NUM> is overlaid onto the video frame <NUM>, over the aircraft <NUM> (not seen here, because of the overlay) for display to the human operator <NUM>. The aircraft model projection <NUM> is generated using a 3D aircraft model (e.g., a computer-aided design (CAD) model), rendered according to the aircraft position <NUM>. In some examples, a boom model projection (based on a boom model <NUM> of <FIG>) is overlaid onto the video frame <NUM> in addition to or instead of the aircraft model projection <NUM>. For reference, the set of aircraft keypoints <NUM> is also shown in <FIG>, which may or may not be shown to the human operator <NUM>.

<FIG> illustrates further detail for the boom tip position estimation pipeline <NUM> in the CV architecture <NUM> of <FIG>. Different classes of operations are possible with the illustrated boom tip position estimation pipeline <NUM>. In one class of operation, the boom model <NUM>, the camera parameters <NUM> (e.g., extrinsic and intrinsic parameters for the camera <NUM>), and the boom control parameters <NUM> are input into a direct calculation <NUM> to calculate the boom tip position <NUM>, from the physical geometry of the aerial refueling boom <NUM> and the refueling platform <NUM>, rather than determining the boom tip position <NUM> from the video stream 202a. In some examples, the boom model <NUM> comprises a CAD model of the aerial refueling boom <NUM>.

Calculation of the boom tip position <NUM>, from the physical geometry of the aerial refueling boom <NUM> uses the known angles, extrinsics, and geometry of the aerial refueling boom <NUM> in relation to the camera <NUM> to determine a projection of the aerial refueling boom <NUM>. The pipeline <NUM> monitors each video frame <NUM> from the stream 202a and determines the pitch and roll states of the boom control <NUM>, the pitch and roll of the aerial refueling boom <NUM> in relation to the camera <NUM>. The intrinsics of the camera <NUM> and its position on the refueling platform <NUM> are known, enabling determination of the location of the aerial refueling boom <NUM> in the 2D pixel space of the camera <NUM>.

Camera parameter information includes the parameters used in a camera model to describe the mathematical relationship between the 3D coordinates of a point in the scene from which the light comes from and the 2D coordinates of its projection onto the image plane. Intrinsic parameters, also known as internal parameters, are the parameters intrinsic to the camera itself, such as the focal length and lens distortion. Extrinsic parameters, also known as external parameters or camera pose, are the parameters used to describe the transformation between the camera and its external world. The camera extrinsic information, resolution, magnification, and other intrinsic information are known.

In an alternative operation, the video stream 202a (including the video frame <NUM>) is input into a boom tip keypoint detector <NUM> (which may also make use of the boom model <NUM>, the camera parameters <NUM>, and the boom control parameters <NUM> for enhanced accuracy), which produces a boom tip keypoint heatmap <NUM>. In some examples, the boom tip keypoint detector <NUM> comprises an NN, for example a CNN (e.g., a ResNet or other type of network). In some examples, the video frame <NUM> is cropped to the boom tip bounding box <NUM> prior to being input into the boom tip keypoint detector <NUM>. The boom tip keypoint heatmap <NUM> has the boom tip keypoint <NUM>, as detected in the video frame <NUM>. In some examples, the boom tip keypoint <NUM> is time-domain filtered (e.g., with a Kalman filter or other type of temporal filter) across video frames (e.g., the video frame <NUM> and the plurality of additional video frames 200b). in some examples, the video frame <NUM> is converted into a pyramid representation, the boom tip keypoint <NUM> is found at each resolution, and the different resolution results are merged, similarly as described for the feature extraction function <NUM>.

The boom tip keypoint heatmap <NUM> is provided to a boom tip 2D to 3D transform <NUM>, which determines the boom tip position <NUM>. In some examples, the boom tip position <NUM> is filtered (e.g., with a Kalman filter or other type of temporal filter). In some examples, the boom tip 2D to 3D transform <NUM> also generates a boom model projection for use by the video compilation <NUM> (of <FIG>) to produce an overlay in the video frame <NUM>. The boom tip keypoint detector <NUM> is initially trained offline, although some examples may employ on-going training during deployment.

With reference now to <FIG>, a flowchart <NUM> illustrates a method of air-to-air refueling (e.g., A3R or human-assisted air-to-air refueling) which may be used with the arrangement <NUM> of <FIG>. In some examples, the operations illustrated in <FIG> are performed, at least in part, by executing instructions 902a (stored in the memory <NUM>) by the one or more processors <NUM> of the computing device <NUM> of <FIG>. For example, any of the feature extraction function <NUM>, the correction and imputation <NUM>, the filter <NUM>, the aircraft 2D to 3D transform <NUM>, the keypoint detector <NUM> (including the keypoint detectors 410a, 410b, and 410c), the boom tip keypoint detector <NUM>, and any other ML component of the CV architecture <NUM> may be trained on a first example of the computing device <NUM> and then deployed on a second (different) example of the computing device <NUM>.

Operation <NUM> includes training any networks any other ML components of the CV architecture <NUM>. In some examples, operation <NUM> is performed prior to deployment <NUM>, although in some examples, operation <NUM> remains ongoing during operational use of the CV architecture <NUM>. In some examples, operation <NUM> includes training an NN with a plurality of labeled images of a scene, the plurality of labeled images having differing decreasing resolutions of a common scene (e.g., similar to the plurality of images <NUM>). In some examples, operation <NUM> includes training an NN to insert an additional aircraft keypoint into a heatmap. Operation <NUM> includes receiving the video frame <NUM>. In some examples, the video frame <NUM> is provided by a single camera (e.g., the camera <NUM>). In some examples, the video frame <NUM> is monocular. Some examples include receiving the video stream 202a comprising the video frame <NUM> and the plurality of additional video frames 202b.

Operation <NUM> includes generating, from the video frame <NUM>, the plurality of images <NUM> having differing decreasing resolutions. In some examples, the plurality of images <NUM> having differing decreasing resolutions comprises a pyramid representation. In some examples, the plurality of images <NUM> having differing decreasing resolutions comprises a Gaussian pyramid representation. Operation <NUM> includes a blur operation <NUM> and a downsample operation <NUM> that are iterated to generate the pyramid representation (e.g., the plurality of images <NUM>). In some examples, generating the plurality of images <NUM> having differing decreasing resolutions comprises blurring the video frame <NUM> and downsampling the blurred video frame. In some examples, generating the plurality of images <NUM> having differing decreasing resolutions comprises iteratively blurring the video frame <NUM> and downsampling to produce a set of multiple reduced-resolution images 408a-408c, each iteration resulting in a lower resolution. In some examples, the blurring uses a Gaussian profile.

Operation <NUM> includes detecting, within each of the plurality of images <NUM>, the sets of aircraft keypoints 314a-314c for the aircraft <NUM> to be refueled. Operation <NUM> includes merging the sets of aircraft keypoints 314a-314c into the set of merged aircraft keypoints <NUM>. Operation <NUM> includes determining whether a merged aircraft keypoint is missing and, based on at least determining that the merged aircraft keypoint is missing, inserting (imputing) an additional aircraft keypoint (e.g., the additional aircraft keypoint <NUM>) into the aircraft keypoint heatmap <NUM>. Operation <NUM> further includes determining whether a merged aircraft keypoint requires correction and, based on at least determining that the merged aircraft keypoint <NUM> requires correction, correcting the merged aircraft keypoint <NUM>. Operation <NUM> includes performing temporal filtering of the aircraft keypoints <NUM>. In some examples, performing temporal filtering comprises performing Kalman filtering.

Operation <NUM> includes, based on at least the merged aircraft keypoints <NUM>, determining a position of the fuel receptacle <NUM> on the aircraft <NUM>. In some examples, determining the position of the fuel receptacle <NUM> on the aircraft <NUM> comprises determining the position of the fuel receptacle <NUM> with 6DoF. In some examples, determining the position of the fuel receptacle <NUM> on the aircraft <NUM> comprises determining the position of the aircraft <NUM>. In some examples, determining the position of the fuel receptacle <NUM> on the aircraft <NUM> comprises determining the position of the fuel receptacle <NUM> using a PnP algorithm. In some examples, determining the position of the fuel receptacle <NUM> on the aircraft <NUM> comprises determining the position of the fuel receptacle <NUM> using a NN. Some examples include operation <NUM>, which involves fusing the proximity sensor measurements <NUM> of the aircraft <NUM> with the merged aircraft keypoints <NUM>. In some examples, the proximity sensor measurements comprise lidar measurements or radar measurements. In some examples, determining the position of the fuel receptacle <NUM> on the aircraft <NUM> comprises fusing the proximity sensor measurements <NUM> of the aircraft <NUM> with the merged aircraft keypoints <NUM>.

Operation <NUM> includes determining a position of the boom tip <NUM> (e.g., the boom tip position <NUM>) of the aerial refueling boom <NUM>. In some examples, determining the boom tip position <NUM> (e.g., the position of the boom tip <NUM>) of the aerial refueling boom <NUM> comprises detecting, within the video frame <NUM>, the boom tip keypoint <NUM>.

A decision operation <NUM> identifies an unsafe condition. Decision operation <NUM> includes, based on at least the position of the fuel receptacle <NUM> and the boom tip position <NUM>, determining whether controlling the aerial refueling boom <NUM> to engage the fuel receptacle <NUM> is within the safety parameters <NUM>. If an unsafe condition exists, operation <NUM> includes, based on at least determining that controlling the aerial refueling boom <NUM> to engage the fuel receptacle <NUM> is not within the safety parameters <NUM>, generating the alert <NUM>. The aircraft <NUM> (e.g., a pilot of the aircraft <NUM>, or the aircraft's autonomous flight control, if the aircraft <NUM> is a UAV) is provided with the maneuvering information <NUM>, in operation <NUM>, for example to avoid a damaging collision of the aircraft <NUM> with the aerial refueling boom <NUM>. Operation <NUM> includes providing, to the aircraft <NUM>, maneuvering information to facilitate engaging the fuel receptacle <NUM> with the aerial refueling boom <NUM>.

Operation <NUM> may also occur even when there is no unsafe condition, in some examples. Operation <NUM> includes, based on at least the position of the fuel receptacle <NUM> and the position of the boom tip, controlling the aerial refueling boom <NUM> to engage the fuel receptacle <NUM>. In some examples, controlling the aerial refueling boom <NUM> to engage the fuel receptacle <NUM> comprises tracking a distance between the boom tip <NUM> and the fuel receptacle <NUM>. In some examples, controlling the aerial refueling boom <NUM> to engage the fuel receptacle <NUM> comprises determining the boom control parameters <NUM> to close the distance between the boom tip <NUM> and the fuel receptacle <NUM>. In situations in which there is unsafe condition, operation <NUM> may instead include controlling the aerial refueling boom <NUM> to avoid damaging the aircraft <NUM>. The flowchart <NUM> returns to operation <NUM> for the next video frame <NUM> of the video stream 202a.

<FIG> shows a flowchart <NUM> illustrating a method of air-to-air refueling according to the disclosure. In some examples, operations illustrated in <FIG> are performed, at least in part, by executing instructions by the one or more processors <NUM> of the computing device <NUM> of <FIG>. Operation <NUM> includes receiving a video frame. Operation <NUM> includes generating, from the video frame, a plurality of images having differing decreasing resolutions. Operation <NUM> includes detecting, within each of the plurality of images, a set of aircraft keypoints for an aircraft to be refueled. Operation <NUM> includes merging the sets of aircraft keypoints into a set of merged aircraft keypoints. Operation <NUM> includes based on at least the merged aircraft keypoints, determining a position of a fuel receptacle on the aircraft. Operation <NUM> includes determining a position of a boom tip of an aerial refueling boom.

With reference now to <FIG>, a block diagram of the computing device <NUM> suitable for implementing various aspects of the disclosure is described. In some examples, the computing device <NUM> includes one or more processors <NUM>, one or more presentation components <NUM> and the memory <NUM>. In some embodiments, one or more of the components may be combined or separated. For example, in some embodiments, the memory <NUM> may be integrated or part of the processor <NUM>. The disclosed examples associated with the computing device <NUM> are practiced by a variety of computing devices, including personal computers, laptops, smart phones, mobile tablets, hand-held devices, consumer electronics, specialty computing devices, etc. Distinction is not made between such categories as "workstation," "server," "laptop," "hand-held device," etc., as all are contemplated within the scope of <FIG> and the references herein to a "computing device. " The disclosed examples are also practiced in distributed computing environments, where tasks are performed by remote-processing devices that are linked through a communications network. Further, while the computing device <NUM> is depicted as a seemingly single device, in one example, multiple computing devices work together and share the depicted device resources. For instance, in one example, the memory <NUM> is distributed across multiple devices, the processor(s) <NUM> provided are housed on different devices, and so on.

In one example, the memory <NUM> includes any of the computer-readable media discussed herein. In one example, the memory <NUM> is used to store and access instructions 902a configured to carry out the various operations disclosed herein. In some examples, the memory <NUM> includes computer storage media in the form of volatile and/or nonvolatile memory, removable or non-removable memory, data disks in virtual environments, or a combination thereof. In one example, the processor(s) <NUM> includes any quantity of processing units that read data from various entities, such as the memory <NUM> or input/output (I/O) components <NUM>. Specifically, the processor(s) <NUM> are programmed to execute computer-executable instructions for implementing aspects of the disclosure. In one example, the instructions are performed by the processor, by multiple processors within the computing device <NUM>, or by a processor external to the computing device <NUM>. In some examples, the processor(s) <NUM> are programmed to execute instructions such as those illustrated in the flowcharts discussed below and depicted in the accompanying drawings.

The presentation component(s) <NUM> present data indications to an operator or to another device. In one example, presentation components <NUM> include a display device, speaker, printing component, vibrating component, etc. One skilled in the art will understand and appreciate that computer data is presented in a number of ways, such as visually in a graphical user interface (GUI), audibly through speakers, wirelessly between the computing device <NUM>, across a wired connection, or in other ways. In one example, presentation component(s) <NUM> are not used when processes and operations are sufficiently automated that a need for human interaction is lessened or not needed. I/O ports <NUM> allow the computing device <NUM> to be logically coupled to other devices including the I/O components <NUM>, some of which is built in. Implementations of the I/O components <NUM> include, for example but without limitation, a microphone, keyboard, mouse, joystick, game pad, satellite dish, scanner, printer, wireless device, etc..

The computing device <NUM> includes a bus <NUM> that directly or indirectly couples the following devices: the memory <NUM>, the one or more processors <NUM>, the one or more presentation components <NUM>, the input/output (I/O) ports <NUM>, the I/O components <NUM>, a power supply <NUM>, and a network component <NUM>. The computing device <NUM> should not be interpreted as having any dependency or requirement related to any single component or combination of components illustrated therein. The bus <NUM> represents one or more busses (such as an address bus, data bus, or a combination thereof). Although the various blocks of <FIG> are shown with lines for the sake of clarity, some implementations blur functionality over various different components described herein.

In some examples, the computing device <NUM> is communicatively coupled to a network <NUM> using the network component <NUM>. In some examples, the network component <NUM> includes a network interface card and/or computer-executable instructions (e.g., a driver) for operating the network interface card. In one example, communication between the computing device <NUM> and other devices occur using any protocol or mechanism over a wired or wireless connection <NUM>. In some examples, the network component <NUM> is operable to communicate data over public, private, or hybrid (public and private) using a transfer protocol, between devices wirelessly using short range communication technologies (e.g., near-field communication (NFC), Bluetooth® branded communications, or the like), or a combination thereof.

Although described in connection with the computing device <NUM>, examples of the disclosure are capable of implementation with numerous other general-purpose or special-purpose computing system environments, configurations, or devices. Implementations of well-known computing systems, environments, and/or configurations that are suitable for use with aspects of the disclosure include, but are not limited to, smart phones, mobile tablets, mobile computing devices, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, mobile computing and/or communication devices in wearable or accessory form factors (e.g., watches, glasses, headsets, or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, VR devices, holographic device, and the like. Such systems or devices accept input from the user in any way, including from input devices such as a keyboard or pointing device, via gesture input, proximity input (such as by hovering), and/or via voice input.

Implementations of the disclosure are described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices in software, firmware, hardware, or a combination thereof. In one example, the computer-executable instructions are organized into one or more computer-executable components or modules. In one example, aspects of the disclosure are implemented with any number and organization of such components or modules. Other examples of the disclosure include different computer-executable instructions or components having more or less functionality than illustrated and described herein. In implementations involving a general-purpose computer, aspects of the disclosure transform the general-purpose computer into a special-purpose computing device when configured to execute the instructions described herein.

By way of example and not limitation, computer readable media comprise computer storage media and communication media. Computer storage media include volatile and nonvolatile, removable, and non-removable memory implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or the like. Computer storage media are tangible and mutually exclusive to communication media. Computer storage media are implemented in hardware and exclude carrier waves and propagated signals. Computer storage media for purposes of this disclosure are not signals per se. In one example, computer storage media include hard disks, flash drives, solid-state memory, phase change random-access memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other nontransmission medium used to store information for access by a computing device. In contrast, communication media typically embody computer readable instructions, data structures, program modules, or the like in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media.

Some examples of the disclosure are used in manufacturing and service applications as shown and described in relation to <FIG>. Examples of the disclosure are described in the context of an apparatus of manufacturing and service method <NUM> shown in <FIG> and apparatus <NUM> shown in <FIG>. In <FIG>, a diagram illustrating an apparatus manufacturing and service method <NUM> is depicted in accordance with an example. In one example, during pre-production, the apparatus manufacturing and service method <NUM> includes specification and design <NUM> of the apparatus <NUM> in <FIG> and material procurement <NUM>. During production, component, and subassembly manufacturing <NUM> and system integration <NUM> of the apparatus <NUM> in <FIG> takes place. Thereafter, the apparatus <NUM> in <FIG> goes through certification and delivery <NUM> in order to be placed in service <NUM>. While in service by a customer, the apparatus <NUM> in <FIG> is scheduled for routine maintenance and service <NUM>, which, in one example, includes modification, reconfiguration, refurbishment, and other maintenance or service subject to configuration management, described herein.

In one example, each of the processes of the apparatus manufacturing and service method <NUM> are performed or carried out by a system integrator, a third party, and/or an operator. In these examples, the operator is a customer. For the purposes of this description, a system integrator includes any number of apparatus manufacturers and major-system subcontractors; a third party includes any number of venders, subcontractors, and suppliers; and in one example, an operator is an owner of an apparatus or fleet of the apparatus, an administrator responsible for the apparatus or fleet of the apparatus, a user operating the apparatus, a leasing company, a military entity, a service organization, or the like.

With reference now to <FIG>, the apparatus <NUM> is provided. As shown in <FIG>, an example of the apparatus <NUM> is a flying apparatus <NUM>, such as an aerospace vehicle, aircraft, air cargo, flying car, satellite, planetary probe, deep space probe, solar probe, and the like. As also shown in <FIG>, a further example of the apparatus <NUM> is a ground transportation apparatus <NUM>, such as an automobile, a truck, heavy equipment, construction equipment, a boat, a ship, a submarine, and the like. A further example of the apparatus <NUM> shown in <FIG> is a modular apparatus <NUM> that comprises at least one or more of the following modules: an air module, a payload module, and a ground module. The air module provides air lift or flying capability. The payload module provides capability of transporting objects such as cargo or live objects (people, animals, etc.). The ground module provides the capability of ground mobility. The disclosed solution herein is applied to each of the modules separately or in groups such as air and payload modules, or payload and ground, etc. or all modules.

With reference now to <FIG>, a more specific diagram of the flying apparatus <NUM> is depicted in which an implementation of the disclosure is advantageously employed. In this example, the flying apparatus <NUM> is an aircraft produced by the apparatus manufacturing and service method <NUM> in <FIG> and includes an airframe <NUM> with a plurality of systems <NUM> and an interior <NUM>. Examples of the plurality of systems <NUM> include one or more of a propulsion system <NUM>, an electrical system <NUM>, a hydraulic system <NUM>, and an environmental system <NUM>. However, other systems are also candidates for inclusion. Although an aerospace example is shown, different advantageous examples are applied to other industries, such as the automotive industry, etc..

The examples disclosed herein are described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program components, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program components including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks, or implement particular abstract data types. The disclosed examples are practiced in a variety of system configurations, including personal computers, laptops, smart phones, mobile tablets, hand-held devices, consumer electronics, specialty computing devices, etc. The disclosed examples are also practiced in distributed computing environments, where tasks are performed by remote-processing devices that are linked through a communications network.

An example method of air-to-air refueling comprises: receiving a video frame; generating, from the video frame, a plurality of images having differing decreasing resolutions; detecting, within each of the plurality of images, a set of aircraft keypoints for an aircraft to be refueled; merging the sets of aircraft keypoints into a set of merged aircraft keypoints; based on at least the merged aircraft keypoints, determining a position of a fuel receptacle on the aircraft; and determining a position of a boom tip of an aerial refueling boom.

An example system for air-to-air refueling comprises: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving a video frame; generating, from the video frame, a plurality of images having differing decreasing resolutions; detecting, within each of the plurality of images, a set of aircraft keypoints for an aircraft to be refueled; merging the sets of aircraft keypoints into a set of merged aircraft keypoints; based on at least the merged aircraft keypoints, determining a position of a fuel receptacle on the aircraft; and determining a position of a boom tip of an aerial refueling boom.

An example computer program product comprises a computer usable medium having a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method of air-to-air refueling, the method comprises: receiving a video frame; generating, from the video frame, a plurality of images having differing decreasing resolutions; detecting, within each of the plurality of images, a set of aircraft keypoints for an aircraft to be refueled; merging the sets of aircraft keypoints into a set of merged aircraft keypoints; based on at least the merged aircraft keypoints, determining a position of a fuel receptacle on the aircraft; and determining a position of a boom tip of an aerial refueling boom.

When introducing elements of aspects of the disclosure or the implementations thereof, the articles "a," "an," "the," and "said" are intended to mean that there are one or more of the elements. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there could be additional elements other than the listed elements. The term "implementation" is intended to mean "an example of.

Although the invention is only defined by the claims, the below embodiments, examples, and aspects are present for aiding in understanding the background and advantages of the invention.

Claim 1:
A method of air-to-air refueling, the method comprising:
receiving a video frame;
generating, from the video frame, a plurality of images having differing decreasing resolutions comprising blurring and downsampling the video frame;
detecting, within each of the plurality of images, a set of aircraft keypoints for an aircraft to be refueled;
merging the sets of aircraft keypoints into a set of merged aircraft keypoints;
based on at least the merged aircraft keypoints, determining a position of a fuel receptacle on the aircraft; and
determining a position of a boom tip of an aerial refueling boom.