Patent ID: 12234028

The present disclosure is susceptible to modifications and alternative forms, with representative embodiments shown by way of example in the drawings and described in detail below. Inventive aspects of this disclosure are not limited to the disclosed embodiments. Rather, the present disclosure is intended to cover alternatives falling within the scope of the disclosure as defined by the appended claims.

DETAILED DESCRIPTION

Embodiments of the present disclosure are described herein. It is to be understood, however, that the disclosed embodiments are illustrative examples, and that other embodiments can take various and alternative forms. The Figures are not necessarily to scale, and may be schematic. Some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the present disclosure.

Referring to the drawings, wherein like reference numbers refer to the same or like components in the several Figures, a fuel-supplying aircraft (“tanker”)10and a fuel-receiving aircraft (“receiver”)14are depicted inFIG.1while engaged in a representative aerial refueling operation. Boom operators12(seeFIG.2), i.e., human personnel trained to perform the disclosed aerial refueling tasks, are situated within the tanker10, such as in proximity to a cockpit20of the tanker10. The boom operators12control operation of a refueling boom16during the aerial refueling operation. As appreciated in the art, aerial or “air-to-air” refueling typically relies heavily on human judgment. The present disclosure seeks to improve upon the current state of the art of aerial refueling in at least this respect.

Referring briefly toFIG.2, attendant benefits of the present disclosure are accomplished using an Automated Air-to-Air (“A3R”) system11. The A3R system11as described below employs automated computer vision/machine learning-based solutions to identify, track, and account for predicted motion of the receiver14ofFIG.1during aerial refueling. The A3R system11as contemplated herein is in wired or wireless/remote communication with one or more cameras25, which provide a clear view of the boom16and the receiver14. Each camera25may be embodied as rearward-facing monocular cameras of an application-specific spectral capability. For example, the camera25may be capable of collecting real-time image data in the human-visible/red-green-blue spectral range, or using near-infrared, infrared, or other portions of the electromagnetic spectrum, e.g., under heavy cloud cover or low-light conditions.

The boom operators12ofFIG.2impart flight control inputs (arrow CC50) to an electronic control unit (“ECU”)50, e.g., using a human-machine interface (“HMI”)500in wired or wireless communication with the ECU50. In accordance with an aspect of the disclosure, the boom operators12are assisted in the accurately visualizing the receiver14and the boom16using image data in the form of a video stream (arrow250) from the camera(s)25. Constituent two-dimensional (“2D”) image frames of the video stream (arrow250) contain therein images of at least the receiver14and the boom16.

The A3R system11ultimately outputs a real-time directional indicator (arrow CCD), e.g., a suitable graphical overlay, annotation, and/or text message, to the HMI500to assist the boom operators12in tracking the current three-dimensional (“3D”) position of the receiver14during aerial refueling. The boom operators12, via the intervening ECU50, can transmit flight control signals (arrow CC17) by-wire to flight control surfaces17of the boom16shown inFIG.1. Machine learning-based real-time tracking of the receiver14in free space is performed herein by monitoring and tracking its various flight control surfaces, as described below in further detail with particular reference toFIGS.4and5.

Referring once again toFIG.1, the representative tanker10includes a fuselage18connected to wings13. In the illustrated configuration, the fuselage18may define a cargo bay with one or more fuel tanks (not shown) holding aviation fuel for eventual delivery to the receiver14. Each of the wings13may be connected in some configurations to a refueling pod21and one or more engines22, e.g., jet turbines, with the engines22providing sufficient thrust for propelling the tanker10. The fuselage18also defines the cockpit20proximate a nose23of the tanker10. An empennage assembly19is connected to the fuselage18diametrically opposite the nose23, i.e., at a tail end123of the tanker10, with the empennage assembly19in the representative construction ofFIG.1including a vertical stabilizer19V and horizontal stabilizers19H.

The tanker10is equipped to perform aerial refueling operations of the types contemplated herein, e.g., as a structurally-modified commercial passenger or transport aircraft having a reinforced airframe suitable for securely transporting the above-noted aviation fuel and associated fuel tanks, and equipped with mission-suitable avionics and control systems. Such modifications collectively enable the tanker10to transport aviation fuel to a predetermined rendezvous site with the receiver14. Upon reaching the rendezvous site, the tanker10flies in close formation with the receiver14, the particular configuration of which may differ from that which is depicted inFIG.1. For example, the tanker10may be used to refuel any suitably-equipped receiver14, such as but not limited to cargo planes, other tankers, surveillance and/or reconnaissance aircraft, air traffic control aircraft, weather monitoring aircraft, etc. The depicted construction of the receiver14ofFIG.1is therefore illustrative of just one possible embodiment thereof.

During the aerial refueling operation represented inFIG.1, the tanker10deploys the refueling boom16based on the real-time monitoring and control provided by the boom operators12and the A3R system11depicted inFIG.2. The tanker10in some configurations could also deploy flexible drogues (not shown), with each drogue fluidly coupled to the refueling pods21or to the fuselage18. The boom16moves within the slipstream of the tanker10with a level of control afforded by the flight control surfaces17. Control inputs to the flight control surfaces17of the boom16are commanded by the ECU50aboard the tanker10, which in turn may be interacted with by the boom operators12via the HMI500. The boom operators12ofFIG.2ultimately guide a nozzle end160of the boom16into a mating receptacle27located on the receiver14, with the location of the receptacle27possibly varying with the construction of the receiver14.

Referring once again toFIG.2, the boom operators12monitor and control operation of the refueling boom16ofFIG.1with the assistance of the HMI500. The HMI500may include, by way of example and not of limitation, a high-resolution display screen or screens501, e.g., touch-sensitive screens, as well as keyboards, joysticks, dials, etc. Ultimately, the electronic control signals (arrow CC50) from the HMI500cause the ECU50, using a processor52and memory54, to electronically control a corresponding attitude of the flight control surfaces17situated on the boom16ofFIG.1. In other words, the boom16is controllable in a fly-by-wire manner such that a kinematic chain does not exist between the boom operators12and the boom16. Other control implementations may be envisioned within the scope of the disclosure, including semi-autonomous or fully-autonomous control implementations, and therefore the present solutions are not limited to crewed aerial refueling operations as described herein.

The ECU50ofFIGS.1and2operates as a process controller, and may be optionally embodied as one or more computer systems configured to execute computer-readable instructions embodying a method50M, a non-limiting exemplary embodiment of which is described below with reference toFIG.4. As contemplated herein, the processor(s)52may be implemented as a microcontroller, one or more Application Specific Integrated Circuit(s) (ASICs), Field-Programmable Gate Array (FPGAs), electronic circuits, central processing units (CPUs), etc. The memory54in turn includes associated transitory and non-transitory memory/storage component(s), e.g., read only memory, programmable read only memory, solid-state memory, random access memory, optical and/or magnetic memory, etc. Computer-readable instructions embodying the method50M ofFIG.4may be recorded in memory54and executed by the processor(s)52, e.g., as machine-readable code/instructions, software, and/or firmware programs.

Other hardware components of the schematically-depicted ECU50are omitted for simplicity but are well understood in the art, such as combinational logic circuits, input/output (I/O) circuits, digital and analog signal conditioning/buffer circuitry, and other hardware components that may be accessed as needed by the processor(s)52to provide the control functionality described herein. Execution of the method50M ofFIG.4as set forth herein also requires, in one or more embodiments, a keypoint machine learning (ML) model55and one or more three-dimensional (“3D”) aerodynamic models56of the receiver14ofFIG.1, the alternatively constructed receivers140or240ofFIGS.3and5, or a fuel-receiving aircraft of a different construction.

Referring toFIG.3, the receiver140is shown flying in close formation behind the tanker10. In this non-limiting example scenario, the receiver140is a cargo/transport aircraft having a fuselage180, one or more wings130, engines220, and an empennage assembly190. The refueling boom16is shown in the process of locating the receptacle27, which in this instance is disposed on an upper surface of the fuselage180proximate a cockpit200of the receiver140.

As with the exemplary receiver14ofFIG.1, as the receiver140ofFIG.3approaches the tanker10the flight crew of the receiver140minimizes relative motion between the receiver140and the tanker10. This occurs while the boom operators12ofFIG.2maneuver the refueling boom16into proper position and alignment with the receptacle27. During this carefully coordinated flying maneuver, the boom operators12remain informed of the current 3D position of the receiver140in free space by a video depiction of the receiver140and boom16. Such information is presented in real-time via the HMI500ofFIG.2, possibly with suitable graphical overlays as determined via the method50M.

With respect to object motion tracking in general, a system based entirely on tracking the current position and velocity of a tracked object, due to system complexities and processing latency, will tend to lag the object's true position and velocity. When tracking the receiver14,140, or240of respectiveFIGS.1,3, and5, for instance, in a worst-case scenario the receiver14,140, or240could accelerate quickly and perhaps unexpectedly. In such a case, a traditional position tracking system will not respond in time, whether such a response includes initiating an evasive action of the boom16or executing a “breakaway” maneuver in which the receiver14,140, or240separates from a nozzle end160of the boom16and flies away from the tanker10. The present approach as exemplified inFIG.4thus adds an additional layer of control safety and situational awareness.

Referring toFIG.4in conjunction with the representative receiver240shown inFIG.5, the method50M is illustrated as a series of logic blocks for illustrative simplicity. As used herein, the term “block” refers to algorithms, code segments, sequences, or other constituent portions of the method50M. The individual logic blocks are implemented by the ECU50ofFIGS.1and2when performing the method50M ofFIG.4in accordance with the present disclosure.

Commencing with block B52(“Input Images”), the method50M includes receiving digital video images from the camera(s)25ofFIGS.1and2via the ECU50. The input images as contemplated herein include the constituent digital image frames of the captured video stream (arrow250) shown inFIG.2. The camera(s)25may include one or more rearward-facing monocular cameras as noted above. For instance, the camera(s)25may be part of an aircraft-mounted vision system providing the boom operators12ofFIG.2with a high-definition real-time view of the refueling boom16and the receiver240, and allowing the boom operators12to control motion of the boom16using a live/real-time image feed. The method50M proceeds to block B54once the ECU50has begun receiving the video stream (arrow250) ofFIG.2and the individual 2D image frames thereof.

Block B54(“Keypoint ML Model”) entails processing the input images from block B52through the keypoint machine learning (“ML”) model55shown schematically inFIG.2. The keypoint ML model55may include one or more 3D models of the receiver240, or more precisely of the various flight control surfaces thereof as represented by arrow FC inFIG.5.

Referring briefly toFIG.5, the depicted receiver240has a plurality of flight control surfaces (arrow FCS), the identities and locations of which will vary with the particular flight configuration of the receiver240. In the illustrated embodiment, for instance, the receiver240includes a cockpit200A located forward of one or more wings310, with a fuselage280integrally formed with the cockpit200A and the wings310. For example, vertical stabilizers290A and290B may include a corresponding rudder30A and30B as shown. Horizontal stabilizers34A and34B may be situated adjacent the respective vertical stabilizers290A and290B, and equipped with elevators37A and37B, or the horizontal stabilizers37A and37B may function as an integrally formed elevator assembly. The wings310in turn may be equipped within elevons in the form of, e.g., inboard ailerons32A,32B and outboard flaps33A,33B. The receiver240or alternative constructions thereof, such as but not limited to the receiver14and140ofFIGS.1and3, respectively, may be equipped with these or other possible flight control surfaces (arrow FCS) within the scope of the disclosure. The method50M ofFIG.4proceeds to block B56once the live video stream from block B52has been processed through the keypoint ML model55as shown inFIG.2.

Block B56(“Predicted Keypoints”) includes recording predicted keypoints on the receiver240within memory54of the ECU50. As part of block B56, the ECU50identifies the flight control surfaces (arrow FCS ofFIG.5) in the various image frames of the video stream (arrow250) received in block B52. For example, the ECU50may detect, identify, and localize specific features of interest in the various images using one or more computer vision algorithms that may be indicative of one or more of the flight control surfaces (arrow FCS ofFIG.5) or other variations of the receiver240.

Identifying relevant keypoints as part of block B56may include, e.g., using background subtraction (“BGS”) to detect the keypoints as objects of interest in real-time, or scale-invariant feature transform (“SIFT”), speeded-up robust features (“SURF”), Faster R-convolutional neural networks, deep neural networks, etc. Once the keypoints have been identified in the 2D image frames, the ECU50may process the keypoints using, e.g., object recognition, image matching, and/or motion analysis software to help enable the A3R system11ofFIG.2to track the current positions of the flight control surfaces (arrow FCS ofFIG.5). The method50M then proceeds to block B58.

At block B58(“Aerodynamic Models”), the ECU50ofFIG.2processes the predicted keypoints from block B56through the 3D aerodynamic model(s)56of the receiver240to ascertain a proper 2D-to-3D correspondence of the 2D images of the receiver240taken by the camera(s)25ofFIGS.1and2, and the 3D representation of the receiver240as represented via the 3D aerodynamic model(s)56.

In one or more embodiments, implementation of block B58may include comparing the positions of the tracked flight control surfaces (arrow FCS ofFIG.5) to data contained in one or more aerodynamic tables stored in memory54of the ECU50. Such aerodynamic tables may be based on the type, airspeed, altitude, and possibly other parameters of the particular receiver240. The 3D aerodynamic model(s)56may then output estimated aerodynamic forces on the receiver240based on the predicted or detected positions of the flight control surfaces (arrow FCS ofFIG.5). This information may be used by the ECU50ofFIGS.1and2to estimate the current state and an imminent/future state of the receiver240, including its 3D position in free space. This in turn informs the boom operators12and the ECU50ofFIG.2as to how and to what extent the refueling boom16ofFIGS.1and3will likely move in a next instant of time, and whether the boom16will likely be required to perform an evasive maneuver, such as executing a breakaway maneuver. The method50M ofFIG.4then proceeds to block B60.

Block B60(“Output Predicted Receiver Movement”) includes outputting, via the ECU50ofFIGS.1and2, a predicted motion trajectory of the receiver240. Movement of the receiver240ofFIG.5may be indicated via the HMI500ofFIG.2in various ways, including but not limited to as a graphical overlay, an audible and/or visible alert, a tactile or haptic alert, and/or a text message indicative of the predicted motion of the receiver240. Such look-ahead information could be used to inform the boom operators12and/or the ECU50of impending motion of the receiver240. That is, with the data collected at a given point in time (t), motion of the flight control surfaces (arrow FCS ofFIG.5) may provide information as to motion at time (t+1) of the receiver240. This would allow the boom operators12and/or the ECU50shown inFIG.2to respond more quickly to the impending motion of the receiver240than would otherwise be possible using a typical position and velocity tracking system of the type noted above.

For example, the receiver240shown inFIG.5may operate in stable flight at a position trailing the tanker10, with the positions of the various flight control surfaces (arrow FCS) corresponding to steady-state operating conditions. Relative motion between the receiver240and the refueling boom16would be minimal in this case. During steady-state operation, a pilot of the receiver240could command a sudden change in elevation, pitch, yaw, and/or roll in response to any number of dynamic conditions, including air turbulence, a possible impending bird strike, glare or obstructed vision, mechanical or electrical faults, etc. Ordinarily, the boom operators12would rely on force feedback or other sensed values on the refueling boom16to alert the boom operators12as to the possible need for a breakaway maneuver. However, using the A3R system11ofFIG.2, the ECU50is able to more quickly discern that the receiver240is about to perform a flight movement that could require the refueling boom16to break away. The ECU50could then inform the boom operators12earlier in the process, and possibly before detectable forces are present on the boom16.

Tracking keypoints on the various control surfaces helps predict motion of the receiver240based on the deflection of its control surfaces. Doing so will allow the ECU50or the boom operators12to call a breakaway condition and/or more accurately track the receiver's position, and the ECU50give a heads-up display to the boom pilot. Earlier warning capabilities of the A3R system11in turn would permit the boom operators12and/or the ECU50ofFIG.2to take preemptive action to protect the refueling boom16, the receiver240, and the tanker10, along with their respective flight crews. In this manner, the present teachings may help render A3R processes more responsive to real-time motion of the receiver240. These and other benefits will be appreciated by those skilled in the art in view of the foregoing disclosure.

The terms “comprising”, “including”, and “having” are inclusive and therefore specify the presence of stated features, steps, operations, elements, or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, or components. Orders of steps, processes, and operations may be altered when possible, and additional or alternative steps may be employed. As used in this specification, the term “or” includes any combinations of the associated listed items. The term “any of” is understood to include any possible combination of referenced items, including “any one of” the referenced items. The term “any of” is understood to include any possible combination of referenced claims of the appended claims, including “any one of” the referenced claims.

For consistency and convenience, directional adjectives may be employed throughout this detailed description corresponding to the illustrated embodiments. Those having ordinary skill in the art will recognize that terms such as “above”, “below”, “upward”, “downward”, “top”, “bottom”, etc., may be used descriptively relative to the figures, without representing limitations on the scope of the invention, as defined by the claims.

The following Clauses provide example configurations of a system and method for providing high-power optical amplification in accordance with the disclosure, as shown in the exemplary scenario ofFIGS.1-3and disclosed herein.

Clause 1: An automated air-to-air refueling (“A3R”) system for use with a tanker having a refueling boom, the A3R system comprising: a camera connected to the tanker in proximity to the refueling boom and configured to output a video stream of the refueling boom and a receiving aircraft (“receiver”) during an aerial refueling process; a human-machine interface (HMI) located aboard the tanker; and an electronic control unit (“ECU”) in communication with the camera and the HMI, wherein the ECU is configured to identify keypoints on the receiver indicative of flight control surfaces of the receiver, track corresponding positions of the flight control surfaces in real-time, predict a change in a three-dimensional (“3D”) position of the receiver, as a predicted 3D position, using the corresponding positions of the one or more flight control surfaces, and output a directional indicator via the HMI that is indicative of the predicted 3D position.

Clause 2: The A3R system of clause 1, wherein the camera includes one or more rearward-facing monocular cameras connected adjacent to an end of the refueling boom.

Clause 3: The A3R system of either of clauses 1 or 2, wherein the HMI includes a display screen, and wherein the directional indicator includes a graphical overlay on the display screen.

Clause 4: The A3R system of any of clauses 1-3, wherein the ECU is programmed with a keypoint machine learning (“ML”) model, and to identify the keypoints on the receiver using the keypoint ML model.

Clause 5: The A3R system of clause 4, wherein the keypoint ML model includes at least one of background subtraction (“SBS”), scale-invariant feature transform (“SIFT”), speeded-up robust features (“SURF”), faster R-convolutional neural networks, or deep neural networks.

Clause 6: The A3R system of any of clauses 1-5, wherein the ECU is programmed with one or more three-dimensional (“3D”) aerodynamic models of the receiver, and is configured to predict the change in a 3D position of the receiver using the one or more 3D aerodynamic models of the receiver.

Clause 7: The A3R system of clause 6, wherein the one or more 3D aerodynamic models of the receiver include one or more aerodynamic tables for a type and airspeed of the receiver, and wherein the one or more 3D aerodynamic models of the receiver are configured to provide an estimated aerodynamic force on the receiver based on the predicted positions of the flight control surfaces.

Clause 8: The A3R system of any of clauses 1-7, wherein the refueling boom is a fly-by-wire device, and wherein the ECU is configured to control a flight maneuver of the refueling boom in response to operator inputs to the HMI.

Clause 9: A method for refueling a fuel-receiving aircraft (“receiver”) during an automated air-to-air refueling (“A3R”) process, comprising: receiving from a camera during the A3R process, via an electronic control unit (“ECU”) aboard a tanker having a refueling boom, a real-time video stream of the receiver and the refueling boom, wherein the camera is connected to the tanker in proximity to the refueling boom; identifying keypoints on the receiver indicative of flight control surfaces thereof; tracking corresponding positions of the flight control surfaces in real-time via the ECU; predicting a change in a three-dimensional (3D) position of the receiver, as a predicted 3D position, using the corresponding positions of the one or more flight control surfaces; and outputting a directional indicator to the HMI, via the ECU, indicative of the predicted 3D position.

Clause 10: The method of clause 9, wherein receiving the real-time video stream includes operating a rearward-facing monocular camera connected adjacent to an end of the refueling boom.

Clause 11: The method of either of clauses 9 or 10, wherein the HMI includes a display screen, and wherein outputting the directional indicator to the HMI includes presenting a graphical overlay on the display screen.

Clause 12: The method of any of clauses 9-11, wherein the ECU is programmed with a keypoint machine learning (“ML”) model, and wherein identifying the keypoints on the receiver includes using the keypoint ML model.

Clause 13: The method of clause 12, wherein using the keypoint ML model includes using one or more of a background subtraction (SBS), a scale-invariant feature transform (SIFT), a speeded-up robust features (SURF), a faster R-convolutional neural networks, or a deep neural network.

Clause 14: The method of any of clauses 9-13, wherein the ECU is programmed with one or more three-dimensional (“3D”) aerodynamic models of the receiver, and wherein predicting the change in the 3D position of the receiver is performed using the one or more 3D aerodynamic models of the receiver.

Clause 15: The method of clause 14, wherein the one or more 3D aerodynamic models of the receiver include one or more aerodynamic tables for a type and airspeed of the receiver, further comprising: using the or more aerodynamic tables to provide an estimated aerodynamic force on the receiver based on the predicted positions of the flight control surfaces.

Clause 16: The method of any of clauses 9-15, wherein the refueling boom is a fly-by-wire device, further comprising: controlling a flight maneuver of the refueling boom in response to operator inputs to the HMI.

Clause 17: A tanker comprising: a fuselage having a nose and a tail end, and configured to transport a supply of aviation fuel; one or more wings connected to the fuselage; a refueling boom connected to the tail end; and an automated air-to-air refueling (“A3R”) system, comprising: a rearward-facing monocular camera connected to the tail end of the fuselage of the tanker in proximity to the refueling boom, wherein the rearward-facing monocular camera is configured to output a video stream of the refueling boom and a fuel-receiving aircraft (“receiver”) during an aerial refueling process; a human-machine interface (“HMI”) having a display screen; and an electronic control unit (“ECU”) in communication with the rearward-facing monocular camera and the HMI, wherein the ECU is configured to identify, using a keypoint machine learning model, keypoints on the receiver indicative of flight control surfaces thereof, track corresponding positions of the flight control surfaces in real-time using the keypoints, predict a change in a three-dimensional (“3D”) position of the receiver as a predicted 3D position using the corresponding positions of the one or more flight control surfaces, and output a directional indicator to the HMI indicative of the predicted 3D position, wherein the directional indicator includes a graphical overlay on the display screen.

Clause 18: The tanker of clause 17, wherein the keypoint ML model includes at least one of background subtraction (“SBS”), scale-invariant feature transform (“SIFT”), speeded-up robust features (“SURF”), faster R-convolutional neural networks, or deep neural networks.

Clause 19: The tanker of either of clauses 17 or 18, wherein the ECU is programmed with one or more 3D aerodynamic models of the receiver, and is configured to predict the change in the 3D position of the receiver using the one or more 3D aerodynamic models of the receiver.

Clause 20: The tanker of clause 17, wherein the one or more 3D aerodynamic models of the receiver include one or more aerodynamic tables for a type and airspeed of the receiver, and wherein the one or more 3D aerodynamic models of the receiver are configured to provide an estimated aerodynamic force on the receiver based on the predicted positions of the flight control surfaces.

While various embodiments have been described, the description is intended to be exemplary rather than limiting. It will be apparent to those of ordinary skill in the art that many more embodiments and implementations are possible that are within the scope of the embodiments. Any feature of any embodiment may be used in combination with or substituted for any other feature or element in any other embodiment unless specifically restricted. Accordingly, the embodiments are not to be restricted except in light of the attached claims and their equivalents. Also, various modifications and changes may be made within the scope of the attached claims.