Patent ID: 12221231

DETAILED DESCRIPTION

The illustrative examples recognize and take into account one or more different considerations. For example, the illustrative examples recognize and take into account that unmanned aircraft systems (UAS) can be used for visual inspection of aircraft. In the illustrative examples, an unmanned aircraft system can include an unpiloted aerial vehicle (UAV) or combination of unpiloted aerial vehicles (UAVs), an unpiloted aerial vehicle controller, monitors for observing navigation, or other unmanned aircraft system components.

The illustrative examples recognize and take account that these inspections, however, may not be performed in a standard or consistent manner. As a result, the illustrative examples recognize and take account that variability in quality and untrustworthy results can occur. The illustrative examples recognize and take into account that conventional techniques can lead to questions about whether the data collected meets regulatory requirements or inspection procedural standards. The illustrative examples recognize and take into account that inconsistent quality results between inspections reduces the ability to make objective comparisons between aircraft or the same aircraft on different dates for aircraft condition assessment.

The illustrative examples recognize and take into account at least one of incorrectly flown flight paths or issues with data collection can result in undetected anomalies and undesired quality in the data collection. For example, issues with data collection can include, for example, out of focus images, features not within a field of view, low lighting, or other undesired issues that can be present in capturing images during data collection.

As used herein, the phrase “at least one of,” when used with a list of items, means different combinations of one or more of the listed items can be used, and only one of each item in the list may be needed. In other words, “at least one of” means any combination of items and number of items may be used from the list, but not all of the items in the list are required. The item can be a particular object, a thing, or a category.

For example, without limitation, “at least one of item A, item B, or item C” may include item A, item A and item B, or item B. This example also may include item A, item B, and item C or item B and item C. Of course, any combinations of these items can be present. In some illustrative examples, “at least one of” can be, for example, without limitation, two of item A; one of item B; and ten of item C; four of item B and seven of item C; or other suitable combinations.

For example, the illustrative examples recognize and take into account that incorrectly traversed flight paths can occur from incorrectly planned paths, missed waypoints, out of order waypoints, flying too close or too far from target locations, unexpected obstacle avoidance maneuvers, and other sources. The illustrative examples recognize and take into account that issues with data collection can include poor lighting, images that are taken at improper locations, using an improper zoom value, incorrect aperture or shutter speed, out of focus, missing reference marks, missing targets, or other issues.

The illustrative examples recognize and take into account that conventional techniques provide images and videos to technicians and inspectors without a data validation process. The illustrative examples recognize and take into account that although the inspections meet various maintenance and planning and regulatory requirements. These guidelines do not currently define objective requirements to show that the data collection performed by an unmanned aircraft system via an automated or semi-automated process was valid as compared to a technicians guided inspection. As result, the illustrative examples recognize and take account that images and videos acquired using unmanned aircraft systems may need intensive human operator analysis to determine whether sufficient data has been collected to meet regulatory requirements or maintenance procedures.

Therefore, it would be desirable to have a method and apparatus that takes into account at least some of the issues discussed above, as well as other possible issues. For example, it would be desirable to have a method and apparatus that overcome a technical problem with performing visual inspections of aircraft in a manner that involves less time and cost while improving quality, safety, accuracy, and repeatability.

Thus, illustrative examples provide a method, apparatus, system, and computer program product for inspecting an aircraft using unmanned aircraft systems with validation of the captured data acquired by the unmanned aircraft system. In the illustrative examples, captured data can include location, offset distance, image data, or a combination thereof.

In one illustrative example, captured data is received for a flight path flown by an unmanned aircraft system to acquire captured data about the aircraft condition. The captured data is compared with reference data for the aircraft to form a comparison. A determination is made as to whether the captured data is within a set of tolerances for valid captured data using the comparison. A determination of a set of corrective actions is made in response to the captured data being outside of the set of tolerances for valid captured data in which the set of corrective actions is performed prior to detecting anomalies for the aircraft using the captured data.

As used herein, a “set of” when used with reference items means one or more items. For example, a set of tolerances is one or more tolerances. As another example, set of corrective actions is one or more corrective actions.

With reference now to the figures and in particular with reference toFIG.1, an illustration of an aircraft inspection environment is depicted in accordance with an illustrative example. In this depicted example, aircraft inspection environment100is an environment in which a commercial airplane102is an example of an aircraft that can be inspected using unmanned aircraft systems. The inspection can be performed to identify anomalies that may need maintenance. Further, this inspection can be performed to acquire data to meet regulatory requirements and enhance maintenance procedures in repair alternatives and predictive scheduling.

As depicted, the inspection is performed by first unmanned aircraft system (UAS)104, second unmanned aircraft system (UAS)106, third unmanned aircraft system (UAS)108, and fourth unmanned aircraft system (UAS)110. In this illustrative example, an unmanned aircraft system is an unpiloted aircraft that includes equipment used to control the unmanned aircraft remotely. An unmanned aircraft system can include more than one unpiloted aircraft. These unmanned aircraft systems have a capability that allows traversing the airplane inspection area for commercial airplane102.

In the illustrative example, these unmanned aircraft systems operate an unmanned fashion following preplanned flight paths to capture data about commercial airplane102and comparative navigational aid data.

In one approach to inspecting commercial airplane102, first unmanned aircraft system104flies on first flight path112to capture data about commercial airplane102. Second unmanned aircraft system106flies on second flight path114to capture data about commercial airplane102. Third unmanned aircraft system108flies on third flight path116to capture data about commercial airplane102and fourth unmanned aircraft system110flies on fourth flight path118to capture data about commercial airplane102.

The unmanned aircraft systems in this illustrative example can stop at points on these depicted flight paths to capture data, such as image, location, or offset data. The captured image data can be images such as still images or images in a video. The captured image data can be for images other than in the visible part of the electromagnetic spectrum. For example, the image data can be for infrared (IR) or ultraviolent (UV) images.

As depicted, first unmanned aircraft system104, second unmanned aircraft system106, third unmanned aircraft system108, and fourth unmanned aircraft system110are in communication with computer120using wireless communication links. As depicted, first unmanned aircraft system104can send first captured data130over first wireless communications link122; second unmanned aircraft system106can send second captured data132over second wireless communications link124; third unmanned aircraft system108can send third captured data134over third wireless communications link126; and fourth unmanned aircraft system110can send fourth captured data136over fourth wireless communications link128. This captured data can be transmitted by the unmanned aircraft systems to computer120while flying on the flight paths or after flying the flight paths.

In this illustrative example, computer120analyzes the captured image data received from the unmanned aircraft systems to determine whether captured data as a level of confidence for use in aircraft maintenance continued airworthiness assessment. Also in this illustrative example, location data is received from the unmanned aircraft systems to determine whether captured data as a level of confidence for use in proper location for image capture. This level of confidence can be determined by computer120analyzing the captured data to determine whether the flight paths were correctly flow using locations and offset distances in captured data as well as other location information such as drawings and inspection maps. The analysis can also include determining whether image data in the captured data contains features of interest and whether the captured data meets standards needed for analyzing the captured data to determine whether anomalies are present on commercial airplane102.

In this example, the validation of the captured data can include captured location data and captured offset distance data recorded from the unmanned aircraft systems flying the fight paths. Captured location data includes locations of the unmanned aircraft systems and offset distance data for distances from the unmanned aircraft systems to a surface of commercial airplane102. The captured data also includes image data recorded while the unmanned aircraft systems fly on the flight paths.

In the illustrative example, computer120can send valid captured data138to a set of operational entities that utilize the image data in aircraft maintenance if the captured data is determined to be valid. The set of entities can be selected from at least one of an airline, an airplane maintenance facility, a regulatory agency, or some other maintenance and airworthiness entity. Entities140can use the data for various purposes. For example, a regulatory agency can log the captured data within the regulators maintenance system that meets an inspection regulation. An airline or airplane maintenance facility can analyze the captured image data to determine whether anomalies are present and initiate maintenance as needed based on the results of the analysis of the captured data.

If the captured image data is determined to be invalid by the computer system, computer120can control the unmanned aircraft systems to perform addition data gathering flights with corrective actions to eliminate the cause of the invalid result. These corrective actions can include repeating image data capture for the flight path, changing settings for the sensors, and altering the flight path flown to acquire new captured image data. For example, if second captured data132is determined to be invalid, computer120can instruct second unmanned aircraft system106to repeat capturing data using second flight path114or a modified flight path. Further, computer120can change camera settings in second unmanned aircraft system106. These and other corrective actions can be taken when second captured data132is invalid.

Thus, the illustrative example inFIG.1can provide captured data with a level of quality that can be used to determine whether anomalies are present that may need maintenance. As a result, airlines maintenance facilities and other entities using captured data for analysis do not need to validate the captured images have been properly acquired prior to performing analysis. Further, this captured data can also be used to meet regulations by regulatory agencies. With the validation of captured image and location data from unmanned aircraft systems, collecting corroborating information outside this automated maintenance inspection system to make decisions may be avoided or reduced. The corroborating information is information gathered from sources other than the automated maintenance inspection system. As result, delays in inspection processes and out of service time for aircraft can be reduced as well as improved quality records of the inspection results.

Although the illustrative example inFIG.1the unmanned aircraft systems outside of commercial airplane102, unmanned aircraft systems can also be used for in-cabin or internal fuselage inspections.

With reference next toFIG.2, an illustration of an aircraft inspection environment is depicted in accordance with an illustrative example. The components in aircraft inspection environment100inFIG.1is example of components that can be implemented in aircraft inspection environment200inFIG.2.

Aircraft inspection system202in aircraft inspection environment200can operate to inspect aircraft204. Aircraft204can take a number of different forms. As used herein, a “number of” when used with reference items means one or more items. A number of different forms is one or more different forms.

For example, aircraft204can be selected from a group comprising one of a commercial aircraft, a commercial passenger airplane, a rotorcraft, a tilt-rotor aircraft, a tilt wing aircraft, a vertical takeoff and landing aircraft, an electrical vertical takeoff and landing vehicle, a partially manufactured aircraft, and other suitable types of aircraft.

In this illustrative example, aircraft inspection system202comprises a number of different components. As depicted, aircraft inspection system202comprises computer system206, controller208, and a set of unmanned aircraft systems210.

Controller208can be implemented in software, hardware, firmware or a combination thereof. When software is used, the operations performed by controller208can be implemented in program code configured to run on hardware, such as a processor unit. When firmware is used, the operations performed by controller208can be implemented in program code and data and stored in persistent memory to run on a processor unit. When hardware is employed, the hardware can include circuits that operate to perform the operations in controller208.

In the illustrative examples, the hardware can take a form selected from at least one of a circuit system, an integrated circuit, an application specific integrated circuit (ASIC), a programmable logic device, or some other suitable type of hardware configured to perform a number of operations. With a programmable logic device, the device can be configured to perform the number of operations. The device can be reconfigured at a later time or can be permanently configured to perform the number of operations. Programmable logic devices include, for example, a programmable logic array, a programmable array logic, a field programmable logic array, a field programmable gate array, and other suitable hardware devices. Additionally, the processes can be implemented in organic components integrated with inorganic components and can be comprised entirely of organic components excluding a human being. For example, the processes can be implemented as circuits in organic semiconductors.

In one illustrative example, controller208can be an automated analysis tool. For example, controller208can be an artificial intelligence system. The artificial intelligence system can be a machine learning model, a knowledge base, or some other suitable type of automated analysis program or software.

Computer system206is a physical hardware system and includes one or more data processing systems. When more than one data processing system is present in computer system206, those data processing systems are in communication with each other using a communications medium. The communications medium can be a network. The data processing systems can be selected from at least one of a computer, a server computer, a tablet computer, or some other suitable data processing system.

As depicted, computer system206includes a number of processor units212that are capable of executing program instructions214implementing processes in the illustrative examples. As used herein a processor unit in the number of processor units212is a hardware device and is comprised of hardware circuits such as those on an integrated circuit that respond and process instructions and program code that operate a computer.

When a number of processor units212execute program instructions214for a process, the number of processor units212is one or more processor units that can be on the same computer or on different computers. In other words, the process can be distributed between processor units on the same or different computers in a computer system. Further, the number of processor units212can be of the same type or different type of processor units. For example, a number of processor units can be selected from at least one of a single core processor, a dual-core processor, a multi-processor core, a general-purpose central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or some other type of processor unit.

In one illustrative example, unmanned aircraft system216in the set of unmanned aircraft systems210can operate using mission218. Mission218can be uploaded to unmanned aircraft system216from controller208.

In this illustrative example, mission218includes flight path220that defines a route to be flown by unmanned aircraft system216and image plan222that defines captured image data250is to be acquired by unmanned aircraft system216for aircraft204under inspection.

Controller208in computer system206receives captured data224for flight path220flown by an unmanned aircraft system216to inspect aircraft204. In this example, this flight path is flown to acquire captured data224for aircraft204under inspection. Captured data224can include captured location data242, captured offset distance data244, and captured image data250.

In this illustrative example, captured data224is acquired by sensor system228in unmanned aircraft system216. Sensor system228can include sensors such as a global positioning system (GPS) receiver, inertial measurement unit (IMU), a laser range meter, a camera system, and other types of sensors that can acquire captured data224.

Captured data224can be received from unmanned aircraft system216in a number of different ways. For example, captured data224can be received in real time over a wireless connection during flight of unmanned aircraft system216on flight path220. When captured data224is received in real time from unmanned aircraft system216by controller208in computer system206, captured data224is received as quickly as possible without any intentional delay.

In other illustrative examples, captured data224is received by controller208in computer system206periodically or after portions of flight path220have been flown by unmanned aircraft system216. And in yet other illustrative examples, captured data224can be received by controller208in computer system206after unmanned aircraft system216has completed flying flight path220.

In this illustrative example, controller208can receive captured data224for flight path220while unmanned aircraft system216is flying flight path220. In this example, captured data224is for portion230of flight path220flown by unmanned aircraft system216. In other words, captured data224can be sent for each portion of flight path220flown by unmanned aircraft system216. In yet another illustrative example, captured data224can be sent in real time as unmanned aircraft system216flies on flight path220. In other words, captured data224can be sent to and received by controller208without any intentional delay during the operation of unmanned aircraft system216.

In another illustrative example, controller208can receive captured data224for flight path220after unmanned aircraft system216has completed flight path220. With this example, captured data224is for all of flight path220.

In this illustrative example, controller208compares captured data224with reference data226for aircraft204to form comparison232. Controller208can determine whether captured data224is within a set of tolerances234for valid captured data236using the comparison232.

The set of tolerances234can be at least one of a range, a threshold, a desired value, or other tolerance for captured data224relative to reference data226. In other words, tolerances234defined how much variation can be in captured data224from reference data226in comparison232for captured data224to be considered valid captured data236.

In one illustrative example, in comparing captured data224with reference data226, controller208can compare captured data224with reference data226, controller208can access captured location data242and captured offset distance data244for the unmanned aircraft system216in captured data224. In making this comparison, controller208can also access reference location data246and reference offset distance data248in the reference data226. Other data can also be present in captured data224such as an angle or angles of camera aiming components (e.g. gimbal) and other input parameters. Controller208then compares captured location data242and captured offset distance data244to reference location data246and reference offset distance data248respectively, to create comparison232.

In another example, controller208can compare captured data224with reference data226by accessing captured image data250in captured data224; access reference image data252in reference data226; and compare captured image data250with reference image data252to form comparison232.

In yet another was example, comparison232can be made by controller208accessing captured location data242, captured offset distance data244, and captured image data250in captured data224. Controller208can also access reference location data246, reference offset distance data248, and reference image data252in reference data226. Controller208compares captured location data242, captured offset distance data244, and captured image data250to reference location data246, reference offset distance data248, and reference image data252, respectively, to form the comparison232.

In this illustrative example, reference data226can be generated for comparison in a number of different ways. For example, controller208can create reference location data246and reference offset distance data248using at least one of a computer aided design (CAD) model of the aircraft, point cloud scans, historical flight paths from one or more prior inspections of aircraft204, other similar aircraft, or from other sources. Controller208can create reference image data252creating, by the computer system, the reference image data using at least one of a computer aided design model of aircraft204, a point cloud scan of aircraft204, historical images of aircraft204, other similar aircraft, or other sources of information about aircraft204.

In this illustrative example, captured data224is defined in absolute coordinate system256. Absolute coordinate system256can have coordinates that are defined using a coordinate system, such as latitude, longitude, and altitude.

In this example, reference data226is defined using relative coordinate system254. In this example, reference location data246can be defined using relative coordinate system254for aircraft204. Relative coordinate system254can be defined using an origin that is relative to a point or location on or in aircraft204. For example, the origin can be the nose of aircraft204. In other words, captured location data242describes locations using relative coordinate system254.

In this illustrative example, controller208converts one of the coordinate systems into the other coordinate system such that both captured data224and reference data226are described using the same coordinate system. This process can also be referred to as aligning the coordinate systems such that the data uses the same coordinate system.

In the illustrative example, captured data224is converted from using absolute coordinate system256to relative coordinate system254to compare captured data224and reference data226to generate comparison232.

When captured data224is not valid captured data236, controller208determines a set of corrective actions238in response to the captured data224being outside of the set of tolerances234for valid captured data236. In this example, the set of corrective actions238is performed prior to detecting anomalies258for the aircraft204using captured data224.

The set of corrective actions238can take a number different forms. For example, the set of corrective actions238is selected from at least one of repeating capturing data for the flight path, repeating image generation for a portion of the flight path on which captured data is invalid, repeating capturing data using an adjusted flight path, adjusting settings of a camera system in sensor system228on the unmanned aircraft system216, or other suitable actions.

The set of corrective actions238can be performed in a number of different ways. For example, the set of corrective actions238can be using at least one of unmanned aircraft system216for another unmanned aircraft system in unmanned aircraft systems210. In other words, the same unmanned aircraft system generating invalid captured data can be used to re-capture data. In another example, a different unmanned aircraft system can be used. In yet another example, the same unmanned aircraft system and one or more other unmanned aircraft systems to be used to capture data from flight path220.

When captured location data242is valid, controller208can perform a number of different actions with captured location data242. For example, controller208can indicate that captured location data242is valid for analysis in response to determining that captured data224is valid. Captured location data242can be collected for a number of different functions including regulatory, planning, management, maintenance, predictive maintenance or other types of functions. This indication can be used to initiate action such as logging or recording captured data224as well as the results in comparison232.

As another example, when captured data224is valid, another action that can be formed is analyzing captured data224for a set of anomalies258for aircraft204. Based on the set of anomalies258identified, a set of maintenance actions260can be performed for aircraft204. If a set of anomalies258is found in the analysis, then maintenance actions260can be scheduled to resolve the set of anomalies258.

In this example, performing the set of maintenance actions260can include resolving the set of anomalies258through at least one of repair, rework, replacement, or other actions. The set of maintenance actions260can also include flying unmanned aircraft system216on a portion230of flight path220where the set of anomalies258were located after maintenance is performed on the set of anomalies258. In other words, another inspection of aircraft204can be performed using unmanned aircraft system216after resolving the set of anomalies258. This inspection can include repeating receiving captured location data242for portion230of flight path220flown by unmanned aircraft system216to acquire captured data224for aircraft204; comparing captured data224with reference data226for aircraft204to form comparison232; determining whether captured data224is within the set of tolerances234for valid captured data236using comparison232; and determining the set of corrective actions238in response to captured data224being outside of the set of tolerances234for valid captured data236prior to detecting the set of anomalies258for aircraft204using captured data224.

In one illustrative example, one or more solutions are present that overcome a problem with performing inspections using unmanned aircraft systems. As a result, one or more solutions can provide an ability to acquire data from inspecting an aircraft that is validated and can be used to perform actions such as maintenance scheduling, fleet planning, fleet management, aircraft analysis, and other actions.

Computer system206can be configured to perform at least one of the steps, operations, or actions described in the different illustrative examples using software, hardware, firmware or a combination thereof. As a result, computer system206operates as a special purpose computer system in which controller208in computer system206enables inspecting aircraft in a more effective manner as compared to current techniques for inspecting aircraft. In particular, controller208transforms computer system206into a special purpose computer system as compared to currently available general computer systems that do not have controller208.

In the illustrative example, the use of controller208in computer system206integrates processes into a practical application for inspecting aircraft in which the captured data for the flight path flown by the unmanned aircraft system can be validated. This validation includes validating the flight path three-dimensional locations, offset distance data, and image data prior to performing analysis to discover anomalies. Further, controller208can provide indications of validity in a manner that is acceptable for use by aviation regulators, airlines, maintenance facilities, and other entities. In this manner, the inspection performed using aircraft inspection system202can be used in place of or in addition to manual inspections.

Turning next toFIG.3, an illustration of a mission is depicted in accordance with an illustrative example. In the illustrative examples, the same reference numeral may be used in more than one figure. This reuse of a reference numeral in different figures represents the same element in the different figures.

The illustration of example components in mission218is depicted in this example, as illustrated, mission218comprises flight path220and image plan302.

Flight path220includes number of different components. As depicted, flight path220comprises waypoints308can be used for location and navigation to acquire captured data224. Waypoints308include locations314, velocities310, and offset distances312.

Locations314include positions304in three-dimensional space that unmanned aircraft system216will fly for flight path220. For example, a position can be the x, y, z coordinates of the unmanned aircraft system.

Locations314can also include orientations316. An orientation can be the orientation of unmanned aircraft system216. The orientation can be described using roll, pitch, and yaw. Orientations316can change over different portions of flight path220.

As depicted, velocities310in waypoints308are the speeds and directions at which unmanned aircraft system216flies on flight path220. Velocities310can vary over portions of flight path220.

Flight path220can also include aim direction306. Aim direction306defines the aiming direction of the camera and sensors in sensor system228in unmanned aircraft system216. This aiming direction may involve multiple rotation axes that are set by a rotation control mechanism like a gimbal, which can control a pan axis and a tilt axis (otherwise known as azimuth and elevation, respectively), which serve to point the camera and sensors in a specific direction.

Offset distances312in waypoints308are the distances between unmanned aircraft system216and the surface of aircraft204. These distances are distances expected or that are optimal for safely flying unmanned aircraft system216near the aircraft204and for capturing captured image data250.

In this illustrative example, image plan222is information used for acquiring captured image data250. Captured image data250that can be a sequence of still images or images in a video.

In this illustrative example, image plan222includes a number of different components. As depicted, image plan222comprises pixel density320, resolution322, field of view (FoV)324, lighting326, and wavelength328.

Pixel density320is the number of pixels within an area of an image acquired by the camera. Pixel density can be, for example, measured in pixels per inch. Resolution322is the amount of detail that can be held in an image acquired by a camera in sensor system228in unmanned aircraft system216. Field of view324is an angle (or in some cases separate horizontal and vertical angles) that represents the extents of the environment that can be viewed and captured by the camera and/or sensor system228. The field of view can be measured in degrees.

Lighting326is the amount of light desired for capturing images. Wavelength328the portion of the electromagnetic spectrum for which images are to be captured by the camera and sensor system228. Wavelength328can include at least one of visible light, infrared light, ultraviolet rays, or other wavelengths.

Turning next toFIG.4, an illustration of dataflow for validating captured data is depicted in accordance with an illustrative example. In this illustrative example, controller208performs a number of validations in determining whether captured data224is valid captured data236. As depicted, controller208performs location validation400, surface offset validation402, and image data validation404. In this illustrative example, these validations can be performed using reference data226and captured data224. The allowed deviations used in these validations can be found in the set of tolerances234inFIG.2.

In location validation400, the validation process used is independent of the source of the three-dimensional location data analyzed in this process. For example, the three-dimensional location data could be provided by a differential global positioning system (GPS) receiver, a simultaneous localization and mapping (SLAM) system, motion capture motion capture system, or other types of three-dimensional localization systems.

In this example, the three-dimensional location is described using x, y, z coordinates.

Location validation400can be performed by controller208performing checks on location406, velocity408, and orientation410in determining whether captured location data242is valid. In this example, location406can be validated for each location in captured location data242.

In this illustrative example, location406can be validated using reference location, X; reference location, Y; reference location, Z; captured location, X; captured location, Y; captured location, Z; and allowed location deviation. A difference between the reference location values and captured location values can be calculated to obtain a location deviation. This location deviation can be compared to the allowed location deviation determine whether the location is valid.

In this illustrative example, velocity408can be validated in location validation400using reference velocity, X; reference velocity, Y; reference velocity, Z; captured velocity, X; captured velocity, Y; captured velocity, Z; and allowed velocity deviation. A difference between the reference velocity values and the captured velocity values can be calculated to obtain a velocity difference. This velocity difference can be compared with the allowed velocity deviation to determine whether the velocity for this data point is valid. In this illustrative example, the velocity should be “0” at time of image capture.

A similar check can be performed on orientation410as part of location validation400. This check can be performed using reference angle, roll; reference angle, pitch; reference angle, yaw; captured angle, roll; captured angle, pitch; captured angle, yaw; and allowed angle deviation.

The orientation deviation can be calculated by taking the difference between the reference angles for orientation and the captured angles orientation. The orientation deviation can be compared to allowed angle deviation whether the orientation is valid.

In this illustrative example, surface offset validation402is a validation of the distances from the unmanned aircraft system camera lens to the surface of aircraft. This validation can be performed for each location of interest. Surface offset validation402for offset412can be performed using reference offset distance in reference offset distance data248; captured offset distance in captured offset distance data244; and allowed offset distance deviation. The difference between the reference offset distance and the captured offset distance is calculated to obtain and offset distance deviation. This offset distance deviation can be used to determine whether offset412from the unmanned aircraft system to the surface of the aircraft is valid for the data point being evaluated.

In this illustrative example, image data validation404can be performed by controller208for various parameters in image plan222. As depicted in this example, image data validation404is performed by checking one or more parameters such as color414, field of view (FoV)416, and feature position418.

As depicted, color414can be validated in image data validation404using reference color, R; reference color, G; reference color, B; captured color, R; captured color, G; captured color, B; allowed color deviation, R; allowed color deviation, G; allowed color deviation, B.

In this illustrative example, the color deviation can be calculated as a difference between the reference color and the captured color for the RGB values. In this illustrative example, color deviation, R; color deviation, G; and color deviation, B the calculated from these differences. These deviations can be compared with allowed color deviation, R; allowed color deviation, G; allowed color deviation, B, respectively, to determine whether the color414is valid. This color can be for a sampling of pixels in the captured image.

Field of view416can be validated using reference field of view; actual field of view; and allowed field of view deviation. The field of view deviation can be calculated as the difference between the reference field of view and the actual field of view. The field of view deviation can be compared with allowed field of view deviation to determine whether the field of view416is valid.

Feature position418for a feature or a list of features can be validated using reference feature position (in pixel dimensions), px; reference feature position, py; captured feature position, px; captured feature position, py; and allowed feature position deviation for a feature of interest in the images. This validation of feature position418can be performed for one or more features in an image. The feature can be, for example, a rivet, a fairing, portal, a joint, or some other suitable feature on aircraft.

The position deviation can be calculated as the difference between the reference feature position coordinates and the captured feature position coordinates. The position deviation can be compared to the allowed feature deviation to determine whether feature position418is valid.

The illustration of image data validation404in this figure is provided an example and not meant to limit the manner in which validation can be performed in other illustrative examples. In other illustrative examples, validations using other data can be performed in addition to or in place of the validations depicted in this figure. For example, other types of image data can be validated in image data validation in addition to or in place of color414, field of view416, and feature position418. For example, other parameters that can be validated include at least one of focus, intensity, lighting, or other suitable parameters.

The illustration of aircraft inspection environment200and the different components inFIG.2-4is not meant to imply physical or architectural limitations to the manner in which an illustrative example may be implemented. Other components in addition to or in place of the ones illustrated may be used. Some components may be unnecessary. Also, the blocks are presented to illustrate some functional components. One or more of these blocks may be combined, divided, or combined and divided into different blocks when implemented in an illustrative example.

For example, one or more in unmanned aircraft systems in unmanned aircraft systems210in addition to unmanned aircraft system216can fly flight paths to inspect aircraft204. Further, one or more unmanned aircraft systems in unmanned aircraft systems210can fly flight path220. This flight may be flown by more than one unmanned aircraft system when different types of sensors are located on the sensor systems in unmanned aircraft systems210. In yet another illustrative example, controller208can control another unmanned aircraft system in unmanned aircraft systems210to inspect another aircraft in addition to aircraft204.

As another example, aircraft inspection system202can be configured to inspect aircraft204using other types of motion platforms or systems in addition to or in place of unmanned aircraft systems210. For example, vehicles selected from at least one of a crawler, an underwater unmanned vehicle (UUV), a remotely operate vehicle (ROV), an autonomous underwater vehicle (AUV), a stationary inspection robot with an arm, or other systems suitable for acquiring captured data224along a path for route control by controller208.

InFIG.5an illustration of a flowchart of a process for inspecting an aircraft is depicted in accordance with an illustrative example. The process inFIG.5can be implemented in hardware, software, or both. When implemented in software, the process can take the form of program code that is run by one of more processor units located in one or more hardware devices in one or more computer systems. For example, the process can be implemented in controller208in computer system206inFIG.2. This process can be implemented using unmanned aircraft systems210to capture data.

The process begins by acquiring airplane inspection requirements (operation500). These requirements can be obtained from various sources such manufacturer's requirements, a knowledgebase, use cases, and other sources. This information can be used to identify inspection areas as well as particular features on the aircraft. Features can include, for example, an antenna, a fairing, a wing surface, a door, a slat, a flap, an aileron, or other as features on aircraft that should be inspected. These requirements can be located in a database by particular aircraft and uploaded in operation500.

The process then evaluates environment conditions (operation502). Aircraft maintenance is performed in many environments. Some factors include for example lighting (e.g., reflection or glare from light sources, or shadows), temperature, air current, weather conditions (e.g., fog/smog, rain, etc.) and physical hanger layout. Aircraft inspection using unmanned aircraft systems takes into account the environment conditions to operate effectively.

The process generates a flight path and an image plan for a mission (operation504). In operation504, the process can use CAD model data or point cloud data, and path planning algorithms generate location data and offset distance data for the flight path. The process can also use CAD model data or point cloud data to determine specifics for imaging different areas for features on the aircraft. This mission is used by the unmanned aircraft system to perform the inspection of the aircraft. The illustrative example, the flight path generated takes into account the airplane inspection requirements determined in operation500and environmental conditions evaluated in step502.

The process loads the mission to the unmanned aircraft system (operation506). The process then starts navigation and data capture (operation508). In this operation, the unmanned aircraft system executes the mission including flying on the flight path. In the illustrative example, this operation is performed autonomously by the unmanned aircraft system. In other illustrative examples, human operator can remotely pilot some aspects of the unmanned aircraft system.

The process acquires image data, location data, and offset distance data along flight path (operation510). In this operation, the process acquires three-dimensional position and orientation data from the environment. This data can be acquired using, for example, an inertial measurement unit (IMU), light detection and ranging (LIDAR) system, a differential global positioning system (GPS) receiver, vision based localization or motion capture system. Image data can be acquired using one or more cameras on the unmanned aircraft system.

The process determines whether the location data, offset distance data, and image data are valid (operation512). In operation512, the process can validate the three-dimensional location data captured from three-dimensional localization sensors, such as a differential global positioning system (GPS) receiver, simultaneous localization mapping (SLAM) sensor, or a motion capture system. This captured location data can be compared to reference location data with variations in the comparison process being to specified tolerances.

In operation512, offset distance data is the distance to surface of the aircraft or component under inspection from the unmanned aircraft system and can be compared to the reference offset distance data. Variations in this comparison can become paired with specified tolerances.

In this illustrative example, imaging data can be validated by comparing acquired images in the captured image data to the reference image data to analyze parameters such as colors, field of view (FoV), and general feature positions, or other aspects of the image.

If the location data, offset distance data, and image data are not valid, the process modifies at least one of the flight path or image plan for the unmanned aircraft system (operation514). This modification can be performed by making adjustments to at least one of the flight path or image parameters to meet at least one of position or image tolerance requirements. The process then returns to operation510as described above.

For example, a shadow or reflection on a part of the aircraft camera and resulting image data being insufficient for analysis. With a reflection or shadow, image data for the part can be captured by modifying the flight path to capture image data from another angle or location. As another example, if low visibility conditions exist, the offset distance, a resolution of image, or other parameters in the image plan can be changed in operation514.

With reference again to operation512, if location data, offset distance data, and image data are valid, the process determines whether the flight path is complete (operation516). If the flight path is not complete, the process returns to operation510.

If the flight path is complete, the process analyzes the image data to find anomalies (operation518). This analysis in operation518can be made using independent mechanic inspection data and historical records519. These records can be generated to conduct mechanic manual inspections of the aircraft that references anomaly type, location, and size. Further, a mechanic image review from images captured by the unmanned aircraft system can be performed in which the review notes anomaly type, location, and size. These records can also include the results of a statistical analysis of comparisons between mechanic manual inspections and mechanic image review.

For example, comparison of the image analysis can be made to a manual account inspection of aircraft to generate the comparison between anomalies detected by image analysis and anomalies detected by mechanic manual inspection.

The process can also confirm unmanned aircraft system image equivalency to mechanic manual inspection.

Maintenance actions can be determined using the validated captured data (operation520). The maintenance actions needed depends on whether anomalies are found and if anomalies are found, the maintenance actions depend the type of anomalies present on the aircraft.

Fleet management decisions and analytics can be performed when using the validated captured data and the results of anomaly analysis (operation522). The process terminates thereafter. In operation522, images and dispositions can be stored in a database for determining alternate or more effective inspection requirements. This database can include the identification of anomalies for aircraft of the same type and for specific aircraft to maintain a history for types of aircraft as well as individual aircraft.

With reference now toFIG.6, a flowchart of a process for validating captured location data is depicted in accordance with an illustrative example. The process illustrated inFIG.6can be used in operation512inFIG.5to determine whether captured location data is valid for use in analysis. This process is independent of the source of the captured location data. The data comparison check performed in this process can be performed for each location in the captured location data for the flight path flown by the unmanned aircraft system. This process is also an example of location validation400performed by controller208inFIG.4.

The process begins by accessing reference x locations, reference y locations, and reference z locations in reference location data (operation600). In this depicted example, the location data is three-dimensional location data in a Cartesian coordinate system using x,y,z coordinates.

The process accesses captured locations x, captured y locations, and captured z locations in captured location data (operation602). In this example, each reference location in the reference location data has a corresponding captured location in the captured location data. The process then calculates location deviations between the reference location data and corresponding captured location data (operation604).

The process indicates that the captured location data is valid if the location deviations are within the allowed location deviation (operation608).

The process then accesses reference velocities in the x direction, reference velocities in the y direction, and reference velocities in the z direction in reference velocity data (operation610). The process then accesses captured velocities in the x direction, captured velocities, captured velocities y direction, and captured velocities in the z direction in captured velocity data (operation612). In this example, each reference velocity in the reference velocity data has a corresponding captured velocity in the captured velocity data.

The process calculates velocity deviations between the reference velocity data and corresponding captured velocity data (operation614). In this example, each captured velocity in the captured velocity data has a corresponding reference velocity in the reference velocity data. The process indicates that the captured velocity data is valid if the velocity deviations are within the allowed velocity deviations (operation616).

The process accesses reference roll angles, reference pitch angles, and reference yaw angles in reference orientation data (operation618). The process accesses captured roll angles, captured pitch angles, and captured yaw angles in captured orientation data (operation620). In this example, each reference orientation angle in the reference location data has a captured reference orientation angle in the captured location data.

The process calculates orientation deviations between the reference orientation data and the captured orientation data (operation622). The process indicates that the orientation deviation data is valid if the orientation deviations are within the allowed orientation deviations (operation624).

A determination is made as to whether all of the captured location data is valid (operation626). If all of the captured location data is valid, the process indicates that captured location data is valid for the flight path (operation628) with the process determining thereafter. Otherwise, the process indicates that captured location data is not valid (operation630) with the process terminating thereafter. The indications can include indicating each deviation that is in or out of tolerance.

With reference now toFIG.7, a flowchart of a process for validating captured offset distance data is depicted in accordance with an illustrative example. The process illustrated inFIG.7can be used in operation512inFIG.5to determine whether captured offset distance data is valid for use in analysis. This process is also an example of surface offset validation402performed by controller208inFIG.4.

The process begins by accessing reference offset distance data (operation700). The process accesses captured offset distances in captured offset distance data (702). In this example, each reference offset distance has a corresponding captured offset distance.

The process calculates distance deviations between the reference offset distances and the corresponding captured offset distances (operation704). A determination is made as to whether all of the distance deviations are within allowed deviations (operation705). If all of the captured location data is valid. If all of the distance deviations are within allowed deviations, the process indicates that the captured offset distance data is valid (operation706) with process terminating thereafter. Otherwise, the process indicates that the captured offset distance data is not valid (operation708) with process terminating thereafter. The indication can include indicating each distance deviation that is in tolerance or out of tolerance.

With reference now toFIG.8, a flowchart of a process for validating image data is depicted in accordance with an illustrative example. The process illustrated inFIG.8can be used in operation512inFIG.5to determine whether image data is valid for use in analysis. This process is also an example of image data validation404performed by controller208inFIG.4.

The process begins by accessing reference RGB values in reference color data (operation800). The process accesses a captured RGB values in the captured color data (operation802). In this example, each reference R (red) value, reference G (green) value, and reference B (blue) value in the reference color data has a corresponding captured R value, captured G value, and captured B value in the captured color data. The process then calculates color deviations between the reference RGB values and corresponding captured RGB values (operation804).

The process indicates that the captured color data is valid if the color deviations are within the allowed color deviations (operation806).

The process reference field of view values in the reference field of view data (operations808). The process accesses captured field of view values in the captured field of view data (operation810).

The process calculates field of view deviations between the reference field of view values and the captured field of view values (operation812). The process indicates that the captured field of view data is valid if the field of view deviations are within allowed field of view deviations (operation814).

The process accesses reference feature positions for features in reference feature data (operation816). In this illustrative example, a feature positions for a feature in an image can be described using x,y pixel coordinates. The process accesses captured feature positions for features in captured feature data (operation818). In this example, each reference feature positions for a feature has a corresponding captured feature position for the feature.

The process calculates feature position deviations for captured feature positions (operation820). The process indicates that the captured feature data is valid if the feature positions deviations are within allowed feature position deviations (operation822).

A determination is then made as to whether all of the captured image data is valid (operation824). If all of the captured location data is valid, the process indicates that the image data is valid for the flight path (operation826) with the process determining thereafter. Otherwise, the process indicates that the captured image data is not valid for the flight path (operation828) with the process terminating thereafter. The indications can include indicating each deviation that is in or out of tolerance.

Turning next toFIG.9an illustration of a flowchart of a process for inspecting an aircraft is depicted in accordance with an illustrative example. The process inFIG.9can be implemented in hardware, software, or both. When implemented in software, the process can take the form of program code that is run by one of more processor units located in one or more hardware devices in one or more computer systems. For example, the process can be implemented in controller208in computer system206inFIG.2.

The process begins by receiving captured data for a flight path flown by an unmanned aircraft system to acquire the captured data for the aircraft (operation900). The process compares the captured data with reference data for the aircraft to form a comparison (operation902).

The process determines whether the captured data is within a set of tolerances for valid captured data using the comparison (operation904). The process determines a set of corrective actions in response to the captured data being outside of the set of tolerances for the valid captured data in which the set of corrective actions is performed prior to detecting anomalies for the aircraft using the captured data (operation906).

The process performs the set of corrective actions using at least one of the unmanned aircraft system or another unmanned aircraft system (operation908). The process terminates thereafter.

With reference next toFIG.10, an illustration of a flowchart of a process for receiving captured location data is depicted in accordance with an illustrative example. The process illustrated inFIG.10is an example of one implementation for operation at900inFIG.9.

The process receives the captured location data for the flight path while the unmanned aircraft system is flying the flight path (operation1000). The process terminates thereafter. The captured data is for a portion of the flight path flown by the unmanned aircraft system.

With reference next toFIG.11, an illustration of a flowchart of a process for receiving captured data is depicted in accordance with an illustrative example. The process illustrated in FIG.11is an example of one implementation for operation at900inFIG.9.

The process receives the captured data for the flight path after the unmanned aircraft system has complete flying the flight path (operation1100). The process terminates thereafter. The captured data is for all of the flight path.

InFIG.12, an illustration of a flowchart of a process for comparing captured location data with reference location data is depicted in accordance with an illustrative example. The process illustrated inFIG.12is an example of one implementation for operation at902inFIG.9.

The process begins by accessing captured location data and captured offset distance data for the unmanned aircraft system in the captured data (operation1200). The process accesses reference location data and reference offset distance data in the reference data (operation1202).

The process compares the captured location data and the captured offset distance data to the reference location data and the reference offset distance data to create the comparison (operation1204). The process terminates thereafter.

With reference toFIG.13, an illustration of a flowchart of another process for comparing captured image data with reference image data is depicted in accordance with an illustrative example. The process illustrated inFIG.13is another example of one implementation for operation902inFIG.9.

The process begins by accessing captured image data in the captured data (operation1300). The process accesses reference image data in the reference data (operation1302).

The process compares the captured image data with the reference image data to form the comparison (operation1304). The process terminates thereafter.

With reference toFIG.14, an illustration of a flowchart of yet another process for comparing captured data with reference data is depicted in accordance with an illustrative example. The process illustrated inFIG.14is yet another example of an implementation for operation at902inFIG.9.

The process begins by accessing captured location data, captured offset distance data, and captured image data in the captured data (operation1400). The process accesses reference location data, reference offset distance data, and reference image data in the reference data (operation1402).

The process compares the captured location data, the captured offset distance data, and the captured image data to the reference location data, the reference offset distance data, and the reference imaged data, respectively, to form the comparison (operation1404). The process terminates thereafter.

Turning toFIG.15, an illustration of a flowchart of a process for indicating that the captured data is valid for analysis in response to captured data being valid is depicted in accordance with an illustrative example. The process inFIG.15is an example of an additional operation that can be performed with the operations inFIG.9.

The process indicates that the captured data is valid for analysis in response to determining that the captured data is valid (operation1500). The process terminates thereafter. In operation1500, the process can add an indicator by, for example, setting a flag, adding a certificate, sending a message, or provide some other type of indication that the captured data is valid.

Turning toFIG.16, an illustration of a flowchart of a process for analyzing the captured data for anomalies in response to captured data being valid is depicted in accordance with an illustrative example. The process inFIG.16is an example of an additional operation that can be performed with the operations inFIG.9after the capture data is determined to be valid.

The process analyzes the captured data for a set of anomalies for the aircraft in response to determining that the captured data is valid (operation1600). The process terminates thereafter.

Turning toFIG.17, an illustration of a flowchart of a process for confirming maintenance performed on an aircraft is depicted in accordance with an illustrative example. The process inFIG.17is an example of an additional operation that can be performed with the operations inFIG.9after maintenance is performed on an aircraft.

A set of maintenance actions is performed for the aircraft in response to the set of anomalies being identified for the aircraft (operation1700). These maintenance actions can include at least one of an additional inspection, repairing a part, reworking a part, replacing a part, adjust a part, or other suitable interactions.

The process flies the unmanned aircraft system on a portion of the flight path where the set of anomalies were located in response to the set of maintenance actions being performed on the set of anomalies (operation1702).

The process repeats receiving the captured location data for the portion of the flight path flown by the unmanned aircraft system to acquire the captured image data for the aircraft; comparing the captured data with the reference data for the aircraft to form the comparison; determining whether the captured data is within the set of tolerances for the valid captured data using on the comparison; and determining the set of corrective actions in response to the captured image data being outside of the set of tolerances for the valid captured data prior to detecting the set of anomalies for the aircraft using the image captured data (operation1704). The process terminates thereafter.

The flowcharts and block diagrams in the different depicted examples illustrate the architecture, functionality, and operation of some possible implementations of apparatuses and methods in an illustrative example. In this regard, each block in the flowcharts or block diagrams can represent at least one of a module, a segment, a function, or a portion of an operation or step. For example, one or more of the blocks can be implemented as program code, hardware, or a combination of the program code and hardware. When implemented in hardware, the hardware can, for example, take the form of integrated circuits that are manufactured or configured to perform one or more operations in the flowcharts or block diagrams. When implemented as a combination of program code and hardware, the implementation may take the form of firmware. Each block in the flowcharts or the block diagrams can be implemented using special purpose hardware systems that perform the different operations or combinations of special purpose hardware and program code run by the special purpose hardware.

In some alternative implementations of an illustrative example, the function or functions noted in the blocks may occur out of the order noted in the figures. For example, in some cases, two blocks shown in succession may be performed substantially concurrently, or the blocks may sometimes be performed in the reverse order, depending upon the functionality involved. Also, other blocks may be added in addition to the illustrated blocks in a flowchart or block diagram.

Turning now toFIG.18, an illustration of a block diagram of a data processing system is depicted in accordance with an illustrative example. Data processing system1800can be used to implement computer120inFIG.1and computer system206inFIG.2. In this illustrative example, data processing system1800includes communications framework1802, which provides communications between processor unit1804, memory1806, persistent storage1808, communications unit1810, input/output (I/O) unit1812, and display1814. In this example, communications framework1802takes the form of a bus system.

Processor unit1804serves to execute instructions for software that can be loaded into memory1806. Processor unit1804includes one or more processors. For example, processor unit1804can be selected from at least one of a multicore processor, a central processing unit (CPU), a graphics processing unit (GPU), a physics processing unit (PPU), a digital signal processor (DSP), a network processor, or some other suitable type of processor. Further, processor unit1804can may be implemented using one or more heterogeneous processor systems in which a main processor is present with secondary processors on a single chip. As another illustrative example, processor unit1804can be a symmetric multi-processor system containing multiple processors of the same type on a single chip.

Memory1806and persistent storage1808are examples of storage devices1816. A storage device is any piece of hardware that is capable of storing information, such as, for example, without limitation, at least one of data, program code in functional form, or other suitable information either on a temporary basis, a permanent basis, or both on a temporary basis and a permanent basis. Storage devices1816may also be referred to as computer-readable storage devices in these illustrative examples. Memory1806, in these examples, can be, for example, a random-access memory or any other suitable volatile or non-volatile storage device. Persistent storage1808can take various forms, depending on the particular implementation.

For example, persistent storage1808may contain one or more components or devices. For example, persistent storage1808can be a hard drive, a solid-state drive (SSD), a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination of the above. The media used by persistent storage1808also can be removable. For example, a removable hard drive can be used for persistent storage1808.

Communications unit1810, in these illustrative examples, provides for communications with other data processing systems or devices. In these illustrative examples, communications unit1810is a network interface card.

Input/output unit1812allows for input and output of data with other devices that can be connected to data processing system1800. For example, input/output unit1812can provide a connection for user input through at least one of a keyboard, a mouse, or some other suitable input device. Further, input/output unit1812can send output to a printer. Display1814provides a mechanism to display information to a user.

Instructions for at least one of the operating system, applications, or programs can be located in storage devices1816, which are in communication with processor unit1804through communications framework1802. The processes in the different examples can be performed by processor unit1804using computer-implemented instructions, which can be located in a memory, such as memory1806.

These instructions are program instructions and are also referred to as program code, computer usable program code, or computer-readable program code that can be read and executed by a processor in processor unit1804. The program code in the different examples can be embodied on different physical or computer-readable storage media, such as memory1806or persistent storage1808.

Program code1818is located in a functional form on computer-readable media1820that is selectively removable and can be loaded onto or transferred to data processing system1800for execution by processor unit1804. Program code1818and computer-readable media1820form computer program product1822in these illustrative examples. In the illustrative example, computer-readable media1820is computer-readable storage media1824.

Computer-readable storage media1824is a physical or tangible storage device used to store program code1818rather than a media that propagates or transmits program code1818. Computer-readable storage media1824, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Alternatively, program code1818can be transferred to data processing system1800using a computer-readable signal media. The computer-readable signal media are signals and can be, for example, a propagated data signal containing program code1818. For example, the computer-readable signal media can be at least one of an electromagnetic signal, an optical signal, or any other suitable type of signal. These signals can be transmitted over connections, such as wireless connections, optical fiber cable, coaxial cable, a wire, or any other suitable type of connection.

Further, as used herein, “computer-readable media1820” can be singular or plural. For example, program code1818can be located in computer-readable media1820in the form of a single storage device or system. In another example, program code1818can be located in computer-readable media1820that is distributed in multiple data processing systems. In other words, some instructions in program code1818can be located in one data processing system while other instructions in program code1818can be located in one data processing system. For example, a portion of program code1818can be located in computer-readable media1820in a server computer while another portion of program code1818can be located in computer-readable media1820located in a set of client computers.

The different components illustrated for data processing system1800are not meant to provide architectural limitations to the manner in which different examples can be implemented. In some illustrative examples, one or more of the components may be incorporated in or otherwise form a portion of, another component. For example, memory1806, or portions thereof, can be incorporated in processor unit1804in some illustrative examples. The different illustrative examples can be implemented in a data processing system including components in addition to or in place of those illustrated for data processing system1800. Other components shown inFIG.18can be varied from the illustrative examples shown. The different examples can be implemented using any hardware device or system capable of running program code1818.

Illustrative examples of the disclosure may be described in the context of aircraft manufacturing and service method1900as shown inFIG.19and aircraft2000as shown inFIG.20. Turning first toFIG.19, an illustration of an aircraft manufacturing and service method is depicted in accordance with an illustrative example. During pre-production, aircraft manufacturing and service method1900may include specification and design1902of aircraft2000inFIG.20and material procurement1904.

During production, component and subassembly manufacturing1906and system integration1908of aircraft2000inFIG.20takes place. Thereafter, aircraft2000inFIG.20can go through certification and delivery1910in order to be placed in service1912. While in service1912by a customer, aircraft2000inFIG.20is scheduled for routine maintenance and service1914, which may include modification, reconfiguration, refurbishment, and other maintenance or service.

Each of the processes of aircraft manufacturing and service method1900may be performed or carried out by a system integrator, a third party, an operator, or some combination thereof. In these examples, the operator may be a customer. For the purposes of this description, a system integrator may include, without limitation, any number of aircraft manufacturers and major-system subcontractors; a third party may include, without limitation, any number of vendors, subcontractors, and suppliers; and an operator may be an airline, a leasing company, a military entity, a service organization, and so on.

With reference now toFIG.20, an illustration of an aircraft is depicted in which an illustrative example may be implemented. In this example, aircraft2000is produced by aircraft manufacturing and service method1900inFIG.19and may include airframe2002with plurality of systems2004and interior2006. Examples of systems2004include one or more of propulsion system2008, electrical system2010, hydraulic system2012, and environmental system2014. Any number of other systems may be included. Although an aerospace example is shown, different illustrative examples may be applied to other industries, such as the automotive industry.

Apparatuses and methods embodied herein may be employed during at least one of the stages of aircraft manufacturing and service method1900inFIG.19.

In one illustrative example, components or subassemblies produced in component and subassembly manufacturing1906inFIG.19can be fabricated or manufactured in a manner similar to components or subassemblies produced while aircraft2000is in service1912inFIG.19. As yet another example, one or more apparatus examples, method examples, or a combination thereof can be utilized during production stages, such as component and subassembly manufacturing1906and system integration1908inFIG.19. One or more apparatus examples, method examples, or a combination thereof may be utilized while aircraft2000is in service1912, during maintenance and service1914inFIG.19, or both. The use of a number of the different illustrative examples may substantially expedite the assembly of aircraft2000, reduce the cost of aircraft2000, or both expedite the assembly of aircraft2000and reduce the cost of aircraft2000.

For example, aircraft inspection system202can be used to perform inspections of aircraft2000during maintenance and service1914. These inspections can be performed in a manner that generates validated inspection data that can be used to schedule maintenance. The inspection data can also be used to satisfy regulatory requirements for inspections. Further, aircraft inspection system202can use during manufacturing of aircraft2000such as during of system integration1908and certification and delivery1910.

Turning now toFIG.21, an illustration of a block diagram of a product management system is depicted in accordance with an illustrative example. Product management system2100is a physical hardware system. In this illustrative example, product management system2100includes at least one of manufacturing system2102or maintenance system2104.

Manufacturing system2102is configured to manufacture products, such as aircraft2000inFIG.20. As depicted, manufacturing system2102includes manufacturing equipment2106. Manufacturing equipment2106includes at least one of fabrication equipment2108or assembly equipment2110.

Fabrication equipment2108is equipment that used to fabricate components for parts used to form aircraft2000inFIG.20. For example, fabrication equipment2108can include machines and tools. These machines and tools can be at least one of a drill, a hydraulic press, a furnace, an autoclave, a mold, a composite tape laying machine, an automated fiber placement (AFP) machine, a vacuum system, a robotic pick and place system, a flatbed cutting machine, a laser cutter, a computer numerical control (CNC) cutting machine, a lathe, or other suitable types of equipment. Fabrication equipment2108can be used to fabricate at least one of metal parts, composite parts, semiconductors, circuits, fasteners, ribs, skin panels, spars, antennas, or other suitable types of parts.

Assembly equipment2110is equipment used to assemble parts to form aircraft2000inFIG.20. In particular, assembly equipment2110is used to assemble components and parts to form aircraft2000inFIG.20. Assembly equipment2110also can include machines and tools. These machines and tools may be at least one of a robotic arm, a crawler, a faster installation system, a rail-based drilling system, an unmanned aircraft system, or a robot. Assembly equipment2110can be used to assemble parts such as seats, horizontal stabilizers, wings, engines, engine housings, landing gear systems, and other parts for aircraft2000inFIG.20.

In this illustrative example, maintenance system2104includes maintenance equipment2112. Maintenance equipment2112can include any equipment needed to perform maintenance on aircraft2000inFIG.20. Maintenance equipment2112may include tools for performing different operations on parts on aircraft2000inFIG.20. These operations can include at least one of disassembling parts, refurbishing parts, inspecting parts, reworking parts, manufacturing replacement parts, or other operations for performing maintenance on aircraft2000inFIG.20. These operations can be for routine maintenance, inspections, upgrades, refurbishment, or other types of maintenance operations.

In the illustrative example, maintenance equipment2112may include ultrasonic inspection devices, x-ray imaging systems, vision systems, drills, crawlers, unmanned aircraft systems, and other suitable devices. In some cases, maintenance equipment2112can include fabrication equipment2108, assembly equipment2110, or both to produce and assemble parts that needed for maintenance.

Product management system2100also includes control system2114. Control system2114is a hardware system and may also include software or other types of components. Control system2114is configured to control the operation of at least one of manufacturing system2102or maintenance system2104. In particular, control system2114can control the operation of at least one of fabrication equipment2108, assembly equipment2110, or maintenance equipment2112.

The hardware in control system2114can be implemented using hardware that may include computers, circuits, networks, and other types of equipment. The control may take the form of direct control of manufacturing equipment2106. For example, robots, computer-controlled machines, and other equipment can be controlled by control system2114. In other illustrative examples, control system2114can manage operations performed by human operators2116in manufacturing or performing maintenance on aircraft2000. For example, control system2114can assign tasks, provide instructions, display models, or perform other operations to manage operations performed by human operators2116. In these illustrative examples, controller208inFIG.2can be implemented in control system2114to manage at least one of the manufacturing or maintenance of aircraft2000inFIG.20. For example, controller208can be implemented in control system2114control an unmanned aircraft system in maintenance equipment2112to perform inspections of an aircraft. As another example, controller208can control an unmanned aircraft system in manufacturing equipment2106to perform inspections of aircraft being manufactured by manufacturing system2102.

In the different illustrative examples, human operators2116can operate or interact with at least one of manufacturing equipment2106, maintenance equipment2112, or control system2114. This interaction can occur to manufacture aircraft2000inFIG.20.

Of course, product management system2100may be configured to manage other products other than aircraft2000inFIG.20. Although product management system2100has been described with respect to manufacturing in the aerospace industry, product management system2100can be configured to manage products for other industries. For example, product management system2100can be configured to manufacture products for the automotive industry as well as any other suitable industries.

Some features of the illustrative examples are described in the following clauses. These clauses are examples of features not intended to limit other illustrative examples.

Clause 1

A method for inspecting an aircraft comprising:receiving, by a computer system, captured data associated with a flight path that is flown by an unmanned aircraft system to acquire the captured data for the aircraft;comparing, by the computer system, the captured data with reference data for the aircraft to form a comparison;determining, by the computer system, whether the captured data is within a set of tolerances for valid captured data using a result of the comparison; andprior to detecting anomalies for the aircraft using the captured data. determining, by the computer system, a set of corrective actions in response to the captured data being outside of the set of tolerances for the valid captured data in which the set of corrective actions is performed.
Clause 2

The method according to clause 1 further comprising:performing, by the computer system, the set of corrective actions using at least one of the unmanned aircraft system or another unmanned aircraft system.
Clause 3

The method according to one of clauses 1 or 2, wherein receiving, by the computer system, the captured data for the flight path comprises:receiving, by the computer system, the captured data for the flight path while the unmanned aircraft system is flying the flight path, wherein the captured data is for a portion of the flight path flown by the unmanned aircraft system.
Clause 4

The method according to one of clauses 1, 2, or 3, wherein receiving, by the computer system, the captured data for the flight path comprises:receiving, by the computer system, the captured data for the flight path in response to the unmanned aircraft system completing flying the flight path, wherein the captured data is for all of the flight path.
Clause 5

The method according to one of clauses 1, 2, 3, or 4, wherein comparing the captured data comprises:accessing, by the computer system, captured location data and captured offset distance data for the unmanned aircraft system in the captured data;accessing, by the computer system, reference location data and reference offset distance data in the reference data; andcomparing, by the computer system, the captured location data and the captured offset distance data to the reference location data and the reference offset distance data, respectively, to create the comparison.
Clause 6

The method according to clause 5 further comprising:creating, by the computer system, the reference location data and the reference offset distance data using at least one of a computer aided design model of the aircraft, point cloud scans, or historical flight paths from one or more prior inspections of the aircraft.
Clause 7

The method according to one of clauses 1, 2, 3, 4, 5, or 6, wherein comparing the captured data with the reference data comprises:accessing, by the computer system, captured image data in the captured data;accessing, by the computer system, reference image data in the reference data; andcomparing, by the computer system, the captured image data with the reference image data to form the comparison.
Clause 8

The method according to clause 7 further comprising:creating, by the computer system, the reference image data using at least one of a computer aided design model of the aircraft, a point cloud scan of the aircraft, or historical images of the aircraft.
Clause 9

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, or 8, wherein comparing, by the computer system, the captured data comprises:accessing, by the computer system, captured location data, captured offset distance data, and captured image data in the captured data;accessing, by the computer system, reference location data, reference offset distance data, and reference image data in the reference data; and

comparing, by the computer system, the captured location data, the captured offset distance data, and the captured image data to the reference location data, the reference offset distance data, and the reference imaged data, respectively, to form the comparison.

Clause 10

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, 8, or 9 further comprising:indicating, by the computer system, that the captured data is valid for analysis in response to determining that the captured data is valid.
Clause 11

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10 further comprising:analyzing, by the computer system, the captured data for a set of anomalies for the aircraft in response to determining that the captured data is valid.
Clause 12

The method according to clause 11, wherein a set of maintenance actions for the aircraft are performed in response to the set of anomalies being identified for the aircraft.

Clause 13

The method of claim12further comprising:flying the unmanned aircraft system on a portion of the flight path where the set of anomalies were located in response to the set of maintenance actions being performed on the set of anomalies; andrepeating, by the computer system, receiving the captured data for the portion of the flight path flown by the unmanned aircraft system to acquire the captured data for the aircraft;comparing the captured data with the reference data for the aircraft to form the comparison;determining whether the captured data is within the set of tolerances for the valid captured data using on the comparison; anddetermining the set of corrective actions in response to the captured data being outside of the set of tolerances for the valid captured data prior to detecting the set of anomalies for the aircraft using the captured data.
Clause 14

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, or 13, wherein the set of corrective actions comprises is selected from at least one of repeating capturing data for the flight path, repeating image generation for a portion of the flight path on which captured data is not valid, repeating capturing data using an adjusted flight path, or adjusting settings of a camera system on the unmanned aircraft system.

Clause 15

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, or 14, wherein an automated analysis tool performs at least one of comparing the captured data with the reference data for the aircraft to form the comparison; determining whether the captured data is within the set of tolerances for the valid captured data using the comparison; or determining the set of corrective actions in response to the captured data being outside of the set of tolerances for the valid captured data.

Clause 16

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, or 15, wherein the aircraft is one of a commercial aircraft, a commercial passenger airplane, a rotorcraft, a tilt-rotor aircraft, a tilt wing aircraft, a vertical takeoff and landing aircraft, an electrical vertical takeoff and landing vehicle, and a partially manufactured aircraft.

Clause 17

An aircraft inspection system comprising:a computer system; anda controller in the computer system wherein the controller is configured to:receive captured data associated with a flight path that is flown by an unmanned aircraft system to acquire the captured data for an aircraft;compare the captured data with reference data for the aircraft to form a comparison;determine whether the captured data is within a set of tolerances for valid captured data using a result of the comparison; andprior to detecting anomalies for the aircraft using the captured data, determine a set of corrective actions in response to the captured data being outside the set of tolerances for the valid captured data in which the set of corrective actions is performed.
Clause 18

The aircraft inspection system according to clause 17, wherein the controller is configured to:perform, by the computer system, the set of corrective actions using at least one of the unmanned aircraft system or another unmanned aircraft system.
Clause 19

The aircraft inspection system according to one of clauses 17 or 18, wherein in receiving the captured data for the flight path, the controller is configured to:receive the captured data for the flight path while the unmanned aircraft system is flying the flight path, wherein the captured data is for a portion of the flight path flown by the unmanned aircraft system.
Clause 20

The aircraft inspection system according to one of clauses 17, 18, or 19, wherein in receiving the captured data for the flight path, is configured to:receive the captured data for the flight path in response to the unmanned aircraft system completing flying the flight path, wherein the captured data is for all of the flight path.
Clause 21

The aircraft inspection system according to one of clauses 17, 18, 19, or 20, wherein comparing the captured data to the reference data, the controller is configured to:access captured location data and captured offset distance data for the unmanned aircraft system in the captured data;access reference location data and reference offset distance data in the reference data; andcompare the captured location data and the captured offset distance data to the reference location data and the reference offset distance data to create the comparison.
Clause 22

The aircraft inspection system according to one of clauses 17, 18, 19, 20, or 21, wherein the controller is configured to:create the reference location data and the reference offset distance data using at least one of a computer aided design model of the aircraft, point cloud scan of the aircraft, or historical flight paths from one or more prior inspections of the aircraft.
Clause 23

The aircraft inspection system according to one of clauses 17, 18, 19, 20, or 21, wherein in comparing the captured data with the reference data, the controller is configured to:access captured image data in the captured data;access reference image data in the reference data; andcompare the captured image data with the reference image data to form the comparison.
Clause 24

The aircraft inspection system according to clause 23, wherein the controller is configured to:create the reference image data using at least one of a computer aided design model of the aircraft, a point cloud scan, or historical images of the aircraft.
Clause 25

The aircraft inspection system according to one of clauses 17, 18, 19, 20, 21, 22, 23, or 24, wherein in comparing the captured data to the references, wherein the controller is configured to:access captured location data, captured offset distance data, and captured image data in the captured data;access reference location data, reference offset distance data, and reference image data in the reference data; andcompare the captured location data, the captured offset distance data, and the captured image data to the reference location data, the reference offset distance data, and the reference imaged data, respectively, to form the comparison.
Clause 26

The aircraft inspection system according to one of clauses 17, 18, 19, 20, 21, 22, 23, 24, or 25, wherein the controller is configured to:indicate that the captured data is valid for analysis in response to determining that the captured data is valid.
Clause 27

The aircraft inspection system according to one of clauses 17, 18, 19, 20, 21, 22, 23, 24, 25, or 26, wherein the captured data is analyzed for a set of anomalies for the aircraft in response to determining that the captured data is valid.

Clause 28

The aircraft inspection system according to clause 27, wherein a set of maintenance actions is performed for the aircraft in response to the set of anomalies being identified for the aircraft.

Clause 29

The aircraft inspection system according to clause 28, wherein the controller is configured to:fly the unmanned aircraft system on a portion of the flight path where the set of anomalies were located in response to the set of maintenance actions being performed on the set of anomalies; andrepeat receiving the captured data for the portion of the flight path flown by the unmanned aircraft system to acquire the captured data for the aircraft; comparing the captured data with the reference data for the aircraft to form the comparison; determining whether the captured data is within the set of tolerances for the valid captured data using on the comparison; and determining the set of corrective actions in response to the captured data being outside of the set of tolerances for the valid captured data prior to detecting the set of anomalies for the aircraft using the captured data.
Clause 30

The aircraft inspection system according to one of clauses 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, or 29, wherein the set of corrective actions comprises is selected from at least one of repeating capturing data for the flight path, repeating image generation for a portion of the flight path on which invalid captured data is present, repeating capturing data using an adjusted flight path, or adjusting settings of a camera system on the unmanned aircraft system.

Clause 31

The aircraft inspection system according to one of clauses 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, or 30, wherein an automated analysis tool performs at least one of comparing the captured data with the reference data for the aircraft to form the comparison; determining whether the captured data is within the set of tolerances for the valid captured data using the comparison; or determining the set of corrective actions in response to the captured data being outside of the set of tolerances for the valid captured data.

Clause 32

The aircraft inspection system according to one of clauses 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, or 31, wherein the aircraft is one of a commercial aircraft, a rotorcraft, a tilt-rotor aircraft, a tilt wing aircraft, a vertical takeoff and landing aircraft, an electrical vertical takeoff and landing vehicle, and a partially manufactured aircraft.

Clause 33

A computer program product for inspecting an aircraft, the computer program product comprising a computer readable storage media having program instructions embodied therewith, the program instructions executable by a computer system to cause the computer system to perform a method of:receiving captured data associated with a flight path that is flown by an unmanned aircraft system to acquire the captured data for the aircraft;comparing the captured data with reference data for the aircraft to form a comparison;determining whether the captured data is within a set of tolerances for valid captured data using a result of the comparison; anddetermining a set of corrective actions in response to the captured data being outside the set of tolerances for the valid captured data in which the set of corrective actions is performed prior to detecting anomalies for the aircraft using the captured data.

Thus, illustrative examples provide a method, apparatus, system, and computer program product for inspecting aircraft. In one illustrative example, a computer system, receives captured locations and image data for a flight path flown by an unmanned aircraft system to acquire the captured data for the aircraft. The computer system compares the captured data with reference data for the aircraft to form a comparison. The computer system determines whether the captured data is within a set of tolerances for valid captured data using the comparison. The computer system determines a set of corrective actions in response to the captured data being outside of the set of tolerances for the valid captured data in which the set of corrective actions is performed prior to detecting anomalies for the aircraft using the captured data.

Thus, an automated inspection system in the illustrative examples can operate to validate captured data such as flight paths and captured image data to determine whether this data can be used for determining whether anomalies are present on aircraft. Further, this validation can also be performed to meet regulatory requirements or manufacturer requirements or performing inspections.

The description of the different illustrative examples has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the examples in the form disclosed. The different illustrative examples describe components that perform actions or operations. In an illustrative example, a component can be configured to perform the action or operation described. For example, the component can have a configuration or design for a structure that provides the component an ability to perform the action or operation that is described in the illustrative examples as being performed by the component. Further, To the extent that terms “includes”, “including”, “has”, “contains”, and variants thereof are used herein, such terms are intended to be inclusive in a manner similar to the term “comprises” as an open transition word without precluding any additional or other elements.

Many modifications and variations will be apparent to those of ordinary skill in the art. Further, different illustrative examples may provide different features as compared to other desirable examples. The example or examples selected are chosen and described in order to best explain the principles of the examples, the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various examples with various modifications as are suited to the particular use contemplated.