Patent Description:
Assemblies such as aircraft may include hundreds or thousands fasteners, which typically require substantial time to inspect, to verify that the fasteners are installed properly. For example, proper installation of a fastener for a given assembly may require that the fastener is perpendicular to the surfaces it joins, such that fasteners installed too oblique to the surface would be considered inadequate. When fastener installation is automated using robots to drill, install, and fasten rivets, for example, the ability to perform in-process open hole inspection may be lost. Current inspection techniques to inspect such fasteners installed in automated processes are destructive in nature. For example, rivet surface measurement is done using manual probes that give point measurements after removing fasteners for inspection. Furthermore, these techniques tend to be time-consuming, costly, and not very repeatable due to being operator-dependent.

The document <CIT> states, in accordance with its abstract, a non-destructive imaging of the internal structure for safe and intuitive operator work. In this context, electronic scanning first creates a virtual image of the surface of the object whose internal structure is the subject of research. Part of the surface of the object and the angle of scanning are set by voice or by movement of the operator's body. The virtual image of the surface of the object is subsequently projected in the stereoscopic glasses, followed by creation of the virtual image of the internal structure of the object for the same angle of scanning. The virtual image of the internal structure is projected in the virtual image of the surface of the object, or replaces the virtual image of the object.

According to the present disclosure, a system for inspecting a fastener installed at least partially through a hole in a part, as defined in the independent claim <NUM>, and a method of inspecting the fastener, as defined in the independent claim <NUM>, are provided. Further embodiments of the invention are defined in the dependent claims. Although the invention is only defined by the claims, the below embodiments, examples, and aspects are present for aiding in understanding the background and advantages of the invention.

Presently disclosed systems and methods may be configured to provide nondestructive, quick, and automated inspection of installed fasteners in an assembly, such as an aircraft component. Such systems and methods may be configured to measure fastener concentricity, fastener flushness with a surface, and/or detect the presence of foreign object debris.

In an example, a system for inspecting a fastener installed at least partially through a hole in a part includes an x-ray imaging system, a first camera device, a second camera device, a first support structure, and at least one processing unit. The x-ray imaging system is oriented and configured to produce an x-ray image of the fastener. The first camera device is positioned and oriented such that it is configured to produce a first image of the fastener from a first vantage point, and the second camera device is positioned and oriented such that it is configured to produce a second image of the fastener from a second vantage point. The x-ray imaging system, the first camera device, and the second camera device are coupled to the first support structure. The first support structure is configured to support and position the first camera device and the second camera device relative to the part and the fastener such that a 3D image of the fastener can be created from the first image and the second image. The at least one processing unit is configured to create the 3D image of the fastener from the first image and the second image, and is further configured to inspect the fastener based on the x-ray image and the 3D image, to determine concentricity and flushness of the fastener.

Disclosed methods of inspecting a fastener installed at least partially through a hole in a part generally include creating an x-ray image of the fastener via an x-ray imaging system, and measuring concentricity of the fastener, using the x-ray image. Disclosed methods include creating a 3D image of the fastener using a first image of the fastener and a second image of the fastener, wherein the first image is taken from a first vantage point, via a first camera device, and wherein the second image is taken from a second vantage point, via a second camera device, and measuring flushness of the fastener using the 3D image. Computer readable media having non-transitory computer readable instructions that, when executed by a processing unit, cause the processing unit to perform the disclosed methods are also disclosed.

<FIG>, <FIG>, and <FIG> provide illustrative, non-exclusive examples of systems <NUM> according to the present disclosure. Elements that serve a similar, or at least substantially similar, purpose are labeled with like numbers in each of <FIG>, <FIG>, and <FIG>, and these elements may not be discussed in detail herein with reference to each of <FIG>, <FIG>, and <FIG>. Similarly, all elements may not be labeled in each of <FIG>, <FIG>, and <FIG>, but reference numerals associated therewith may be utilized herein for consistency. Elements, components, and/or features that are discussed herein with reference to one or more of <FIG>, <FIG>, and <FIG> may be included in and/or utilized with any of <FIG>, <FIG>, and <FIG> without departing from the scope of the present disclosure. In general, elements that are likely to be included in a given (i.e., a particular) embodiment are illustrated in solid lines, while elements that are optional to a given embodiment are illustrated in dashed lines. However, elements that are shown in solid lines are not essential to all embodiments, and an element shown in solid lines may be omitted from a particular embodiment without departing from the scope of the present disclosure.

With reference to <FIG>, <FIG>, and <FIG>, systems <NUM> are configured for inspecting one or more fasteners <NUM> installed at least partially through one or more respective holes <NUM> in a part <NUM>. Examples will be described herein with reference to one or more fasteners <NUM> installed in an aircraft component (e.g., rivets in a wing panel). However, systems <NUM> may be used to inspect fasteners <NUM> installed in a wide variety of parts <NUM> from any industry, including but not limited to, the automotive, aerospace, marine, OEM, military, construction, transit, shipping, shelter, and cargo industries. Fasteners <NUM> may be configured to be at least substantially flush with one side of part <NUM>, at least substantially flush with both sides of part <NUM>, and/or may extend beyond one or both sides of part <NUM>, as schematically represented in <FIG>. Systems <NUM> are configured to inspect fasteners <NUM> non-destructively, meaning that fasteners <NUM> may be inspected without being damaged or removed from part <NUM>. Systems <NUM> generally inspect fasteners <NUM> using a combination of radiography and image processing, combining computer vision and image processing algorithms for real-time detection and decision-making. Systems <NUM> further may be configured to inspect fastener <NUM> with only one-sided access to part <NUM>, in some examples.

Systems <NUM> generally include an x-ray imaging system <NUM>, a first camera device <NUM>, and a second camera device <NUM>. X-ray imaging system <NUM> is oriented and configured to produce an x-ray image of one or more fasteners <NUM> in part <NUM>. In some examples, x-ray imaging system <NUM> may be configured to produce an x-ray image of a single fastener <NUM> (e.g., fastener 12a) at a time, and then may be moved with respect to part <NUM> to produce other x-ray images of other fasteners (e.g., fastener 12b). In some examples, x-ray imaging system <NUM> may be configured to produce an x-ray image that encompasses a plurality of fasteners in a single x-ray image (e.g., fasteners 12a, 12b, and 12c).

Similarly, first camera device <NUM> is positioned and oriented such that it is configured to produce a first image of one or more fasteners <NUM> in part <NUM>, from a first vantage point. In some examples, first camera device <NUM> may be configured to produce a respective first image of a single fastener <NUM> (e.g., fastener 12a) at a time, and then may be moved with respect to part <NUM> to produce other first images of other fasteners (e.g., fastener 12b). In some examples, first camera device <NUM> may be configured to produce a first image that encompasses a plurality of fasteners in a single first image (e.g., fasteners 12a, 12b, and 12c). Likewise, second camera device <NUM> is positioned and oriented such that it is configured to produce a second image of one or more fasteners <NUM> in part <NUM>, from a second vantage point. In some examples, second camera device <NUM> may be configured to produce a respective second image of a single fastener <NUM> (e.g., fastener 12a) at a time, and then may be moved with respect to part <NUM> to produce other second images of other fasteners (e.g., fastener 12b). In some examples, second camera device <NUM> may be configured to produce a second image that encompasses a plurality of fasteners in a single second image (e.g., fasteners 12a, 12b, and 12c).

Systems <NUM> also include at least one processing unit <NUM> configured to receive information (e.g., x-ray images, first images, and second images) from x-ray imaging system <NUM>, first camera device <NUM>, and second camera device <NUM>. Said information may be transmitted to processing unit <NUM> wirelessly, or processing unit <NUM> may be directly electrically coupled to x-ray imaging system <NUM>, first camera device <NUM>, and/or second camera device <NUM>. Processing unit <NUM> is configured to create a 3D image of one or more fasteners <NUM> from said information, such as using stereovision techniques. Processing unit <NUM> is configured to inspect fastener <NUM> based on the x-ray image received from x-ray imaging system <NUM>, and based on the 3D image of fastener <NUM>, as will be described in further detail herein.

Systems <NUM> also include a first support structure <NUM>, to which x-ray imaging system <NUM>, first camera device <NUM>, and second camera device <NUM> are coupled. First support structure <NUM> is configured to support and position first camera device <NUM> and second camera device <NUM> relative to part <NUM> and the fastener <NUM> being inspected such that a 3D image of fastener <NUM> can be created from the first image produced by first camera device <NUM> and the second image produced by second camera device <NUM>. Generally, first support structure <NUM> serves to position first camera device <NUM> and second camera device <NUM> such that first camera device <NUM> and second camera device <NUM> take images of fastener <NUM> from different angles, or vantage points. Support structure <NUM> generally supports x-ray imaging system <NUM> in a position in between first camera device <NUM> and second camera device <NUM>, though systems <NUM> are not limited to this arrangement. In some examples, first support structure <NUM> may include two or more support structures that may be linked together or that may move independently of one another, for positioning x-ray imaging system <NUM>, first camera device <NUM>, and second camera device <NUM> with respect to part <NUM> and fastener <NUM>.

First support structure <NUM> may be coupled to a first robot <NUM> that may be configured to control, orient, move, and/or position first support structure <NUM>, such as via a first robot arm <NUM> (<FIG>). First robot arm <NUM> may be configured to move first support structure <NUM> relative to part <NUM>, thereby adjusting a position of x-ray imaging system <NUM>, first camera device <NUM>, and second camera device <NUM> relative to fastener <NUM>. Systems <NUM> also may include an x-ray detector <NUM>, which may be coupled to a second support structure <NUM> and/or a second robot <NUM> (e.g., via a second robot arm <NUM> as shown in <FIG>). Second robot arm <NUM> may be configured to move second support structure <NUM> relative to part <NUM>, thereby adjusting a position of x-ray detector <NUM> relative to fastener <NUM>. As shown in <FIG> and <FIG>, x-ray imaging system <NUM> may be positioned on one side of part <NUM>, while x-ray detector <NUM> may be positioned on the opposite side of part <NUM>. Accordingly, first robot arm <NUM> may be positioned on one side of part <NUM>, while second robot arm <NUM> may be positioned on the opposite side of part <NUM>. In systems <NUM> including first robot <NUM> and second robot <NUM>, first robot <NUM> may be a commander, primary, or leader robot, that dictates movement of second robot <NUM>, which may be a secondary, or follower, robot. In some examples, first robot <NUM> and second robot <NUM> may be calibrated such that they move in tandem and with second robot arm <NUM> positioning x-ray detector <NUM> to face first support structure <NUM> and x-ray imaging system <NUM>, as positioned by first robot arm <NUM>, on the opposite side of part <NUM>.

Systems <NUM> are configured to measure concentricity of one or more fasteners <NUM> and flushness of one or more fasteners <NUM> with respect to part <NUM>. As used herein, "concentricity" of a fastener <NUM> refers to concentricity between two sides of the fastener, because fasteners <NUM> are configured to be perpendicular to part <NUM> when correctly installed in presently disclosed examples. Thus, when installed properly (i.e., such that a longitudinal axis <NUM> of fastener <NUM> is perpendicular to a plane defined by part <NUM>), a circular plane defined by a first side <NUM> of fastener <NUM> is substantially concentric with a circular plane defined by a second side <NUM> of fastener <NUM>. As used herein, concentricity between first side <NUM> and second side <NUM> of fastener <NUM> also generally indicates concentricity with respect to hole <NUM> in which fastener <NUM> is installed.

With reference to <FIG>, processing unit <NUM> (<FIG>) may be configured to detect a first circle <NUM> and a second circle <NUM> from an x-ray image <NUM> of fastener <NUM>. First circle <NUM> corresponds to first side <NUM> of fastener <NUM> (e.g., the head of fastener <NUM>), while second circle <NUM> corresponds to second side <NUM> of fastener <NUM> (e.g., the shaft of fastener <NUM>). For example, processing unit <NUM> may include instructions to detect circular features whose nominal diameter is that of fastener <NUM> (e.g., the head or shaft of fastener <NUM>). Processing steps may include basic denoising (Gaussian blurring, median filter, etc.), Canny edge detection, and/or Hough Circle detection.

Processing unit <NUM> is further configured to determine a first center <NUM> of first circle <NUM> and a second center <NUM> of second circle <NUM>, and to determine the offset, or distance, between first center <NUM> and second center <NUM>. Processing unit <NUM> may then determine whether the offset is within a predetermined pass/fail threshold offset distance. If first center <NUM> is farther from second center <NUM> than the predetermined pass/fail threshold offset distance, this may indicate that first circle <NUM> and second circle <NUM> are not sufficiently concentric, and therefore that fastener <NUM> is not adequately installed in part <NUM> because it is not sufficiently perpendicular to part <NUM>. In other words, the greater the offset between first center <NUM> and second center <NUM>, the more oblique fastener <NUM> may be with respect to part <NUM>. <FIG> illustrates this concept in a different way, showing a fastener 12d that is adequately installed in part <NUM>, whereas a fastener 12e is oblique to part <NUM>, and thus inadequately installed. X-ray images <NUM> of each fastener 12d, 12e are also shown. For example, x-ray image 70d of fastener 12d illustrates that first circle <NUM> and second circle <NUM> are substantially concentric, which would correspond to a "pass" result for measuring concentricity of fastener 12b. On the other hand, x-ray image 70e of fastener 12e reveals that first circle <NUM> is significantly offset from second circle <NUM> (due to the obliqueness of fastener 12e with respect to part <NUM>), and thus x-ray image 70e is an example of an image that would be interpreted as indicating an inadequately installed fastener, under certain threshold criteria. Fastener 12e accordingly may be indicated as inadequately installed and flagged for repair (e.g., reinstallation), removal, and/or replacement.

The predetermined threshold offset distance may be stored, for example, on a non-transitory computer readable memory <NUM> (also referred to herein as memory <NUM>) (<FIG>) of system <NUM>. In this manner, system <NUM> may determine the quality of the installation of fastener <NUM>, as determined by concentricity of the two sides <NUM>, <NUM> of fastener <NUM>. Processing unit <NUM> may include an x-ray image processing module for determining such concentricity of fastener <NUM>. Systems <NUM> may be said to be configured to inspect first side <NUM> (also referred to herein as first portion <NUM>) and second side <NUM> (also referred to herein as second portion <NUM>) of fastener <NUM> at least substantially simultaneously, because x-ray imaging system <NUM> may be configured to create an x-ray image of both first side <NUM> and second side <NUM> of fastener <NUM> in a single x-ray image.

As used herein, "flushness" refers to the degree to which the fastener <NUM> protrudes beyond a surface of part <NUM>, though systems <NUM> may be used to inspect fasteners <NUM> whether or not the fasteners are configured to be installed in hole <NUM> such that they are flush with part <NUM>. In other words, in some examples, fastener <NUM> may be entirely within hole <NUM> when properly installed, while in other examples, at least a portion of fastener <NUM> may be positioned externally to hole <NUM> when fastener <NUM> is properly installed. For example, fastener 12b is installed completely in hole 14b in <FIG>, such that first side, or first portion, 42b of fastener 12b is positioned adjacent a first surface <NUM> of part <NUM> and such that a second side, or second portion, 44b of fastener 12b is positioned adjacent a second surface <NUM> of part <NUM>, with first surface <NUM> facing the first side of part <NUM> and second surface <NUM> facing the second side of part <NUM>. On the other hand, first side <NUM> and second side <NUM> of fastener 12a are positioned externally to its hole <NUM>, such that they protrude beyond first surface <NUM> and second surface <NUM>, respectively, of part <NUM>.

In some examples, system <NUM> is configured to inspect a height of a head of fastener <NUM> with respect to first surface <NUM> and/or second surface <NUM> of part <NUM>, thereby measuring flushness of fastener <NUM>. Processing unit <NUM> may include a 3D reconstruction module configured to determine flushness of fastener <NUM> by creating the 3D image of fastener <NUM> and the surface of part <NUM>. For example, once the 3D image of the fastener <NUM> is created from two or more images of fastener <NUM>, a point cloud of the 3D surface may be generated by processing unit <NUM>. Based on the 3D image, processing unit <NUM> may be configured to identify a surface of fastener <NUM> (e.g., the surface corresponding to first side <NUM>), and to identify a surface of part <NUM> (e.g., first surface <NUM>). Processing unit <NUM> may then be configured to fit a first plane to the surface of fastener <NUM>, and to fit a second plane to the surface of part <NUM>, in order to determine a height difference between the first plane and the second plane, thereby measuring flushness of fastener <NUM> with respect to first surface <NUM> of part <NUM>. One or more filters may be applied by processing unit <NUM> to the point cloud of the 3D surface of fastener <NUM> to identify the two surfaces, in some examples.

Additionally or alternatively, systems <NUM> may be configured to detect foreign object debris around fastener <NUM>, such as debris generated during installation of fastener <NUM>. Processing unit <NUM> may include a 2D processing module configured to detect foreign object debris around fastener <NUM>. In some examples of system <NUM>, processing unit <NUM> may be configured to identify objects with a surface area that is dissimilar to that of the head of fastener <NUM>, using image processing techniques (e.g., image classification algorithms), and based on the x-ray image of fastener <NUM>, the 3D reconstruction of fastener <NUM> and/or part <NUM>, and/or 2D images of fastener <NUM> and/or part <NUM>. For example, processing unit <NUM> may be configured to detect circular features having a diameter matching a known diameter of a head of fastener <NUM>, and to apply high pass filtering to eliminate background, and thereby detect any foreign object debris adjacent, or in the vicinity of, fastener <NUM>.

Memory <NUM> of processing unit <NUM> may store non-transitory computer readable instructions that, when executed by processing unit <NUM>, cause processing unit <NUM> to measure concentricity of fastener <NUM>, measure flushness of fastener <NUM> with respect to part <NUM>, and/or detect foreign object debris near fastener <NUM>. Threshold pass/fail criteria for fastener inspections may be stored on memory <NUM>. Furthermore, the instructions, when executed by processing unit <NUM>, may be configured to cause processing unit <NUM> to automatically mark, indicate, or flag any detected defective fasteners <NUM>.

Systems <NUM> may be configured to inspect a plurality of fasteners <NUM> installed in part <NUM>, while first robot <NUM> may be configured to scan part <NUM> as the plurality of fasteners <NUM> are inspected, in order to determine contours of part <NUM>. In this manner, processing unit <NUM> may determine movement of first robot arm <NUM>, based on the scan of part <NUM>, in order to position x-ray imaging system <NUM>, first camera device <NUM>, and second camera device <NUM> relative to the respective fastener <NUM> of the plurality of fasteners being inspected. In some examples, first support structure <NUM> is configured to be moved with respect to part <NUM> in between subsequent respective fastener inspections, such that x-ray imaging system <NUM>, first camera device <NUM>, and second camera device <NUM> are correctly positioned and oriented with respect to each respective fastener <NUM> being inspected. Second robot <NUM> may be configured to synchronously scan the part, along with first robot <NUM>, as the plurality of fasteners <NUM> are inspected, in order to determine contours of part <NUM>, thereby determining movement of second robot arm <NUM> to position x-ray detector <NUM> relative to the respective fastener <NUM> of the plurality of fasteners being inspected, or relative to first support structure <NUM>.

As shown in <FIG>, commander robot <NUM> and/or follower robot <NUM> may be positioned on a sliding motion stage <NUM>. Sliding motion stage <NUM> may be configured to slide, or translate, within a workspace environment in order to position first robot <NUM> and second robot <NUM> to examine fasteners <NUM> in a plurality of different, spaced apart, parts <NUM>. Additionally or alternatively, sliding motion stage <NUM> may be configured to allow for movement of first robot <NUM> and/or second robot <NUM> with respect to part <NUM>, such that systems <NUM> may be positioned and oriented as desired for inspecting fasteners <NUM>.

First support structure <NUM> may be, or include, a first rigid linear platform, in some systems <NUM>. For example, first support structure <NUM> may be, or include, a C-beam railing. Similarly, second support structure <NUM> may be, or include, a second rigid linear platform, such as a C-beam railing. An example of a structure that may serve as first support structure <NUM> and/or second support structure <NUM> is illustrated in <FIG>. X-ray imaging system <NUM>, first camera device <NUM>, and second camera device <NUM> may be coupled to first support structure <NUM> collinearly.

With continued reference to <FIG>, <FIG>, and <FIG>, X-ray imaging system <NUM> is a portable, mobile, and/or handheld x-ray imaging system <NUM> in some examples, though generally x-ray imaging system <NUM> may be any x-ray system configured to produce x-ray images of fastener <NUM>. First camera device <NUM> and/or second camera device <NUM> may be first and second pinhole cameras in some examples, though generally first camera device <NUM> and second camera device <NUM> may be any camera device configured to produce images of fastener <NUM>. System <NUM> in general may be portable and mounted on various robot arms in different work cells or locations.

Systems <NUM> may be configured to automatically mark, or indicate, defectively installed or unsatisfactorily installed fasteners, if any are deemed unsatisfactory as a result of inspection. Such fasteners that fail inspection may be flagged for manual inspection and/or for removal from part <NUM>. In some examples, systems <NUM> may be configured to inspect each fastener <NUM> in a given part <NUM>. In other examples, systems <NUM> may be configured to inspect just a subset of fasteners <NUM> in a given part <NUM>, though said systems <NUM> may be configured to inspect additional fasteners <NUM> in the vicinity of a defectively installed fastener. First robot arm <NUM> also may be used to install fasteners <NUM> in part <NUM>, and systems <NUM> may thus inspect such fasteners <NUM> in real-time, as they are installed. Systems <NUM> also may be configured for automated examination of x-ray images produced by x-ray imaging system <NUM>, such that fastener concentricity may be determined in real-time, at the time the fastener is installed.

As noted above, x-ray imaging system <NUM> may be positioned between first camera device <NUM> and second camera device <NUM>. In some examples, first camera device <NUM> and second camera device <NUM> are symmetrically positioned on either side of x-ray imaging system <NUM>, such that first camera device <NUM> and second camera device <NUM> may be at least substantially equidistant from x-ray imaging system <NUM>. Additionally or alternatively, first camera device <NUM> and second camera device <NUM> may be symmetrically positioned on either side of the center of hole <NUM> of fastener <NUM> being inspected.

With reference to <FIG>, a distance <NUM> between first camera device <NUM> and second camera device <NUM>, a first angle <NUM> of first camera device <NUM> with respect to first support structure <NUM> (<FIG>), and/or a second angle <NUM> of second camera device <NUM> with respect to first support structure <NUM> may be selectively adjusted to position fastener <NUM> within a first central region <NUM> of a first field of view <NUM> of first camera device <NUM> and within a second central region <NUM> of a second field of view <NUM> of second camera device <NUM>. Distance <NUM> between first camera device <NUM> and second camera device <NUM> (and/or angles <NUM> and <NUM>) may be chosen such that fastener <NUM> is positioned at the center of first field of view <NUM> and second field of view <NUM>. In some examples, distance <NUM> may be about <NUM>,<NUM>, about <NUM>,<NUM>, about <NUM>,<NUM>, about <NUM>,<NUM>, about <NUM>, <NUM>, about <NUM>,<NUM>, about <NUM>,<NUM>, about <NUM>,<NUM>, about <NUM>,<NUM>, about <NUM>,<NUM>, and/or greater than <NUM>,<NUM> (<NUM> inches, about <NUM> inches, about <NUM> inches, about <NUM> inches, about <NUM> inches, about <NUM> inches, about <NUM> inches, about <NUM> inches, about <NUM> inches, about <NUM> inches, and/or greater than <NUM> inches). Distance <NUM> may vary depending on the size of part <NUM>, the size of fastener <NUM>, and/or first angle <NUM> and second angle <NUM>. First angle <NUM> and second angle <NUM> may be at least substantially equal in some examples. In other examples, first angle <NUM> may be different from second angle <NUM>. First angle <NUM> and/or second angle <NUM> may be about <NUM> degrees in some examples. In other examples, first angle <NUM> and/or second angle <NUM> may be between <NUM>-<NUM> degrees, between <NUM>-<NUM> degrees, between <NUM>-<NUM> degrees, and/or between <NUM>-<NUM> degrees.

Additionally or alternatively, a first standoff distance <NUM> between first support structure <NUM> and fastener <NUM> may be optimized for both stereographs and radiographs. For example, first standoff distance <NUM> may be at least <NUM>,<NUM>, at least <NUM>,<NUM>, at least <NUM>,<NUM>, at least <NUM>,<NUM>, at least <NUM>,<NUM>, at least <NUM>,<NUM>, at least <NUM>,<NUM>, and/or at least <NUM>,<NUM> (at least <NUM> inches, at least <NUM> inches, at least <NUM> inches, at least <NUM> inches, at least <NUM> inches, at least <NUM> inches, at least <NUM> inches, and/or at least <NUM> inches). In a specific example, first standoff distance <NUM> may be between <NUM>-<NUM> inches. First standoff distance <NUM> may be selected or set relative to distance <NUM> between first camera device <NUM> and second camera device <NUM>, in view of first angle <NUM> and second angle <NUM>. First standoff distance <NUM> may be less than distance <NUM>, such as about <NUM>% of distance <NUM>, about <NUM>% of distance <NUM>, and/or about <NUM>% of distance <NUM>. In some examples, first standoff distance <NUM> may be between <NUM>-<NUM>% of distance <NUM>. In other examples, first standoff distance <NUM> may be greater than distance <NUM>. Additionally or alternatively, a second standoff distance between x-ray detector <NUM> (<FIG>) and fastener <NUM> may be selectively adjusted according to a desired geometric magnification of the x-ray image produced by x-ray imaging system <NUM>. First standoff distance <NUM> may be selectively adjusted via movement of first robot arm <NUM>, while the second standoff distance may be selectively adjusted via movement of second robot arm <NUM> (<FIG>).

Generally, system <NUM> includes processing unit <NUM>, where, in operation, processing unit <NUM> executes computer-readable instructions (stored on a memory <NUM> of processing unit <NUM>) to fasteners <NUM> in order to detect any fasteners that are unsatisfactorily installed, in which case processing unit <NUM> may automatically indicate the fastener or fasteners that failed inspection. Accordingly, system <NUM> may serve as an automated, real-time fastener installation and inspection system. In a specific example, an onboard processing unit <NUM> such as Raspberry Pi can be used to provide various commands, data collection, and to perform the analysis of fasteners <NUM>. Processing unit <NUM> may be positioned on board first robot <NUM>, coupled to support structure <NUM>, and/or on board second robot <NUM>. In other examples, processing unit <NUM> may be positioned remotely from first robot <NUM> and second robot <NUM>. Processing unit <NUM> may be integrated into first robot <NUM> and/or second robot <NUM> at the time of manufacture. In other examples, first robot <NUM> and/or second robot <NUM> may be outfitted (e.g., retrofit) with processing unit <NUM> after its initial manufacture.

Turning now to <FIG>, illustrative non-exclusive examples of first support structure <NUM> and/or second support structure <NUM>, in the form of a C-beam railing <NUM> are illustrated. Where appropriate, the reference numerals from the schematic illustrations of <FIG>, <FIG>, and <FIG> are used to designate corresponding parts in <FIG> however, the examples of <FIG>, <FIG>, and <FIG> are non-exclusive and do not limit first support structure <NUM> or second support structure <NUM> to the illustrated example of <FIG>. That is, first support structure <NUM> and second support structure <NUM> are not limited to the illustrated C-beam railing <NUM> and may incorporate any number of the various aspects, configurations, characteristics, properties, etc. of first support structure <NUM> or second support structure <NUM> that are illustrated in and discussed with reference to the schematic representations of <FIG>, <FIG>, or <FIG>, and/or the example of <FIG>, as well as variations thereof, without requiring the inclusion of all such aspects, configurations, characteristics, properties, etc. For the purpose of brevity, each previously discussed component, part, portion, aspect, region, etc. or variants thereof may not be discussed, illustrated, and/or labeled again with respect to C-beam railing <NUM>; however, it is within the scope of the present disclosure that the previously discussed features, variants, etc. may be utilized therewith.

<FIG> illustrates an example of first support structure <NUM> and/or second support structure <NUM>, in the form of a C-beam railing <NUM>. C-beam railing <NUM> includes a plurality of holes <NUM> spaced apart along a length <NUM> of C-beam railing <NUM>. For example, C-beam railing <NUM> may include a plurality of holes <NUM> along a first, or upper lip, or flange, <NUM> and along a second, or lower lip, or flange, <NUM>. One or more of holes <NUM> may be threaded, in some examples. Holes <NUM> may be said to include a first plurality of holes <NUM> along upper lip <NUM> and a second plurality of holes <NUM> along lower lip <NUM>, with the respective holes <NUM> being spaced relative to one another on upper lip and lower lip <NUM>, <NUM> respectively, such that set screws may be inserted through one or more holes <NUM> on upper lip <NUM> and one or more holes <NUM> on lower lip <NUM> to limit or restrict movement of x-ray imaging system <NUM>, first camera device <NUM>, and/or second camera device <NUM>, via rings <NUM>. Rings <NUM> may be configured to linearly translate such that they slide longitudinally along length <NUM> of C-beam railing <NUM>, unless a set screw (or bolt, post, pin, etc.) is inserted through holes <NUM> to prevent such sliding movement of rings <NUM>. For example, a first set screw inserted through hole 80a and a second set screw inserted through hole 80b may substantially limit or prevent linear translation of ring 88a along C-beam railing <NUM>.

In some examples, x-ray imaging system <NUM> may be coupled to one of rings <NUM> (e.g., ring 88a), first camera device <NUM> may be coupled to one of rings <NUM> (e.g., ring 88b), and second camera device <NUM> may be coupled to one of rings <NUM> (e.g., ring 88c). In this manner, linear translation of rings <NUM> thereby causes movement of the device coupled to the respective ring <NUM>. In some examples, the angle and position of x-ray imaging system <NUM> may be adjusted with respect to fastener <NUM> via ring 88a. Similarly, the angle and position of first camera device <NUM> may be adjusted with respect to fastener <NUM> via ring 88b, and the angle and position of second camera device <NUM> may be adjusted with respect to fastener <NUM> via ring 88c. When moving first camera device <NUM> and/or second camera device <NUM> (and/or when changing the focus of first camera device <NUM> and/or second camera device <NUM>, systems <NUM> may be configured to allow for correction of lens distortion and for calibration of first camera device <NUM> and/or second camera device <NUM> (e.g., calibration for pixels-to-inches conversion). For example, calibration of first camera device <NUM> and/or second camera device <NUM> may be performed using one or more images of a standard check-board pattern and a calibration module stored on processing unit <NUM>. Similarly, when moving x-ray imaging system <NUM>, x-ray parameters may be set to optimize image quality of resulting x-ray images. For example, image quality indicators may be placed on part <NUM> and/or fastener <NUM> to verify x-ray image quality.

<FIG> schematically provide flowcharts that represent illustrative, non-exclusive examples of methods according to the present disclosure. In <FIG>, some steps are illustrated in dashed boxes indicating that such steps may be optional or may correspond to an optional version of a method according to the present disclosure. That said, not all methods according to the present disclosure are required to include the steps illustrated in solid boxes. The methods and steps illustrated in <FIG> are not limiting and other methods and steps are within the scope of the present disclosure, including methods having greater than or fewer than the number of steps illustrated, as understood from the discussions herein.

<FIG> illustrates methods <NUM> of inspecting a fastener (e.g., fastener <NUM>) installed at least partially through a hole in a part (e.g., hole <NUM> of part <NUM>). Methods <NUM> generally include creating an x-ray image of the fastener via an x-ray imaging system (e.g., x-ray imaging system <NUM>), at <NUM>, and measuring concentricity of the fastener using the x-ray image, at <NUM>. Measuring concentricity at <NUM> is performed by one or more processing units (e.g., processing unit <NUM>). Measuring concentricity at <NUM> may include detecting a first circle corresponding to a first side of the fastener (e.g., a head of the fastener, such as first side <NUM>), detecting a second circle corresponding to a second side of the fastener (e.g., a shaft of the fastener, such as second side <NUM>), determining a first center of the first circle (e.g., first center <NUM> of first circle <NUM>), determining a second center of the second circle (e.g., second center <NUM> of second circle <NUM>), and determining an offset distance between the first center and the second center. Measuring concentricity at <NUM> also may include determining whether the offset distance is within a predetermined pass/fail threshold offset distance and/or indicating whether the offset distance is within the predetermined pass/fail threshold offset distance for each fastener inspected.

Methods <NUM> may include determining whether a particular fastener is pass/fail (e.g., whether it is installed adequately, overall, for the requirements or installation criteria of the fastener), at <NUM>. For example, if the measuring concentricity at <NUM> indicated that the offset distance is not within the predetermined pass-fail threshold offset distance, then that may be sufficient criteria for a "fail" determination at step <NUM>. The determining pass/fail for a given fastener at <NUM> generally includes determining whether the fastener is installed adequately for the given circumstances, and may be based on measuring concentricity at <NUM>, measuring flushness at <NUM>, and/or inspecting for foreign object debris at <NUM>. In some examples, the determining pass/fail for a given fastener at <NUM> includes performing a quality threshold calculation.

The determining pass/fail for a fastener at <NUM> may including indicating which fasteners have been installed adequately and/or which fasteners are inadequately installed. For example, an inadequately installed fastener may be indicated at <NUM> by recording or noting the location of the inadequately installed fastener, by physically marking the inadequately installed fastener, by alerting an operator of the inadequately installed fastener, by recording or noting an identification number or other identifier of the inadequately installed fastener, and/or by any other means of indicating that a particular fastener is not installed adequately. In some methods <NUM>, after determining that a respective fastener is installed unsatisfactorily at <NUM>, the respective fastener may be removed from the part at <NUM>, and the respective fastener may be replaced with a new fastener installed in the part.

Methods <NUM> also may include creating a 3D image, or 3D reconstruction, of the fastener using a first image of the fastener and a second image of the fastener, at <NUM>. The first image of the fastener is taken from a first vantage point (e.g., by first camera device <NUM>), and the second image of the fastener is taken from a second vantage point (e.g., by second camera device <NUM>). The 3D image of the fastener is created at <NUM> by the processing unit using stereovision. The creating the 3D image of the fastener at <NUM> includes capturing two images of the fastener (e.g., via first camera device and second camera device), reconstructing a 3D surface of the fastener, and generating a point cloud of the 3D surface using stereo vision techniques. Additionally, the creating the 3D image of the fastener at <NUM> may include calibrating the first camera device and the second camera device relative to one another and relative to the fastener such that the first camera device and the second camera device are configured to take images configured to create a 3D reconstruction of the fastener installed in the part.

Flushness of the fastener with a surface of the part is measured by the processing unit, using the 3D image of the fastener, at <NUM>. For example, the measuring flushness of a fastener at <NUM> may include identifying a first surface of the fastener, identifying a second surface of the part, fitting a first plane to the first surface, fitting a second plane to the second surface, and determining a height difference between the first plane and the second plane, thereby measuring flushness of the fastener. One or more filters may be applied to a point cloud of the 3D surface of the fastener to identify the first surface and/or the second surface.

Additionally or alternatively, a vicinity of the fastener may be inspected for foreign object debris at <NUM>. For example, the detecting foreign object debris at <NUM> may include identifying objects with a surface area that is dissimilar to that of the head of the fastener, using image processing techniques (e.g., image classification algorithms), and based on the x-ray image of the fastener, the 3D reconstruction of the fastener and part surface, and/or 2D images of the fastener and part surface. In a specific example, the inspecting for foreign object debris at <NUM> may include applying image processing to detect circular features having a diameter matching a known diameter of a head of the fastener, applying high pass filtering to eliminate background, and thereby detecting any foreign object debris adjacent, or in the vicinity of, the fastener.

In some methods <NUM>, the x-ray imaging system is coupled to a first support structure (e.g., first support structure <NUM>), at <NUM>. The coupling the x-ray imaging system to the first support structure at <NUM> also may include coupling the first camera device and/or the second camera device to the first support structure as well. The x-ray imaging system may be moved and/or positioned with respect to the part and/or fastener at <NUM>, along with the first and second camera devices, such as by coupling the first support structure to a robot arm and moving the robot arm to position the x-ray imaging system, the first camera device, and the second camera device relative to the fastener and the part. Some methods <NUM> may include scanning the part at <NUM> before the moving and/or positioning the x-ray imaging system at <NUM>, with the scan of the part informing the movement of the robot arm and thereby the movement of the x-ray imaging system. The scanning the part at <NUM> may include determining contours of the part and planning movement of the first robot arm to position the x-ray imaging system relative to a respective fastener of a plurality of fasteners being inspected. In some methods <NUM>, the scanning the part at <NUM> may be performed in tandem with the measuring concentricity at <NUM>, the measuring flushness at <NUM>, and/or the inspecting for foreign object debris at <NUM>. For example, the processing unit of disclosed systems may be determining concentricity at the current location, while another region of the part is being scanned in preparation for measuring concentricity at the next location of the subsequent fastener to be measured.

Methods <NUM> may include positioning the x-ray imaging system on a first side of the part via the moving the x-ray imaging system at <NUM>, and also coupling an x-ray detector (e.g., x-ray detector <NUM>) to a second support structure at <NUM> and positioning the x-ray detector on the opposite side of the part from the x-ray imaging system. Coupling and positioning the x-ray detector at <NUM> may include selectively adjusting a stand-off distance between the x-ray detector and the part, based on, for example, the desired geometric magnification of the x-ray image. The second support structure may be coupled to a second robot arm, such as one coupled to a second robot, which may thereby control movement of the x-ray detector relative to the part and the fastener (and relative to the first robot arm and the x-ray imaging system). The second robot may be a follower robot to a commander robot controlling movement of the first support structure. In this manner, the two robot arms may be configured to move in a coordinated manner such that the x-ray imaging system and x-ray detector are positioned relative to each other and relative to the fastener to create an x-ray image of the fastener as described herein.

In some methods <NUM>, the flushness, concentricity, and/or presence of foreign object debris is measured or detected for a plurality of fasteners in a given part. In other words, the measuring concentricity at <NUM>, the measuring flushness at <NUM>, and/or the inspecting a fastener vicinity for foreign object debris at <NUM> may be performed a plurality of times, such as being performed for each fastener being inspected. In these examples, methods <NUM> may include the moving and/or positioning the x-ray imaging system at <NUM> between each performance of the measuring concentricity at <NUM>, the measuring flushness at <NUM>, and/or the inspecting a fastener vicinity for foreign object debris at <NUM>. For example, concentricity of a first fastener may be measured at <NUM>, flushness of the first fastener may be measured at <NUM>, and/or the first fastener may be inspected for foreign object debris in its vicinity at <NUM>, and then the x-ray imaging system may be moved and/or positioned at <NUM> before measuring concentricity of a second fastener may be measured at <NUM>, flushness of the second fastener may be measured at <NUM>, and/or the second fastener may be inspected for foreign object debris in its vicinity at <NUM>.

Fasteners may be inspected on a zone basis, in some methods <NUM>. For example, a part having a plurality of fasteners may be divided into two or more different zones, with each zone having a plurality of fasteners therein. When inspecting fasteners in a part, disclosed systems may inspect a predetermined number of fasteners (one or more) within a given zone, rather than every fastener in the zone. If the inspected fastener or fasteners in a zone are adequately installed, the system may move on to a different zone and inspect one or more fasteners in that zone, at <NUM>. If, on the other hand, one or more fasteners in a given zone are found to be inadequately installed, then one or more other fasteners in the same zone may be inspected, beyond the original number of fasteners inspected, at <NUM>. In this manner, disclosed systems may inspect a sample of fasteners in a part, and increase the sampling in areas, or zones, where one or more fasteners is determined to be inadequately installed. Thus, the measuring concentricity at <NUM> may include measuring concentricity of a first fastener in a first zone of the part and measuring concentricity of a second fastener within a second zone of the part. The measuring concentricity at <NUM> may include measuring concentricity of at least one fastener in each of a plurality of zones of the part.

Non-transitory computer readable instructions for performing the measuring concentricity at <NUM>, the moving and/or positioning the x-ray system at <NUM>, the scanning the part at <NUM>, the creating the 3D image of the fastener at <NUM>, the measuring flushness at <NUM>, the inspecting for foreign object debris at <NUM>, and/or the determining a pass/fail status of the fastener at <NUM> may be stored on a computer readable medium and/or on the memory of the processing unit of disclosed systems, and that may be executed by the processing unit of systems. <FIG> schematically illustrates an example algorithm <NUM> that may be executed by processing unit <NUM> and stored on memory <NUM>, in order to inspect one or more fasteners <NUM> installed in part <NUM>. Briefly, the system may be positioned with respect to the fastener being inspected, as indicated by move to position at <NUM>. An x-ray system control module of the processing unit may be activated at <NUM> to trigger x-ray image collection of the faster. An x-ray image processing module of the processing unit may be activated at <NUM> to perform a concentricity calculation at <NUM>. Whether in parallel or in series, a visual camera control module of the processing unit may be activated at <NUM> to trigger optical image collection by the first and second camera devices. A 2D image processing module of the processing unit may be activated at <NUM> to perform foreign object debris detection at <NUM>, and a 3D reconstruction module of the processing unit may be activated at <NUM> to perform flushness estimation at <NUM>. Quality metric assessment may be performed by the processing unit at <NUM> to determine whether a given fastener being inspected has passed all metrics that were measured, at <NUM>. If all the measured metrics have a "pass" result, then the inspected fastener is deemed adequately installed, whereas if one or more of the measured metrics have a "fail" result, then the inspected fastener is deemed inadequately installed, and flagged for removal or repair.

As used herein, the terms "selective" and "selectively," when modifying an action, movement, configuration, or other activity of one or more components or characteristics of an apparatus, mean that the specific action, movement, configuration, or other activity is a direct or indirect result of dynamic processes and/or user manipulation of an aspect of, or one or more components of, the apparatus. The terms "selective" and "selectively" thus may characterize an activity that is a direct or indirect result of user manipulation of an aspect of, or one or more components of, the apparatus, or may characterize a process that occurs automatically, such as via the mechanisms disclosed herein.

As used herein, the phrase "at least one," in reference to a list of one or more entities should be understood to mean at least one entity selected from any one or more of the entities in the list of entities, but not necessarily including at least one of each and every entity specifically listed within the list of entities and not excluding any combinations of entities in the list of entities. This definition also allows that entities may optionally be present other than the entities specifically identified within the list of entities to which the phrase "at least one" refers, whether related or unrelated to those entities specifically identified. Thus, as a non-limiting example, "at least one of A and B" (or, equivalently, "at least one of A or B," or, equivalently "at least one of A and/or B") may refer, in one embodiment, to at least one, optionally including more than one, A, with no B present (and optionally including entities other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including entities other than A); in yet another embodiment, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other entities). In other words, the phrases "at least one," "one or more," and "and/or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C" and "A, B, and/or C" may mean A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, and optionally any of the above in combination with at least one other entity.

A processing unit may be any suitable device or devices that are configured to perform the functions of the processing unit discussed herein. For example, the processing unit may include one or more of an electronic controller, a dedicated controller, a special-purpose controller, a personal computer, a special-purpose computer, a display device, a logic device, a memory device, and/or a memory device having computer readable media suitable for storing computer-executable instructions for implementing aspects of systems and/or methods according to the present disclosure. Additionally or alternatively, the processing unit may include, or be configured to read, non-transitory computer readable storage, or memory, media suitable for storing computer-executable instructions, or software, for implementing methods or steps of methods according to the present disclosure. Examples of such media include CD-ROMs, disks, hard drives, flash memory, etc. As used herein, storage, or memory, devices and media having computer-executable instructions as well as computer-implemented methods and other methods according to the present disclosure are considered to be within the scope of subject matter deemed patentable in accordance with Section <NUM> of Title <NUM> of the United States Code.

As used herein, the phrase "at least substantially," when modifying a degree or relationship, includes not only the recited "substantial" degree or relationship, but also the full extent of the recited degree or relationship. A substantial amount of a recited degree or relationship may include at least <NUM>% of the recited degree or relationship. For example, a first direction that is at least substantially parallel to a second direction includes a first direction that is within an angular deviation of <NUM>° relative to the second direction and also includes a first direction that is identical to the second direction.

The various disclosed elements of apparatuses and steps of methods disclosed herein are not required to all apparatuses and methods according to the present disclosure, and the present disclosure includes all novel and non-obvious combinations and subcombinations of the various elements and steps disclosed herein. Moreover, one or more of the various elements and steps disclosed herein may define independent inventive subject matter that is separate and apart from the whole of a disclosed apparatus or method. Accordingly, such inventive subject matter is not required to be associated with the specific apparatuses and methods that are expressly disclosed herein, and such inventive subject matter may find utility in apparatuses and/or methods that are not expressly disclosed herein.

Claim 1:
A system (<NUM>) for inspecting a fastener (<NUM>) installed at least partially through a hole (<NUM>) in a part (<NUM>), the system (<NUM>) comprising:
an x-ray imaging system (<NUM>) oriented and configured to produce an x-ray image (<NUM>) of the fastener (<NUM>);
a first camera device (<NUM>) positioned and oriented such that it is configured to produce a first image of the fastener (<NUM>) from a first vantage point;
a second camera device (<NUM>) positioned and oriented such that it is configured to produce a second image of the fastener (<NUM>) from a second vantage point;
a first support structure (<NUM>) to which the x-ray imaging system (<NUM>), the first camera device (<NUM>), and the second camera device (<NUM>) are coupled, wherein the first support structure (<NUM>) is configured to support and position the first camera device (<NUM>) and the second camera device (<NUM>) relative to the part (<NUM>) and the fastener (<NUM>) such that a 3D image of the fastener (<NUM>) can be created from the first image and the second image; and
at least one processing unit (<NUM>) configured to create the 3D image of the fastener (<NUM>) from the first image and the second image, wherein the at least one processing unit (<NUM>) is further configured to inspect the fastener (<NUM>) based on the x-ray image (<NUM>) and the 3D image, and wherein the system (<NUM>) is configured to measure concentricity of the fastener (<NUM>) using the x-ray image (<NUM>), and to measure flushness of the fastener (<NUM>) with respect to the part (<NUM>) using the 3D image.