Automated rivet measurement system

An automated rivet measurement system comprises a number of end effectors, a number of cameras, a processor, and a comparator. The number of end effectors is configured to perform drilling and riveting on a structure. The number of cameras is connected to the number of end effectors. The number of cameras is configured to take a first image of a hole in the structure and a second image of a rivet in the hole. The processor is configured to process the first image and the second image to identify a number of reference points in the first image and the second image. The comparator is configured to determine a rivet concentricity using the hole in the first image and the rivet in the second image, in which the first image and the second image are aligned using the number of reference points.

BACKGROUND INFORMATION

The present disclosure relates generally to manufacturing vehicles, and in particular, to installing fasteners in vehicles using a rivet measurement system. Still more particularly, the present disclosure relates to an automated rivet measurement system configured to install rivets and determine rivet concentricity.

In manufacturing vehicles, fasteners are installed to connect parts to each other. For example, hundreds of thousands or more rivets may be installed in a commercial aircraft. Inspection of the installed rivets is performed to determine whether the rivets meet specifications for the commercial aircraft.

The inspection may be performed in a number of different ways. The inspections may be performed using non-destructive testing and destructive testing.

Non-destructive testing may be performed by having human operators make measurements with tools, such as gauges or probes. Additionally, non-destructive testing may be performed using robots that have an end effector configured to make measurements. Measurements of rivets also may be obtained using cameras or laser measurement tools.

Some measurements, however, may not be easily made using human operators or robots with gauges, probes, cameras, or laser measurement tools. For example, some parameters cannot be viewed on installed rivets. One parameter of interest is rivet concentricity. This parameter is measured by drilling out the rivet and making measurements of the uninstalled rivet. Drilling out the rivet destroys and removes the rivet. The drilling and measurements are a form of destructive testing.

As a result, destructive testing may be employed to obtain measurements for rivet concentricity and other parameters. With destructive testing, the disassembly and removal of rivets to make measurements is often more time consuming and more costly than desired. Another drawback with destructive testing, however, is that an additional step occurs in which another rivet is installed in the inspected location in the structure in the aircraft. This situation also increases the cost for manufacturing the commercial aircraft.

Therefore, it would be desirable to have a method and apparatus that take into account at least some of the issues discussed above, as well as other possible issues. For example, it would be desirable to have a method and apparatus that overcome a technical problem with obtaining measurements of desired parameters without using destructive testing.

SUMMARY

An illustrative embodiment of the present disclosure provides a method for automated non-destructive testing of rivet concentricity. A rivet is installed within a hole in a structure using an end effector. A first image of the hole in the structure and a second image of the rivet in the structure are processed to identify a number of reference points located in both the first image and the second image. The first image and the second image are aligned using the number of reference points. The first image is analyzed to determine a number of locations of the hole. The second image is analyzed to determine a number of locations of the rivet. A rivet concentricity is automatically determined using the number of locations of the hole and the number of locations of the rivet.

Another illustrative embodiment of the present disclosure provides an automated rivet measurement system. The automated rivet measurement system comprises a number of end effectors, a number of cameras, a processor, and a comparator. The number of end effectors is configured to perform drilling and riveting on a structure. The number of cameras is connected to the number of end effectors. The number of cameras is configured to take a first image of a hole in the structure and a second image of a rivet in the hole. The processor is configured to process the first image and the second image to identify a number of reference points in the first image and the second image. The comparator is configured to determine a rivet concentricity using the hole in the first image and the rivet in the second image, in which the first image and the second image are aligned using the number of reference points.

A further illustrative embodiment of the present disclosure provides a method for automated non-destructive testing of rivet concentricity. A hole is drilled in a structure using an end effector configured to perform drilling. A first image is taken of the hole in the structure. A rivet is installed within the hole using an end effector configured to perform riveting. A second image is taken of the rivet in the structure. The first image and the second image are processed to identify a number of reference points located in both the first image and the second image. The first image and the second image are aligned within a coordinate system relative to the number of reference points. The first image is analyzed to determine a number of locations of the hole. The second image is analyzed to determine a number of locations of the rivet. A rivet concentricity is automatically determined using the number of locations of the hole and the number of locations of the rivet.

DETAILED DESCRIPTION

The illustrative embodiments recognize and take into account one or more different considerations. For example, the illustrative embodiments recognize and take into account that it would be desirable to perform measurements without destructive testing. The illustrative embodiments recognize and take into account one type of non-destructive testing that may be used to identify a rivet parameter, such as concentricity, is the use of an x-ray system.

The illustrative embodiments recognize and take into account, however, that an x-ray system is a less desirable non-destructive testing system. An x-ray system may require removal or disassembly of parts to perform x-ray measurements. After an image is generated, the parts are reinstalled or reassembled. In other cases, the removal or disassembly of parts may not be needed but the cost of the equipment, specialized training for operators, and time-consuming steps for x-ray measurements make this type of technique undesirable.

The illustrative embodiments recognize and take into account that reducing human exposure to x-rays is desirable. During operation of an x-ray system, human operators in range of the x-ray system within the manufacturing environment will be shielded from the x-rays. The illustrative embodiments recognize that x-ray inspections may be scheduled during times without scheduled manufacturing activities to minimize human operators in the manufacturing environment. The illustrative embodiments recognize that human operators within range of the x-ray system will be evacuated or shielded prior to operating the x-ray system. Evacuating or shielding human operators will result in work stoppage during operation of the x-ray system.

Thus, the illustrative embodiments recognize and take into account that an x-ray system is a less desirable non-destructive testing system because of the increase in time and effort needed to make measurements. The illustrative embodiments recognize that this type of system may increase the time and effort needed by great amounts when millions of rivets may be present.

The illustrative examples recognize and take into account that a rivet installed in a structure has a head and a button. The illustrative examples recognize and take into account that a head of a rivet is on an outer surface of the structure. The illustrative examples recognize and take into account that the outer surface of the structure is the surface through which the rivet is inserted for installation. The illustrative examples recognize and take into account that the inner surface of the structure is opposite to outer surface. The illustrative examples recognize and take into account that the inner surface is the surface on which the button forms when the rivet is installed.

The illustrative examples recognize and take into account that to install the rivet, the shaft of the rivet is inserted into a hole of the structure so that the head of the rivet contacts the outer surface of the structure. The illustrative examples recognize and take into account that the button is formed by contacting the shaft of the rivet and pressing downwards to deform the end of the rivet shaft into a button on the inner surface. The illustrative examples recognize and take into account that due to process variation, the button may be irregularly shaped. The illustrative examples recognize and take into account that due to process variation, the button may not be symmetric relative to the rivet shaft. The illustrative examples recognize and take into account that due to process variation, the button may not cover the hole.

The illustrative examples recognize and take into account that rivet concentricity describes how symmetrical the button is in the rivet. The illustrative examples recognize and take into account that rivet concentricity may be determined relative to the hole or relative to the rivet shaft.

The illustrative examples recognize and take into account that environmental factors influence the installation of rivets into a structure. The illustrative examples recognize and take into account that environmental factors may influence the concentricity of a resulting rivet.

The illustrative examples recognize and take into account that environmental factors may include a particular fastening machine, a particular team of human operators, or other suitable factors. The illustrative examples recognize and take into account that environmental factors also may include machine process parameters, for example, an installation process, hammering time, force exertion settings, drill speed, and other parameters. The illustrative examples recognize and take into account that these parameters may impact installations of rivets. The illustrative examples recognize and take into account that examples of environmental factors may include process parameters such as material type, coating, and drill bit wear. The illustrative examples recognize and take into account that additional environmental factors include temperature, humidity, machine location, and other types of environmental factors.

The illustrative embodiments recognize and take into account that performing non-destructive testing for rivet concentricity may take less time than performing measurements using destructive testing. The illustrative examples recognize and take into account that it may be desirable to perform inspections in situ. The illustrative embodiments recognize and take into account that performing non-destructive testing for rivet concentricity in situ may take less time than performing non-destructive testing as a separate step. The illustrative embodiments recognize and take into account that performing non-destructive testing for rivet concentricity in situ may include at least one of taking images, processing the images, or analyzing the images during other manufacturing operations. The illustrative embodiments recognize and take into account that performing non-destructive testing for rivet concentricity in situ may including performing non-destructive testing during at least one of drilling operations, riveting operations, or during movement between drilling or riveting operations.

The illustrative embodiments recognize and take into account that variability exists within manufacturing operations. The illustrative embodiments recognize and take into account that a tolerance exists for movement of end effectors within a manufacturing environment. The illustrative embodiments recognize and take into account that an end effector instructed to drive to the same location multiple times will not drive to precisely the same location each time due to movement tolerances. The illustrative embodiments recognize and take into account that images purportedly taken from the same location may have differences in position based on movement tolerances.

Referring now to the figures, and in particular, with reference toFIG. 1, an illustration of an aircraft is depicted in which an illustrative embodiment may be implemented. In this illustrative example, aircraft100has wing102and wing104connected to body106. Aircraft100includes engine108connected to wing102and engine110connected to wing104.

Aircraft100is an environment in which rivets installed and measured by an automated rivet measurement system may be found. For example, rivets installed and measured by an automated rivet measurement system are present in at least one of wing102, wing104, or body106.

This illustration of aircraft100is provided for the purposes of illustrating one environment in which different illustrative embodiments may be implemented. The illustration of aircraft100inFIG. 1is not meant to imply architectural limitations as to the manner in which different illustrative embodiments may be implemented. For example, aircraft100is shown as a commercial passenger aircraft. The different illustrative embodiments may be applied to other types of aircraft, such as a private passenger aircraft, rotorcraft, or other suitable types of aircraft.

Although the illustrative examples for an illustrative embodiment are described with respect to an aircraft, the illustrative embodiments may be applied to other types of structures. The structure may be, for example, a mobile structure, a stationary structure, a land-based structure, an aquatic-based structure, or a space-based structure. More specifically, the structure may be a surface ship, a tank, a personnel carrier, a train, a spacecraft, a space station, a satellite, a submarine, a manufacturing facility, a building, or other suitable types of structures.

Turning now toFIG. 2, an illustration of a block diagram of a manufacturing environment with an automated rivet measurement system is depicted in accordance with an illustrative embodiment. Components of aircraft100may be manufactured in manufacturing environment200. For example, at least one of wing102or wing104may have rivets installed and measured in manufacturing environment200. In manufacturing environment200, automated rivet measurement system202is used to install and measure rivets203in structure204.

Structure204may take different forms. For example, a structure may be selected from at least one of a skin panel, a conduit, a monument, an engine, an engine housing, a fuselage section, a wing box, a spar, a rib, a line replaceable unit (LRU), an electrical assembly, and other types of structures that may be used in a vehicle. In this illustrative example, structure204may be formed from one or more parts or components that are connected to each other using rivets203. In some illustrative examples, structure204may only be formed from a single part that includes rivets203. In some illustrative examples, structure204is at least one part or component for use in aircraft100.

Automated rivet measurement system202comprises number of end effectors206, number of cameras207, processor208, and comparator209. Number of end effectors206is configured to perform drilling210and riveting211on structure204. As used herein, a “number of items” is one or more items. For example, number of end effectors206is one or more end effectors.

In some illustrative examples, number of end effectors206comprises one end effector to perform both drilling210and riveting211. In these illustrative examples, number of end effectors206is end effector212. In these illustrative examples, number of end effectors206is end effector212configured to perform both drilling210and riveting211.

In other illustrative examples, number of end effectors206comprises more than one end effector to perform drilling210and riveting211. In some illustrative examples, number of end effectors206is end effector212configured to perform drilling210and end effector213configured to perform riveting211.

Number of cameras207is connected to number of end effectors206. Number of cameras207is configured to take first image214of hole216in structure204and a second image218of rivet219in hole216. First image214has hole216a, a visual depiction of hole216in structure204. Second image218has rivet219b, a visual depiction of rivet219in structure204.

In some illustrative examples, number of cameras207comprises camera220connected to end effector212. In these illustrative examples, camera220takes both first image214and second image218.

In some illustrative examples, number of cameras207comprises camera220connected to end effector213. In these illustrative examples, camera220takes both first image214and second image218.

In other illustrative examples, number of cameras207comprises camera220and camera222. In these illustrative examples, camera220connected to end effector212takes first image214. In these illustrative examples, camera222connected to end effector213takes second image218.

In some illustrative examples, number of end effectors206comprises a first end effector, end effector212, to perform drilling and a second end effector, end effector213, to perform the riveting. In some of these illustrative examples, number of cameras207comprises camera220connected to the first end effector configured to take first image214and a second camera, camera222, connected to the second end effector configured to take second image218. In other of these illustrative examples, number of cameras207comprises camera220connected to the second end effector, end effector213.

Processor208is configured to process first image214and second image218to identify number of reference points224in first image214and second image218. Number of reference points224may take the form of any type of feature of structure204. For example, number of reference points224may include at least one of a rivet, a hole, or an edge of structure204.

Comparator209is configured to analyze first image214to determine number of locations226of hole216a, analyze second image218to determine number of locations228of rivet219b, and determine rivet concentricity230using number of locations226of hole216aand number of locations228of rivet219b. In some illustrative examples, number of locations226of hole216aare a detected edge of hole216a. In some illustrative examples, number of locations226may not be identical to an edge of hole216. In some illustrative examples, number of locations228of rivet219bare a detected edge of rivet219b. In some illustrative examples, number of locations228may not be identical to an edge of the button of rivet219.

Comparator209is a device configured to determine rivet concentricity230. Comparator209is also configured to compare determined rivet concentricity230to specification limits280.

As depicted, comparator209may be located in computer system232. Computer system232is a physical hardware system and includes one or more data processing systems. When more than one data processing system is present, those data processing systems are in communication with each other using a communications medium. The communications medium may be a network. The data processing systems may be selected from at least one of a computer, a server computer, a tablet, or some other suitable data processing system.

End effector212drills hole216into location234of structure204. In some illustrative examples, camera220is attached to end effector212and takes first image214when end effector212is at first position236. In some illustrative examples, first position236is a position of end effector212to drill hole216into location234. In some illustrative examples, first position236is a position of end effector212as end effector212moves away from location234after drilling210hole216.

In some other illustrative examples, camera220is attached to end effector213and takes first image214when end effector213is at first position236. In these illustrative examples, camera220takes first image214prior to end effector213performing riveting211to complete rivet219.

One of end effector212or end effector213performs riveting211to complete rivet219. Completing rivet219includes forming a button (not depicted) of rivet219. The button (not depicted) of rivet219is the visible portion of rivet219that is imaged by number of cameras207. Rivet concentricity230is determined using visual depictions of the button (not depicted) of rivet219.

Number of cameras207takes second image218after rivet219is completed. In some illustrative examples, camera220attached to end effector212takes second image218. In these illustrative examples, camera220takes second image218when end effector212is at second position238.

In some illustrative examples, second position238is a position of end effector212to complete rivet219in hole216. In some illustrative examples, second position238is a position of end effector212as end effector212moves away from rivet219after completing rivet219in hole216.

In some illustrative examples, camera220is attached to end effector213and takes second image218. In these illustrative examples, camera220takes second image218when end effector213is at second position238.

In some illustrative examples, camera222is attached to end effector213and takes second image218. In these illustrative examples, camera222takes second image218when end effector213is at second position238.

In some illustrative examples, second position238is a position of end effector213to complete rivet219in hole216. In some illustrative examples, second position238is a position of end effector212as end effector213moves away from rivet219after completing rivet219in hole216.

In some illustrative examples, first position236and second position238are substantially the same. In some illustrative examples, first position236is offset from second position238.

As depicted, structure204has first feature240and second feature242. When first feature240and second feature242are present in two images, first feature240and second feature242may be used as number of reference points224for those two images. As depicted, first feature240and second feature242are present in two images, first image214and second image218.

First feature240ais a visual depiction of first feature240within first image214. First feature240bis a visual depiction of first feature240within second image218.

Second feature242ais a visual depiction of second feature242within first image214. Second feature242ais a visual depiction of second feature242within second image218.

In this illustrative example, first feature240takes the form of rivet243of rivets203. Rivet243ais a visual depiction of rivet243within first image214. Rivet243bis a visual depiction of rivet243within second image218.

In this illustrative example, second feature242takes the form of rivet244of rivets203. Rivet244ais a visual depiction of rivet244within first image214. Rivet244bis a visual depiction of rivet244within second image218.

In other non-depicted examples, first feature240may be a hole (not depicted) in structure204. In other non-depicted examples, first feature240may be an edge (not depicted) of structure204. In other non-depicted examples, second feature242may be a hole (not depicted) in structure204. In other non-depicted examples, second feature242may be an edge (not depicted) of structure204.

Processor208aligns first image214and second image218using number of reference points224. In some illustrative examples, processor208positions first image214relative to coordinate system246. In some illustrative examples, processor208positions second image218relative to coordinate system246.

In some illustrative examples, number of locations226of hole216aare identified within coordinate system246. In some illustrative examples, number of locations228of rivet219bare identified within coordinate system246.

In some illustrative examples, first image214and second image218are aligned using number of reference points224prior to overlaying second image218and first image214. In some illustrative examples, first image214and second image218are aligned using number of reference points224prior to overlaying number of locations226of hole216aonto second image218.

Comparator209performs any desirable procedures to determine rivet concentricity230. In some illustrative examples, comparator209is configured to create outline248of hole216aand create outline250of rivet219b. Outline248is representative of number of locations226of hole216a. Outline250is representative of number of locations228of rivet219b. In some illustrative examples, determining rivet concentricity230comprises determining distances252between points253of outline248of hole216aand points254of outline250of rivet219b. In some illustrative examples, rivet concentricity230is a smallest value of distances252.

In some illustrative examples, distances252include distances between each of points253and each of points254. In other illustrative examples, distances252include distances between each of points254and a subset of points253for each point of points254. In other illustrative examples, distances252include distances perpendicular to tangents of outline248.

Number of cameras207takes number of images264as number of end effectors206perform drilling210and riveting211on structure204. Number of images264are sent to computer system232for processing by processor208and analysis by comparator209. Number of images264may include any desirable quantity of images. Number of images264may be stored as images266within database268.

Automated rivet measurement system202may use metadata270of images266to determine which of images266to process to identify number of reference points224. For example, each of images266may have two completed rivets of rivets203in similar locations that may be used as number of reference points224. However, each rivet of rivets203will have a uniquely shaped and positioned button (not depicted). To reduce processing resources, subsets of images266may be eliminated as unrelated.

The two images of images266including a hole and a rivet later installed with the hole may be identified using metadata270. For example, first image214and second image218to be processed and analyzed may be identified using metadata270.

In some illustrative examples, first image214is identified as having hole216ausing at least one of time272first image214was taken, location274within manufacturing environment200, function276, or camera identification278. In some illustrative examples, location274is a location of the camera of number of cameras207that took first image214. In some illustrative examples, location274is a location of the end effector of number of end effectors206the respective camera of number of cameras207that took first image214is attached to. Location274may be identified using any desirable method. In some illustrative examples, location274is identified using positional sensors within manufacturing environment200.

Function276is a manufacturing function, such as drilling210or riveting211, that is performed either prior to or during taking of first image214. Camera identification278is a unique identifier for the camera of number of cameras207that took first image214.

Processor208identifies first image214using metadata270. Processor208identifies second image218using metadata270. For example, time272of second image218will be later than time272of first image214. In some illustrative examples, location274of first image214will be substantially the same as location274of second image218. In some illustrative examples, location274of first image214and location274of second image218are such that first feature240, second feature242, and location234are visible in first image214and second image218.

In some illustrative examples, output282is an alert or an alarm. Output282may trigger an audio or visual alarm when rivet concentricity230does not satisfy specification limits280. In some illustrative examples, output282is an entry in a report. In some illustrative examples, output282is part of a map of locations of structure204that do not satisfy specification limits280.

In some illustrative examples, when output282indicates that rivet concentricity230of rivet219does not satisfy specification limits280, output282will be reviewed by an operator. In some illustrative examples, when output282indicates that rivet concentricity230of rivet219does not satisfy specification limits280, rivet219will be reworked.

In one illustrative example, one or more technical solutions are present that overcome a technical problem with obtaining measurements of desired parameters without using destructive testing. As a result, one or more technical solutions may provide a technical effect to identifying destructive testing measurements without having to perform destructive testing. One or more technical solutions are present that provide an ability to identify a rivet concentricity without drilling out the rivet.

As a result, computer system232operates as a special purpose computer system in which comparator209in computer system232enables determining a rivet concentricity from images in situ. In particular, comparator209transforms computer system232into a special purpose computer system, as compared to currently available general computer systems that do not have comparator209.

In some illustrative examples, structure204may include other non-depicted features. For example, rivets203in structure204may include more than three rivets. In some illustrative examples, number of reference points224includes more than two reference points. For example, number of reference points224may include three reference points.

Turning now toFIG. 3, an illustration of a side view of a manufacturing environment with an automated rivet measurement system is depicted in accordance with an illustrative embodiment. Manufacturing environment300is one physical implementation of manufacturing environment200ofFIG. 2. End effector302in manufacturing environment300is a physical implementation of end effector213ofFIG. 2.

In this illustrative example, end effector302is configured to performing riveting operations. In this illustrative example, end effector302is positioned to install rivets in holes (not depicted) in structure304in manufacturing environment300. In this illustrative example, a different end effector (not depicted) is used to drill holes in structure304.

End effector302and camera306are components of automated rivet measurement system308. In this illustrative example, camera306is a physical implementation of a camera of number of cameras207ofFIG. 2. In some illustrative examples, camera306is an implementation of camera220that takes images of holes and rivets, such as first image214and second image218. In some illustrative examples, camera306is an implementation of camera222that takes image of installed rivets, such as second image218. Automated rivet measurement system308automatically determines rivet concentricity in situ using at least one image from camera306.

Camera306takes images of structure304during operation of end effector302. In one illustrative example, camera306takes images before and after a riveting operation at a hole. In this illustrative example, camera306takes both a first image, such as first image700ofFIG. 7, and a second image, such as second image800ofFIG. 8. In this illustrative example, camera306takes the first image when end effector302is at a first position and the second image when end effector302is at a second position. In some illustrative examples, the first position and the second position are substantially the same. In other illustrative examples, the first position and the second position are offset from each other.

In another illustrative example, camera306takes an image after installing a rivet in a hole. In this illustrative example, camera306may take only a second image, such as second image800ofFIG. 8. In this illustrative example, camera306takes the second image when end effector302is at a second position. In this illustrative example, a different camera attached to another end effector (not depicted) configured to drill holes in structure304takes a first image when the other end effector is at a first position. In some illustrative examples, the first position and the second position are substantially the same. In other illustrative examples, the first position and the second position are offset from each other.

When first position and second position are substantially the same, first image and second image may still be aligned due to end effector movement tolerances. When first position and second position are offset, first image and second image are registered or aligned due to the offset between the first position and the second position.

The illustration of automated rivet measurement system308inFIG. 3is not meant to imply architectural limitations as to the manner in which different illustrative embodiments may be implemented. For example, in other non-depicted examples, an end effector (not depicted) both drills a hole into structure304and installs a rivet in the hole. In these non-depicted examples, a camera attached to the end effector (not depicted) takes images of both the hole and the rivet. In these illustrative examples, the camera takes a first image when the end effector is in a first position relative to structure304. After drilling the hole in structure304, the camera will take a second image when the end effector is at a second position. In some illustrative examples, the first position and the second position are substantially the same. In some illustrative examples, the first position is a drilling position and the second position is a riveting position. In some illustrative examples, the first position is a position of the end effector after drilling. In some illustrative examples, the second position is a position of the end effector after riveting.

Turning now toFIG. 4, an illustration of an end effector with a connected camera is depicted in accordance with an illustrative embodiment. View400is a view of a side view of end effector302ofFIG. 3.

Camera306is connected to end effector302and moves with end effector302. As end effector302moves relative to a structure, such as structure304ofFIG. 3, camera306is also moved relative to the structure.

End effector302has clamp foot402and bucking tool404. During riveting, bucking tool404contacts a rivet to form a button.

Turning now toFIG. 5, an illustration of a top view of a structure with installed rivets is depicted in accordance with an illustrative embodiment. View500is a top view of structure304ofFIG. 3. Structure304is a physical implementation of structure204ofFIG. 2.

In this illustrative example, structure304is formed of three components riveted together. As depicted, structure304is formed of first plate502, second plate504, and beam506. Each of first plate502, second plate504, and beam506are head together using rivets508. Rivets508are installed and measured using automated rivet measurement system308ofFIG. 3.

Turning now toFIG. 6, an illustration of a perspective view of a structure with installed rivets is depicted in accordance with an illustrative embodiment. View600is a perspective view of structure304ofFIG. 3.

In view600, rivets602shielded by beam506in view500are seen. To measure rivets602, images are taken by camera306ofFIG. 3at an angle relative to structure304. Automated rivet measurement system308provides compensation for the angle during image processing and image analysis. Rivets604of rivets508may be measured using images of a top view of structure304.

Turning now toFIG. 7, an illustration of a first image is depicted in accordance with an illustrative embodiment. First image700is an implementation of first image214ofFIG. 2. First image700is an image of a structure, such as structure304ofFIGS. 3, 5, and 6.

First image700has rivet702, rivet704, hole706, and hole708. Any of rivet702, rivet704, hole706, or hole708may be a reference point for first image700.

Rivet702is a visual depiction within first image700of a physical rivet. Rivet704is a visual depiction within first image700of a physical rivet. Hole706is a visual depiction within first image700of a physical hole. Hole708is a visual depiction within first image700of a physical hole.

When first image700is taken, metadata is saved along with first image700. The metadata includes data to uniquely identify first image700. The metadata includes at least one of camera identification, a location of the camera in a coordinate system, a time, a date, a function performed by the end effector prior to taking first image700, or any other desirable data.

Turning now toFIG. 8, an illustration of a second image is depicted in accordance with an illustrative embodiment. Second image800is an image of a structure, such as structure304ofFIGS. 3, 5, and 6. Second image800is an implementation of second image218ofFIG. 2. In some illustrative examples, second image800is an image of the same structure as inFIG. 7. In some illustrative examples, second image800is an image of the same location as imaged inFIG. 7. Second image800has rivet802, rivet804, hole806, and rivet808.

Rivet802is a visual depiction within second image800of a physical rivet. Rivet804is a visual depiction within second image800of a physical rivet. Hole806is a visual depiction within second image800of a physical hole. Rivet808is a visual depiction within second image800of a physical rivet.

When second image800is taken, metadata is saved along with second image800. The metadata includes data to uniquely identify second image800. The metadata includes at least one of camera identification, a location of the camera in a coordinate system, a time, a date, a function performed by the end effector prior to taking second image800, or any other desirable data.

Metadata for second image800is compared to metadata for other images to determine any potentially related images. When compared to metadata for first image700, metadata for second image800may indicate that first image700and second image800include at least some of the same features.

For illustrative purposes, first image700and second image800are visual depictions of the same location of a structure before and after installing a rivet.FIGS. 9-11depict image analysis using illustrative markings over first image700and second image800. Although first image700and second image800are depicted and discussed as images of approximately the same location of a structure, in some illustrative examples, first image700and second image800may be images of different locations of the same or different structures.

Turning now toFIG. 9, an illustration of a first image after processing is depicted in accordance with an illustrative embodiment. View900is a view of first image700with illustrative markings overlaid.

First image700and second image800are views of the same location of structure901at two different times. First image700is taken after drilling hole708. Second image800is taken after inserting rivet808into hole708.

In some illustrative examples, only metadata may be used to identify first image700and second image800as containing the same features. In other illustrative examples, metadata may be used in conjunction with visual data of first image700and second image800to identify first image700and second image800as containing the same features. For example, after identifying first image700and second image800as possibly containing the same features, visual data of first image700and second image800may be compared to identify inconsistencies or other unique characteristics present in both first image700and second image800.

For example, visual data of first image700and second image800may be used to confirm that rivet702is the same as rivet802. For example, a divot is present in each of rivet702and rivet802.

First image700is processed to determine number of reference points902. Number of reference points902include features present in both first image700and second image800. Rivet702is the same as rivet802in second image800. Rivet704is the same as rivet804in second image800. Hole706is the same as hole806in second image800. Hole708is not seen in second image800.

Number of reference points902include rivet702, rivet704, and hole706. In view900illustrative markings904are representative of number of reference points902. Illustrative markings904representative of number of reference points902are overlaid on rivet702, rivet704, hole706.

Number of reference points902are used to register first image700. In some illustrative examples, number of reference points902are used to position first image700relative to a coordinate system.

After processing first image700, first image700is analyzed. First image700is analyzed to determine number of locations912of hole708.

Number of reference points902may include any desirable type of features present in first image700. Although not depicted in first image700, number of reference points902could include an edge of structure901, an inconsistency of structure901, a marking, or any other desirable feature.

Turning now toFIG. 10, an illustration of a second image after processing is depicted in accordance with an illustrative embodiment. View1000is a view of second image800with illustrative markings overlaid.

First image700and second image800are views of the same location of structure901at two different times. First image700is taken after drilling hole708. Second image800is taken after inserting rivet808into hole708.

Second image800is processed to determine number of reference points1002. Number of reference points1002include features present in both first image700and second image800. Rivet802is the same as rivet702in first image700. Rivet804is the same as rivet704in first image700. Hole806is the same as hole706in first image700. Rivet808is not seen in first image700.

Number of reference points1002include rivet802, rivet804, and hole806. In view1000illustrative markings1004are representative of number of reference points1002. Illustrative markings1004representative of number of reference points1002are overlaid on rivet802, rivet804, hole806.

Number of reference points1002are used to register second image800. In some illustrative examples, number of reference points1002are used to position second image800relative to a same coordinate system as first image700.

After processing second image800, second image800is analyzed. Second image800is analyzed to determine number of locations1012of rivet808.

Turning now toFIG. 11, an illustration of a second image after processing is depicted in accordance with an illustrative embodiment. View1100is a view of second image800with number of reference points1002and outline1102. Outline1102comprises points1104representative of hole708of first image700. Outline1102is representative of number of locations912inFIG. 9of hole708.

Turning now toFIG. 12, an illustration of representative analysis overlaid on a portion of a second image is depicted in accordance with an illustrative embodiment. View1200is a view of rivet808of second image800with representative analysis1202.

Representative analysis1202is a series of visual indicators of analysis performed by comparator209ofFIG. 2.

As discussed above with reference toFIG. 9, an analysis is performed to determine number of locations912of hole708ofFIG. 9. Number of locations912of hole708are a detected edge of hole708.

As discussed above with reference toFIG. 10, an analysis is performed to determine number of locations1012of rivet808. Number of locations1012of rivet808are a detected edge of the button of rivet808.

In this illustrative example, determining the rivet concentricity comprises determining distances1210between points1204of outline1102of hole708and points1208of outline1206of rivet808. In this illustrative example, the rivet concentricity is a smallest value of distances1210.

Representative analysis1202includes distance1212, distance1214, distance1216, distance1218, and distance1220. Although only five distances are depicted, any desirable number of distances may be determined. In this illustrative example, distance1220is the smallest value. In this illustrative example, distance1220is set as the rivet concentricity.

Turning now toFIG. 13, an illustration of representative analysis overlaid on a portion of a second image is depicted in accordance with an illustrative embodiment. View1300is a view of rivet808of second image800with representative analysis1302. Representative analysis1302is a series of visual indicators of analysis performed by comparator209ofFIG. 2.

As discussed above with reference toFIG. 9, an analysis is performed to determine number of locations912of hole708. Number of locations912of hole708are a detected edge of hole708.

As discussed above with reference toFIG. 10, an analysis is performed to determine number of locations1012of rivet808. Number of locations1012of rivet808are a detected edge of the button of rivet808.

In this illustrative example, determining the rivet concentricity comprises determining distances1306between points1208of outline1206of rivet808and centerpoint1304. In this illustrative example, the rivet concentricity is a smallest value of distances1306.

Representative analysis1302includes distance1308, distance1310, and distance1312. Although only three distances are depicted, any desirable number of distances may be determined. In this illustrative example, distance1312is the smallest value. In this illustrative example, distance1312is set as the rivet concentricity.

Turning now toFIG. 14, an illustration of representative analysis overlaid on a portion of a second image is depicted in accordance with an illustrative embodiment. View1400is a view of rivet808of second image800with representative analysis1402. Representative analysis1402is a series of visual indicators of analysis performed by comparator209ofFIG. 2.

As discussed above with reference toFIG. 9, an analysis is performed to determine number of locations912of hole708. Number of locations912of hole708are a detected edge of hole708.

As discussed above with reference toFIG. 10, an analysis is performed to determine number of locations1012of rivet808. Number of locations1012of rivet808is a detected edge of the button of rivet808.

In this illustrative example, determining the rivet concentricity comprises determining distance1406between centerpoint1304and mass centerpoint1404. In this illustrative example, the rivet concentricity is distance1406.

FIGS. 12-14are representation of analyses performed on a first image and a second image. Although the depictions of representative analysis1202, representative analysis1302and representative analysis1402are illustrations and overlays, in some illustrative examples these analyses may instead be pictorially depicted. In some illustrative examples, at least some of the analyses may be calculations without graphic depictions.

FIGS. 12-14are each images of rivet808ofFIG. 8. In other illustrative examples, a button of a rivet may be a different shape, a different size, or a different location relative to a hole containing the rivet. For example, in each ofFIGS. 12-14, rivet808completely covers hole708from view. In other illustrative examples, a rivet may not completely cover the associated hole. In these illustrative examples, at least one distance between a point of the outline of the rivet and a point of the outline of the hole is negative. In some illustrative examples, when the distance is negative, a respective rivet, such as rivet808, will automatically be reworked.

Turning now toFIG. 15, an illustration of a flowchart of a process for automated rivet measurement is depicted in accordance with an illustrative embodiment. Automated rivet measurement system202may perform method1500within manufacturing environment200ofFIG. 2. Method1500may be performed in manufacturing environment300ofFIG. 3using automated rivet measurement system308. Method1500may be performed on structure304ofFIGS. 3-5. Method1500may be performed using first image700and second image800ofFIGS. 7-11. Method1500installs a rivet within a hole in a structure using an end effector (operation1502).

Method1500processes a first image of the hole in the structure and a second image of the rivet in the structure to identify a number of reference points located in both the first image and the second image (operation1504). In some illustrative examples, the first image and the second image are taken in situ. For example, each of the first image and the second image may be taken during other manufacturing operations, such as drilling, riveting, or movement after drilling or riveting. When the first image and the second image are taken in situ, additional inspection steps are not created. When the first image and the second image are taken in situ, additional end effector movements may not be used. When the first image and the second image are taken in situ, inspection time may be reduced.

In some illustrative examples, first image and second image are taken using the same camera. For example, the first image and the second image may be taken using a camera attached to an end effector used to drill the hole and install the rivet. As another example, a first end effector is used to drill the hole and the first image and the second image may be taken using a camera attached to a second end effector used to install the rivet.

In other illustrative examples, first image and second image are taken using different cameras. For example, the first image may be taken using a camera attached to an end effector used to drill the hole while the second image is taken using a second camera attached to a second end effector used to install the rivet.

In some illustrative examples, small positioning movements are added to position the number of cameras to take first image and second image. In some illustrative examples, small additional positioning movements may be added between other manufacturing operations. In some illustrative examples, an end effector is stopped momentarily between manufacturing operations to take at least one of the first image or the second image.

Method1500aligns the first image and second image using the number of reference points (operation1506). In some illustrative examples, the first image and the second image are oriented relative to a shared coordinate system. In some illustrative examples, coordinates of features of the first image and the second image are saved relative to a shared coordinate system. In some illustrative examples, the second image may be overlaid over the first image.

Method1500analyzes the first image to determine a number of locations of the hole (operation1508). In some illustrative examples, the number of locations include locations of a boundary of the hole.

Method1500analyzes the second image to determine a number of locations of the rivet (operation1510). In some illustrative examples, the number of locations include a boundary of the button of the rivet.

Method1500automatically determines a rivet concentricity using the number of locations of the hole and the number of locations of the rivet (operation1512). Afterwards the method terminates.

In some illustrative examples, method1500determines the rivet concentricity in situ. When rivet concentricity is determined in situ, method1500may not increase manufacturing time. When rivet concentricity is determined in situ, method1500may not add end effector movements. In some illustrative examples, method1500determines the rivet concentricity during at least one of drilling, installing rivets, or movements after drilling or installing rivets.

Turning now toFIG. 16, an illustration of a flowchart of analysis options is depicted in accordance with an illustrative embodiment. Method1600presents analysis options that may be implemented as part of method1500or method1700.

Method1600creates an outline of the hole (operation1602). Method1600creates an outline of the rivet (operation1604). In some illustrative examples, determining the rivet concentricity comprises determining distances between points of the outline of the hole and points of the outline of the rivet, wherein the rivet concentricity is a smallest value of the distances (operation1606).

In some illustrative examples, method determines a centerpoint of the outline of the hole (operation1608). In some illustrative examples, determining the rivet concentricity comprises determining distances between points of the outline of the rivet and the centerpoint, wherein the rivet concentricity is a smallest value of the distance (operation1610). In some illustrative examples, method determines a mass centerpoint of the outline of the rivet, wherein determining the rivet concentricity comprises determining a distance between the centerpoint and the mass centerpoint (operation1612). After any of operation1606, operation1610, or operation1612, method1600terminates.

Turning now toFIG. 17, an illustration of a flowchart of a process for automated rivet measurement is depicted in accordance with an illustrative embodiment. Automated rivet measurement system202may perform method1700within manufacturing environment200ofFIG. 2. Method1700may be performed in manufacturing environment300ofFIG. 3using automated rivet measurement system308. Method1700may be performed on structure304ofFIGS. 3-5. Method1700may be performed using first image700and second image800ofFIGS. 7-11.

Method1700drills a hole in a structure using an end effector configured to perform drilling (operation1702). The hole in the structure is drilled using a first end effector. Method1700takes a first image of the hole in the structure (operation1704). In some illustrative examples, the first image is taken using a camera attached to the first end effector. In some illustrative examples, the first image is taken in situ. For example, the first image may be taken during other manufacturing operations, such as drilling, riveting, or movement after drilling or riveting. When the first image is taken in situ, additional inspection steps are not created. When the first image is taken in situ, additional end effector movements may not be used. When the first image is taken in situ, inspection time may be reduced.

Method1700installs a rivet within the hole using an end effector configured to perform riveting (operation1706). The rivet is installed using an end effector. In some illustrative examples, the rivet is installed using the first end effector used to drill the hole. In other illustrative examples, the rivet is installed with a second end effector. Method1700takes a second image of the rivet in the structure (operation1708).

In some illustrative examples, first image and second image are taken using the same camera. In these illustrative examples, taking the first image and taking the second image are performed by a same camera. For example, the first image and the second image may be taken using a camera attached to the first end effector used to drill the hole and install the rivet. As another example, the first end effector is used to drill the hole and the first image and the second image may be taken using a camera attached to the second end effector used to install the rivet. In some illustrative examples, at least one of taking the first image or taking the second image is performed by a camera attached to the end effector configured to perform drilling. In some illustrative examples, at least one of taking the first image or taking the second image is performed by a camera attached to the end effector configured to perform riveting.

In other illustrative examples, first image and second image are taken using different cameras. For example, the first image may be taken using a camera attached to the first end effector used to drill the hole while the second image is taken using a second camera attached to the second end effector used to install the rivet.

In some illustrative examples, taking the first image and taking the second image are performed in situ. In some illustrative examples, small positioning movements are added to position the number of cameras to take first image and second image. In some illustrative examples, small additional positioning movements may be added between other manufacturing operations. In some illustrative examples, an end effector is stopped momentarily between manufacturing operations to take at least one of the first image or the second image. In some illustrative examples, taking the first image and taking the second image do not add movement steps to the end effector configured to perform drilling or the end effector configured to perform riveting.

Method1700processes the first image and the second image to identify a number of reference points located in both the first image and the second image (operation1710). In some illustrative examples, method1700processes each of the first image and the second image automatically upon receipt. In some illustrative examples, method1700processes the first image and the second image after receiving both the first image and the second image.

Method1700aligns the first image and second image within a coordinate system relative to the number of reference points (operation1712). Method1700analyzes the first image to determine a number of locations of the hole (operation1714).

Method1700analyzes the second image to determine a number of locations of the rivet (operation1716). Method1700automatically determines a rivet concentricity using the number of locations of the hole and the number of locations of the rivet (operation1718). Afterwards the method terminates.

In some illustrative examples, method1700determines the rivet concentricity in situ. When rivet concentricity is determined in situ, method1700may not increase manufacturing time. When rivet concentricity is determined in situ, method1700may not add end effector movements. In some illustrative examples, method1700determines the rivet concentricity during at least one of drilling, installing rivets, or movements after drilling or installing rivets.

In some examples, in method1500, identifying the number of reference points located in both the first image and the second image comprises identifying a rivet present in the first image and the second image as a first feature of the number of reference points. In some examples, in method1500, identifying the number of reference points located in both the first image and the second image comprises identifying another rivet present in the first image and the second image as a second feature of the number of reference points. In some examples, in method1500, the number of reference points comprises at least one of a rivet, a hole, or an edge of the structure.

Turning now toFIG. 18, an illustration of a block diagram of a data processing system is depicted in accordance with an illustrative embodiment. Data processing system1800may be used to implement computer system232inFIG. 2. In this illustrative example, data processing system1800includes communications framework1802, which provides communications between processor unit1804, memory1806, persistent storage1808, communications unit1810, input/output unit1812, and display1814. In this example, communications framework1802may take the form of a bus system.

Processor unit1804serves to execute instructions for software that may be loaded into memory1806. Processor unit1804may be a number of processors, a multi-processor core, or some other type of processor, depending on the particular implementation.

Memory1806and persistent storage1808are examples of storage devices1816. A storage device is any piece of hardware that is capable of storing information, such as, for example, without limitation, at least one of data, program code in functional form, or other suitable information either on a temporary basis, a permanent basis, or both on a temporary basis and a permanent basis. Storage devices1816may also be referred to as computer readable storage devices in these illustrative examples. Memory1806, in these examples, may be, for example, a random-access memory or any other suitable volatile or non-volatile storage device. Persistent storage1808may take various forms, depending on the particular implementation.

For example, persistent storage1808may contain one or more components or devices. For example, persistent storage1808may be a hard drive, a solid state hard drive, a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination of the above. The media used by persistent storage1808also may be removable. For example, a removable hard drive may be used for persistent storage1808.

Communications unit1810, in these illustrative examples, provides for communications with other data processing systems or devices. In these illustrative examples, communications unit1810is a network interface card.

Input/output unit1812allows for input and output of data with other devices that may be connected to data processing system1800. For example, input/output unit1812may provide a connection for user input through at least one of a keyboard, a mouse, or some other suitable input device. Further, input/output unit1812may send output to a printer. Display1814provides a mechanism to display information to a user.

Instructions for at least one of the operating system, applications, or programs may be located in storage devices1816, which are in communication with processor unit1804through communications framework1802. The processes of the different embodiments may be performed by processor unit1804using computer implemented instructions, which may be located in a memory, such as memory1806.

Program code1818is located in a functional form on computer readable media1820that is selectively removable and may be loaded onto or transferred to data processing system1800for execution by processor unit1804. Program code1818and computer readable media1820form computer program product1822in these illustrative examples. In one example, computer readable media1820may be computer readable storage media1824or computer readable signal media1826.

In these illustrative examples, computer readable storage media1824is a physical or tangible storage device used to store program code1818rather than a medium that propagates or transmits program code1818.

Alternatively, program code1818may be transferred to data processing system1800using computer readable signal media1826. Computer readable signal media1826may be, for example, a propagated data signal containing program code1818. For example, computer readable signal media1826may be at least one of an electromagnetic signal, an optical signal, or any other suitable type of signal. These signals may be transmitted over at least one of communications links, such as wireless communications links, optical fiber cable, coaxial cable, a wire, or any other suitable type of communications link.

The illustrative embodiments of the present disclosure may be described in the context of aircraft manufacturing and service method1900as shown inFIG. 19and aircraft2000as shown inFIG. 20. Turning first toFIG. 19, an illustration of a block diagram of an aircraft manufacturing and service method is depicted in accordance with an illustrative embodiment. During pre-production, aircraft manufacturing and service method1900may include specification and design1902of aircraft2000inFIG. 20and material procurement1904.

During production, component and subassembly manufacturing1906and system integration1908of aircraft2000inFIG. 20takes place. Thereafter, aircraft2000inFIG. 20may go through certification and delivery1910in order to be placed in service1912. While in service1912by a customer, aircraft2000inFIG. 20is scheduled for routine maintenance and service1914, which may include modification, reconfiguration, refurbishment, or other maintenance and service.

With reference now toFIG. 20, an illustration of a block diagram of an aircraft is depicted in which an illustrative embodiment may be implemented. In this example, aircraft2000is produced by aircraft manufacturing and service method1900inFIG. 19and may include airframe2002with plurality of systems2004and interior2006. Examples of systems2004include one or more of propulsion system2008, electrical system2010, hydraulic system2012, and environmental system2014. Any number of other systems may be included. Although an aerospace example is shown, different illustrative embodiments may be applied to other industries, such as the automotive industry.

The apparatuses and methods embodied herein may be employed during at least one of the stages of aircraft manufacturing and service method1900inFIG. 19. In one illustrative example, components or subassemblies produced in component and subassembly manufacturing1906inFIG. 19may be fabricated or manufactured in a manner similar to components or subassemblies produced while aircraft2000is in service1912inFIG. 19. As yet another example, one or more apparatus embodiments, method embodiments, or a combination thereof may be utilized during production stages, such as component and subassembly manufacturing1906and system integration1908inFIG. 19. One or more apparatus embodiments, method embodiments, or a combination thereof may be utilized while aircraft2000is in service1912, during maintenance and service1914inFIG. 19, or both. For example, inspection of rivets, such as rivets203, may be performed using an automated rivet measurement system during component and subassembly manufacturing1906to inspect the rivets. Additionally, the automated rivet measurement system, such as automated rivet measurement system202ofFIG. 2, also may be used during maintenance and service1914to inspect already installed rivets, rivets installed during routine maintenance including reconfiguration, refurbishment, and other maintenance or service.

One or more apparatus embodiments, method embodiments, or a combination thereof may be utilized in any desirable components of aircraft2000. For example, inspection of rivets, such as rivets203ofFIG. 2, may be performed using an automated rivet measurement system during installation of rivets in airframe2002. As another example, inspection of rivets, such as rivets203, may be performed using an automated rivet measurement system during installation of rivets in interior2006.

The use of a number of the different illustrative embodiments may substantially expedite the assembly of aircraft2000, reduce the cost of aircraft2000, or both expedite the assembly of aircraft2000and reduce the cost of aircraft2000. For example, the use of an automated rivet measurement system, in accordance with an illustrative example, may reduce the time and cost for manufacturing aircraft2000. For example, the time needed to inspect rivets may be reduced. In this manner, aircraft2000may be manufactured more quickly as compared to using current techniques that involve destructive testing measurements.

Thus, one or more technical solutions are present that overcome a technical problem with obtaining measurements of rivet concentricity without using destructive testing. One or more technical solutions may provide a technical effect identifying rivet concentricity without having to perform destructive testing. Also, one or more technical solutions are present that provide an ability to determine a rivet concentricity in situ.

The description of the different illustrative embodiments has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the embodiments in the form disclosed. The different illustrative examples describe components that perform actions or operations. In an illustrative embodiment, a component may be configured to perform the action or operation described. For example, the component may have a configuration or design for a structure that provides the component an ability to perform the action or operation that is described in the illustrative examples as being performed by the component.