Patent ID: 12189376

DETAILED DESCRIPTION

The illustrative examples recognize and take into account one or more different considerations as described below. For example, illustrative examples recognize and take into account that it would be desirable to have a method and apparatus that overcome a technical problem with reducing noise and outliers in test data. For example, the illustrative examples recognize and take into account that various factors can contribute to the presence of outliers in test data. For example, improper test sample preparation, excessive processing defects, improper acquisition of test data during testing, and equipment error are some examples of factors that can cause outliers and noise to be present in test data. Multiple factors can be present leading to the formation of different types of outliers, affecting the accuracy of the test data. Undetected outliers can lead to improper characterization of material properties of physical structures.

Current outlier detection techniques have a number of challenges. For example, the illustrative examples recognize and take into account current outlier detection techniques are unable to detect all types of outliers. Further, as the amount of test data decreases, it becomes more challenging to detect outliers using current outlier detection techniques. Additionally, one manner in which test data is examined for outliers involves operator know-how, engineering judgment, and experience. This type of technique can introduce error.

Thus, the illustrative examples provide a method, apparatus, system, and computer program product for managing outliers. In one illustrative example, a set of features derived from the test data is analyzed using a plurality of outlier detection methods to generate a result of outliers identified by the plurality outlier detection methods, wherein the test data is obtained from testing a physical structure. A causality for a set of outliers in the result matrix is determined. Retesting of the physical structure the physical structure can be performed with a set of changes determined based on the causality identified for the set of outliers. The retesting generates new test data for the physical structure.

With reference now to the figures and, in particular, with reference toFIG.1, a pictorial representation of a network of data processing systems is depicted in which illustrative examples may be implemented. Network data processing system100is a network of computers in which the illustrative examples may be implemented. Network data processing system100contains network102, which is the medium used to provide communications links between various devices and computers connected together within network data processing system100. Network102may include connections, such as wire, wireless communication links, or fiber optic cables.

In the depicted example, server computer104and server computer106connect to network102along with storage unit108. In addition, client devices110connect to network102. As depicted, client devices110include client computer112, client computer114, and client computer116. Client devices110can be, for example, computers, workstations, or network computers. In the depicted example, server computer104provides information, such as boot files, operating system images, and applications to client devices110. Further, client devices110can also include other types of client devices such as mobile phone118, tablet computer120, and smart glasses122. In this illustrative example, server computer104, server computer106, storage unit108, and client devices110are network devices that connect to network102in which network102is the communications media for these network devices. Some or all of client devices110may form an Internet of things (IoT) in which these physical devices can connect to network102and exchange information with each other over network102.

Client devices110are clients to server computer104in this example. Network data processing system100may include additional server computers, client computers, and other devices not shown. Client devices110connect to network102utilizing at least one of wired, optical fiber, or wireless connections.

Program instructions located in network data processing system100can be stored on a computer-recordable storage media and downloaded to a data processing system or other device for use. For example, program instructions can be stored on a computer-recordable storage media on server computer104and downloaded to client devices110over network102for use on client devices110.

In the depicted example, network data processing system100is the Internet with network102representing a worldwide collection of networks and gateways that use the Transmission Control Protocol/Internet Protocol (TCP/IP) suite of protocols to communicate with one another. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers consisting of thousands of commercial, governmental, educational, and other computer systems that route data and messages. Of course, network data processing system100also may be implemented using a number of different types of networks. For example, network102can be comprised of at least one of the Internet, an intranet, a local area network (LAN), a metropolitan area network (MAN), or a wide area network (WAN). In this illustrative example, network data processing system100can be used to provide a cloud computing environment.FIG.1is intended as an example, and not as an architectural limitation for the different illustrative examples.

As used herein, “a number of” when used with reference to items, means one or more items. For example, “a number of different types of networks” is one or more different types of networks.

Further, the phrase “at least one of,” when used with a list of items, means different combinations of one or more of the listed items can be used, and only one of each item in the list may be needed. In other words, “at least one of” means any combination of items and number of items may be used from the list, but not all of the items in the list are required. The item can be a particular object, a thing, or a category.

For example, without limitation, “at least one of item A, item B, or item C” may include item A, item A and item B, or item B. This example also may include item A, item B, and item C or item B and item C. Of course, any combinations of these items can be present. In some illustrative examples, “at least one of” can be, for example, without limitation, two of item A; one of item B; and ten of item C; four of item B and seven of item C; or other suitable combinations.

As depicted, physical structures in the form composite parts130are tested in testing facility132. The testing of composite parts130can include the determination of bulk properties through the use of tension, compression, and share tests. These tests can be used explore properties of composite parts130such as open hole tension, open hole compression, interim laminar fracture toughness, compression after impact, fatigue, and other properties. These tests can be performed over a range of environments. This testing of composite parts130generates test data134. In this illustrative example, client computer114sends test data134over network102to data manager136located in server computer104for processing.

Data manager136manages test data134. For example, data manager136can analyze test data134, identify outliers in test data134, reduce noise in test data134, or perform other operations on test data134. In managing test data134, data manager136can identify outliers more efficiently as compared to current techniques for outlier detection. For example, data manager136can use multiple ones of outlier detection methods138to analyze test data134to detect outliers140in test data134. In this illustrative example, outlier detection methods138comprises detection methods that are suitable for detecting different types of outliers140.

The selection of outlier detection methods138can be made such that different outlier detection methods can identify different outliers such that the group of outlier detection methods138can detect more types of outliers as compared to individual outlier detection methods. As result, the selection of two or more of different ones of detection methods138can provide increase performance in detecting outliers as compared to current techniques.

In this illustrative example, data manager136can identify outlier types142for outliers140identified in test data134. Additionally, data manager136can identify the cause of outliers140based on the determination of outlier types142. By identifying the cause of outliers140, data manager136can determine whether the testing of composite parts130is needed.

Data manager136can retest composite parts130in a number of different ways. The retesting can be initiated by data manager136sending at least one of instructions, commands, or other types of information to testing facility132. The retesting of composite parts130can involve changing one or more measurement processes used by testing facility132to test composite parts130.

As another example, data manager136may determine that one or more of outliers140is caused by composite parts130not meeting specifications for composite parts130. With this type of causation of an outlier, data manager136can retest composite parts130by having composite parts130remanufactured with changes in manufacturing such that the remanufactured version of composite parts130meet specifications for composite parts130. This retesting generates new test data that can be analyzed.

Further, in managing test data134data manager136can also remove noise143from test data134. Thus, data manager136can manage test data134to provide test data134that can be used to show that composite parts130meets criteria or specifications specified by regulations, a manufacturer, or other source.

With reference next toFIG.2, an illustration of a test data environment is depicted in accordance with an illustrative example. In this illustrative example, test data environment200includes components that can be implemented in hardware such as the hardware shown in network data processing system100inFIG.1.

In this illustrative example, outlier management system202in test data environment200can operate to process test data204. As depicted, test data204is generated from testing of physical structure206by testing system208.

In this illustrative example, the testing of physical structure206is a set of physical tests performed on physical structure206. Physical structure206can take a number of different forms. For example, physical structure206can be selected from a group comprising a composite part, a test coupon, an assembly, a system, an alloy part, metal structure, and other types of physical structures. In this illustrative example, a composite part can be, for example, a skin panel, a wing, or some other suitable type of composite part. A system or assembly can be comprised of multiple materials. For example, an assembly can be comprised of parts formed from composite materials, plastic materials, and metal materials.

Testing system208can be one or more pieces of test equipment that can be used to perform tests on physical structure206. This test equipment can be used to perform testing such as hardness testing, impact testing, fracture toughness testing, pretesting, fatigue testing, nondestructive testing, and other types of testing to generate test data204.

Test data204is sent from testing system208to outlier management system202for processing. As depicted, outlier management system202comprises data manager210in computer system212.

Data manager210can be implemented in software, hardware, firmware or a combination thereof. When software is used, the operations performed by data manager210can be implemented in program code configured to run on hardware, such as a processor unit. When firmware is used, the operations performed by data manager210can be implemented in program code and data and stored in persistent memory to run on a processor unit. When hardware is employed, the hardware can include circuits that operate to perform the operations in data manager210.

In the illustrative examples, the hardware can take a form selected from at least one of a circuit system, an integrated circuit, an application specific integrated circuit (ASIC), a programmable logic device, or some other suitable type of hardware configured to perform a number of operations. With a programmable logic device, the device can be configured to perform the number of operations. The device can be reconfigured at a later time or can be permanently configured to perform the number of operations. Programmable logic devices include, for example, a programmable logic array, a programmable array logic, a field programmable logic array, a field programmable gate array, and other suitable hardware devices. Additionally, the processes can be implemented in organic components integrated with inorganic components and can be comprised entirely of organic components excluding a human being. For example, the processes can be implemented as circuits in organic semiconductors.

Computer system212is a physical hardware system and includes one or more data processing systems. When more than one data processing system is present in computer system212, those data processing systems are in communication with each other using a communications medium. The communications medium can be a network. The data processing systems can be selected from at least one of a computer, a server computer, a tablet computer, or some other suitable data processing system.

As depicted, computer system212includes a number of processor units214that are capable of executing program instructions216implementing processes in the illustrative examples. As used herein a processor unit in the number of processor units214is a hardware device and is comprised of hardware circuits such as those on an integrated circuit that respond and process instructions and program code that operate a computer. When a number of processor units214execute program instructions216for a process, the number of processor units214is one or more processor units that can be on the same computer or on different computers. In other words, the process can be distributed between processor units on the same or different computers in a computer system. Further, the number of processor units214can be of the same type or different type of processor units. For example, a number of processor units can be selected from at least one of a single core processor, a dual-core processor, a multi-processor core, a general-purpose central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or some other type of processor unit.

In this illustrative example, data manager210in computer system212can detect outliers218for set of features220using different outlier detection methods222. In this illustrative example, data manager210can identify the set of features220from test data204. In this illustrative example, a feature in the set of features220represents an individual measurable property or characteristic of a phenomenon for physical structure206. The feature can be a physical response or failure mode that can occur for physical structure206. For example, a feature can be a peak in a forced displacement curve, a length of a crack in an image, or some other suitable feature.

Different outlier detection methods222can take a number of different forms. For example, different outlier detection methods222can be selected from two or more of a cosine similarity, a correlation analysis, a principal component analysis, a Sprague-Geers analysis, a robust principal component analysis, or some other suitable outlier detection method that is currently available. In this example, different outlier detection methods222can compare different parameters. For example, different outlier detection methods can compare at least one of lengths, angles, Eigen values, Eigen vectors, or other metrics or combinations of metrics used for outlier detection in different outlier detection methods222.

Further, the phrase “two or more of,” when used with a list of items, means different combinations of two or more of the listed items can be used, and only one of each item in the list may be needed. In other words, “two or more of” means any combination of items and number of items may be used from the list, but not all of the items in the list are required. The item can be a particular object, a thing, or a category.

For example, without limitation, “two or more of item A, item B, or item C” may include item A and item B, item A and item C, or item B and item C. This example also may include item A, item B. Of course, any combinations of these items can be present. In some illustrative examples, “two or more of” can be, for example, without limitation, two of item A and item B; one of item B and3of item C; and ten of item C; four of item B and seven of item C; or other suitable combinations.

As depicted, data manager210generates result224of a set of outliers218identified by each outlier detection in different outlier detection methods222. In one illustrative example, result224can take the form of result matrix226for identifying outliers218. Result matrix226identifies outliers218and outlier detection methods that identified outliers218.

Further, data manager210can determine causality228for the set of outliers218. Causality228for an outlier in the set of outliers218is an identification of a cause of the outlier. Causality228can be determined using historical data230from prior tests and analysis of physical structures.

In some cases, and outlier in the set of outliers218can be an outlier without causality228. This type of outlier can be discarded.

In determining causality228, data manager210identifies a set of outlier types232for an outlier in result224. In other words, an outlier can have one more than one outlier type. Data manager210determines causality228for the outlier using the set of outlier types232identified for the outlier.

In this illustrative example, data manager210can retest physical structure206with a set of changes234determined using causality228identified for the set of outliers218. The set of changes234can take a number of different forms. For example, the set of changes234can be selected from at least one of changes comprises at least one of a measurement process change, a geometry change, a manufacturing parameter change, a manufacturing process change, or some other suitable change.

The retesting generates new test data236for physical structure206. As depicted, new test data236can be analyzed by data manager210to determine a presence of outliers218.

The retesting can comprise retesting physical structure206using a change to measurement process238used in testing system208in response to causality228indicating that measurement process238was a cause of an outlier in the set of outliers218. The change in the measurement process can be, for example, changing measurement equipment, recalibrating measurement equipment, changing a measurement technique, adding a new measurement process, changing handling of physical structure206during testing, or some other suitable change in measurement process238.

In another the example, the retesting can involve data manager210manufacturing new physical structure240with the set of changes234identified and retesting new physical structure240. In this illustrative example, physical structure206may have an incorrect dimension. For example, physical structure206may have a thickness that is less than the specified thickness for physical structure206. In this example, the change is a geometry change made to remanufacture is physical structure206with the specified thickness. In another example, physical structure206using a waterjet. The waterjet cause inconsistencies. The change can include cutting physical structure206using a diamond cutter.

Additionally, data manager210can remove noise242in outliers218from test data204as well as other types of outliers218. For example, data manager210can remove noise242from the set of features220using at least one of removing noise242in outliers218prior to analyzing the set of features220derived from test data204using different outlier detection methods222or using an outlier detection method in different outlier detection methods222that removes noise242in outliers218.

The illustration of test data environment200inFIG.2is not meant to imply physical or architectural limitations to the manner in which an illustrative example may be implemented. Other components in addition to or in place of the ones illustrated may be used. Some components may be unnecessary. Also, the blocks are presented to illustrate some functional components. One or more of these blocks may be combined, divided, or combined and divided into different blocks when implemented in an illustrative example.

For example, the testing of physical structure206can be performed using a simulation of physical structure206. With this type of implementation, testing system208can be, for example, a simulation model such as a finite element analysis model.

Additionally, the illustrative example can also use different types of features including transform features based on theory and governing physics of the problem to find the underlying features. This type of feature can also be referred to as a theory guided feature transformation. This transformed data can be analyzed using the different outlier detection methods with greater accuracy as compared to current techniques.

In yet another example, retesting physical structure206is unnecessary. Test data204with the removal of outliers218can provide the quality needed for using test data204. For example, test data204with the removal of outliers218can be used validate a simulation model with the outliers218removed from features220.

In another illustrative example, noise removal250can be used to remove noise242in test data204prior to detecting outliers218using different outlier detection methods222. For example, noise removal250can use filters for noise reduction processes that may not detect outliers218.

Turning now toFIG.3, an illustration of types of features is depicted in accordance with an illustrative example. In the illustrative examples, the same reference numeral may be used in more than one figure. This reuse of a reference numeral in different figures represents the same element in the different figures.

Features220can take a number of different forms that can be identified by data manager210for use in detecting outliers218using different outlier detection methods222. As depicted, the set of features220can be selected from at least one of standard feature300, transformed feature302, or selected feature304. In other words, the set of features220can be one or more of these feature types and can be in any combination.

In this illustrative example, standard feature300comprises test data204from a test. In this illustrative example, test data204can be, for example, data from multiple tests performed multiple times or from a single test performed multiple times. Standard feature300can be, for example, test data for a force displacement curve from values in the test data from a test measuring displacement of physical structure206in response to force.

In another example, standard feature300can be an image. The image can be an image of a failure, fracture, the elimination, or other type of image showing the response of physical structure206.

In this illustrative example, transformed feature302is test data204that has been transformed by a mathematical operation. Transformed feature302can also be referred to as a theory guided feature. For example, transformed feature302can be result of a derivative of test data204, an integral of test data204, or from some other mathematical operation performed on test data204.

For example, the derivative of test data204from a force displacement test can be a tangent or secant stiffness. The integral of test data from a force displacement test can be fractured energy. The selection of the operation performed in the test data is driven by an understanding of the response of the material. For example, the response can be a failure, delamination, or fracture of the material.

Selected feature304is a selection of a portion of test data204. For example, selected feature304can be a peak of a force displacement curve, an area of interest in a force displacement curve, a slope of a line, or other selected feature. In other words, selected feature304can be a subset of test data204containing data of interest.

In another example, selected feature304can be a portion of an image. For example, selected feature304can be a length of a crack, a number of parallel lines, or some other selected feature in an image.

In one illustrative example, one or more technical solutions are present that overcome a technical problem with detecting outliers in test data. As a result, one or more technical solutions can provide a technical effect enabling managing outliers using a plurality of different types of outlier detection methods.

Turning toFIG.4, an illustration of outlier types and causes is depicted in accordance with an illustrative example. In this illustrative example, table500depicts outlier types401and causes421of outlier types401. These outlier types are examples of some of the different types of outliers that can be present along with causes421for outlier types401. As depicted, outlier types401comprise type 1402, type 2404, type 3406, type 4408, type 5410, type 6412, and type 7414.

In this illustrative example, type 1402is caused by premature failure403. Premature failure403can be due to defects or other issues resulting from manufacturing variability in physical structure206. Type 2404is caused by stiffness variation405. In this example, stiffness variation405can result from, for example, from at least one of improper strange gauge measurements, improper layout, or improper cutting of physical structure206.

As depicted, type 3406is caused by stiffness error407. Stiffness error407can occur due to tabs sliding, loading fixture rotation, or other test fixture issues in handling the physical structure206during the testing process.

Type 4408is caused by nonlinear variation409. Nonlinear variation409can be identified from jumps in loading and load displacement data which can occur in response to manufacturing variability in physical structure206.

In another example, type 5410is caused by noisy data411, which can be caused by test equipment. For example, a faulty strain gauge or other data acquisition system equipment can cause this type of outlier. Noise filtering may be used to obtain an acceptable amount of data without retesting. Type 6412is caused by unexpected load drop413. This unexpected load drop can occur due to delamination of physical structure206.

As depicted, type 7414is caused by different damage modes415. These damage modes are present in images of the physical structure and are caused by testing the physical structure.

The outlier types are examples of different outlier types for outliers218inFIG.2. By identifying the different types of outliers218, causes421associated with those outliers can be identified. Some of causes421may simply require performing noise removal.

In other illustrative examples, causes421can be true outliers that may be discarded. In other illustrative examples, causes of causes421can be used to determine a set of changes234that may need to be performed in retesting physical structure206. This retesting may include at least one of changes to the measurement processor or remanufacturing physical structure206with changes to obtain suitable test data.

With reference next toFIG.5, an illustration of outlier detection methods is depicted in accordance with an illustrative example. Table500identifies the outlier detection method in column502and the feature type in column504for the data analyzed by the outlier detection method.

Table500also identifies outlier types that can be detected by each outlier detection method using a particular feature type. In this example, type 1 is column508, type 2 is column510, type 3 is column512, type 4 is column514, type 5 is column516, type 6 is column518, and type 7 is in column519. These outlier types correspond to the outlier types401described inFIG.4.

In row520, robust principal component analysis (RPCA) is the outlier detection that that can be used with standard features. This type of outlier detection method can compare sparse matrices. This type of outlier detection method can also automatically remove noise as part of the outlier detection process. With standard features, robust principal component analysis can detect type 1, type 5, and type 6 outliers.

Next, in row522, Sprague-Geers (SG) comprehensive error is a type of outlier detection method that can be used to detect outliers using standard features. With this type of test data, Sprague-Geers comprehensive error can detect type 1, type 2, and type 3 outliers.

Next, robust principal component analysis in row524can be used with transform features to detect outliers. With transform features, robust principal component analysis can detect type 1, type 3, type 4, type 5, and type 6 outliers. In row526, robust principal component analysis is used with selected features to detect outliers. With the use of selected features, robust principal component analysis can detect type 1, type 2, type 3, and type 4 outliers.

In row528, Sprague-Geers (SG) comprehensive error is used with selected features to detect outliers. In this example, Sprague-Geers (SG) comprehensive error can detect type 1, type 4, and type 7 outliers.

Thus, with selection of different outlier detection methods, all of the different types of outliers can be detected. This illustration is not meant to limit the manner in which other selections of outliers can be made for different outlier detection methods222inFIG.2. Other numbers and other types of outlier detection methods can be used depending on the particular implementation. The selection of the outlier detection methods for use in different outlier detection methods222can be made such that the different types of outliers can be detected. As result, all outlier types of interest can be detected even though a particular outlier detection method is unable to detect all of the outlier types of interest.

Turning toFIG.6, an illustration of dataflow in identifying causality for various outliers is depicted in accordance with an illustrative example. As depicted, historical experimental data600is test data602obtained from prior testing of physical structures. Historical experimental data600comprises test data602from tests performed on a physical structure. Historical experimental data600can be labeled to form labeled historical data604(operation601). In this illustrative example, labeled historical data604can include test data identification609, outlier presence606, outlier type608, and cause610for the different outlier types.

This information can be determined by engineers or other subject matter experts analyzing test data602from different tests and historical experimental data600. In other illustrative examples, the test data can also be analyzed using models or other software systems.

In this illustrative example, outlier detection code612is program code implementing outlier detection methods. Outlier detection code612can receive test data602(operation603) and detect outliers using different outlier detection methods (operation605).

The result of this process is analyzed historical data614. Analyzed historical data614identifies which of the different outlier detection methods identified in columns617detected outliers in different sets of test data602identified in rows615in analyzed historical data614.

Labeled historical data604is compared with analyzed historical data614(operation607), resulting in causality map620. As depicted, causality map620identifies outlier detection methods and what outliers can be detected by each of the outlier detection method in section622. Outlier types for the detected outliers in section622are identified in section624. Causality for outlier types in section624identified in section626.

In this illustrative example, causality228for outliers218detected using different outlier detection methods222inFIG.2can be identified using causality map620. As a result, if the test data is not sufficient or outliers218are not truly outliers but have causes, a set of changes234can be identified for use in retesting physical structure206.

The illustrative example enables more accurate outlier detection with smaller amounts of test data as compared to current techniques in which the accuracy of outlier detection in these current techniques depend on the amount of test data available. Further, features in the test data can be analyzed using a plurality of different outlier detection methods in the illustrative example.

Thus, the illustrative example can use these different features collectively to detect many types of outliers as well as remove noise from data. This processing of test data can allow identifying the true response to the material absent with the reduction or removal of outliers and noise from the test data.

Computer system212can be configured to perform at least one of the steps, operations, or actions described in the different illustrative examples using software, hardware, firmware or a combination thereof. As a result, computer system212with data manager210operates as a special purpose computer system in which in computer system212enables detecting outliers and causes of outliers a more efficient manner as compared to current techniques. For example, data manager210transforms computer system212into a special purpose computer system as compared to currently available general computer systems that do not have data manager210.

In the illustrative example, the use of data manager210in computer system212integrates processes into a practical application for managing outliers in test data to meet goals. This managing of outliers includes retesting a physical structure when outliers are not true outliers and have causes that are identified. With the identified of the causes, data manager210in computer system212can retest the physical structure with a set of changes identified using because is determined the outliers.

With reference toFIG.7, an illustration of a flowchart of a process for outlier detection is depicted in accordance with an illustrative example. The process illustrated in this figure can be implemented in hardware, software, or both. When implemented in software, the process can take the form of program code that is run by one of more processor units located in one or more hardware devices in one or more computer systems. For example, the process can be implemented in data manager210in computer system212inFIG.2.

The process begins by testing coupons (operation700). Coupons are examples of test structures and can be manufactured using composite materials in this example. In operation700, the coupons can have different orientations for layers of composite materials. Further, tab cuts, notches, or other features can also be cut into the coupons for testing. The testing can include pulling, twisting, bending, and application loads to determine the response the coupons to these different types of forces.

The process generates test data from the testing of test coupons (operation702). The test data can include, for example, forced displacement data images of the coupons and other data.

The process then performs feature engineering to identify features from the test data (704). In this example, features can be standard features, transform features, selected features, or some combination thereof. With coupons, transform features can be created using a derivative of the test data can be performed to obtain tangent or secant stiffness. An integral of the test data can be performed to determine fractured energy is a transformed feature. Further, different features can be selected such as peak load, sharp drop, cracking, or other features in the test data or images.

The process then performs outlier detection and noise removal using different outlier detection methods (operation706). The result of operation706identifies the outliers detected by the different outlier detection methods. Operation706can also optionally include removing noise from the test data before using outlier detection processes. The noise removal can be performed using filtering processes. In other cases, noise can be an outlier that can be removed as part of the outlier detection process.

The process identifies outlier types and causes (operation708). The process terminates thereafter.

Turning next toFIG.8, an illustration of a flowchart of a process for managing outliers in test data is depicted in accordance with an illustrative example. The process inFIG.8can be implemented in hardware, software, or both. When implemented in software, the process can take the form of program code that is run by one of more processor units located in one or more hardware devices in one or more computer systems. For example, the process can be implemented in data manager210in computer system212inFIG.2.

The process begins by analyzing a set of features derived from the test data using different outlier detection methods to generate a result of a set of outliers identified by the different outlier detection methods, wherein the test data is obtained from testing a physical structure (operation800). The process determines a causality for the set of outliers in the result (operation802).

The process retests the physical structure with a set of changes determined using the causality identified for the set of outliers, wherein the retesting generates new test data for the physical structure (operation804). The process terminates thereafter.

Turning now toFIG.9, an illustration of a flowchart of a process for detecting outliers is depicted in accordance with an illustrative example. The process illustrated in this figure is an example of additional operations that can be performed in the process inFIG.8.

The process begins by detecting outliers for the set of features using the different outlier detection methods (operation900). The process generates the result of the set of outliers detected by each outlier detection method in the different outlier detection methods (operation9002). The process terminates thereafter.

With reference toFIG.10, an illustration of a flowchart of a process for feature identification is depicted in accordance with an illustrative example. The operation in this flowchart is an example of an additional operation that can be performed in the process inFIG.8.

The process identifies the set of features from the test data, wherein the set of features is selected from at least one of a standard feature, a transformed feature, or a selected feature (operation1000). The process terminates thereafter.

Turning toFIG.11, an illustration of a flowchart of a process for removing noise is depicted in accordance with an illustrative example. The operation in this flowchart is an example of an additional operation that can be performed with the process inFIG.8.

The process removes noise from the set of features using at least one of removing the noise prior to analyzing the set of features derived from the test data using the different outlier detection methods or using an outlier detection method in the different outlier detection methods that removes the noise (operation1100). The process terminates thereafter.

InFIG.12, an illustration a flowchart of a process for determining causality is depicted in accordance with an illustrative example. The operations in this flowchart is an example of one implementation for operation802inFIG.8.

The process begins by identifying a set of outlier types for an outlier in the result (operation1200). The process determines the causality for the outlier using the set of outlier types identified for the outlier (operation1202). The process terminates thereafter. The identification of the causality for the outlier using outlier types can be made based on historical data. For example, previous data collection and analysis of data can determine what causes of different types of outliers.

With reference next toFIG.13, an illustration a flowchart of a process for retesting for a physical structure is depicted in accordance with an illustrative example. The process inFIG.13is an example of one implementation for operation804inFIG.8.

The process retests the physical structure using a change to a measurement process in response to the causality indicating that the measurement process was a cause of an outlier in the set of outliers (operation1300). The process terminates thereafter.

Next inFIG.14, an illustration a flowchart of another process for retesting a physical structure is depicted in accordance with an illustrative example. The process inFIG.14is an example of one implementation for operation804inFIG.8.

The process begins by manufacturing a new physical structure with the set of changes identified (operation1400). In operation1400, the set of changes can comprise at least one of, a measurement process change, a geometry change, a manufacturing parameter change, or a manufacturing process change.

The process retests the new physical structure (operation1402). The process terminates thereafter.

With reference toFIG.15, an illustration a flowchart of a process for outlier detection is depicted in accordance with an illustrative example. The process illustrated in this figure can be implemented in hardware, software, or both. When implemented in software, the process can take the form of program code that is run by one of more processor units located in one or more hardware devices in one or more computer systems. For example, the process can be implemented in data manager210in computer system212inFIG.2.

The process begins by analyzing a set of features derived from the test data using different outlier detection methods to generate a result of a set of outliers identified by the different outlier detection methods, wherein the test data is obtained from testing a physical structure (operation1500).

The process removes the outliers from the features (operation1502). The process validates a simulation model with the outliers removed from the set of features (operation1504). The process terminates thereafter.

The flowcharts and block diagrams in the different depicted examples illustrate the architecture, functionality, and operation of some possible implementations of apparatuses and methods in an illustrative example. In this regard, each block in the flowcharts or block diagrams can represent at least one of a module, a segment, a function, or a portion of an operation or step. For example, one or more of the blocks can be implemented as program code, hardware, or a combination of the program code and hardware. When implemented in hardware, the hardware can, for example, take the form of integrated circuits that are manufactured or configured to perform one or more operations in the flowcharts or block diagrams. When implemented as a combination of program code and hardware, the implementation may take the form of firmware. Each block in the flowcharts or the block diagrams can be implemented using special purpose hardware systems that perform the different operations or combinations of special purpose hardware and program code run by the special purpose hardware.

In some alternative implementations of an illustrative example, the function or functions noted in the blocks may occur out of the order noted in the figures. For example, in some cases, two blocks shown in succession may be performed substantially concurrently, or the blocks may sometimes be performed in the reverse order, depending upon the functionality involved. Also, other blocks may be added in addition to the illustrated blocks in a flowchart or block diagram.

Turning now toFIG.16, an illustration of a block diagram of a data processing system is depicted in accordance with an illustrative example. Data processing system1600can be used to implement server computer104, server computer106, client devices110, inFIG.1. Data processing system1600can also be used to implement computer system212inFIG.2. In this illustrative example, data processing system1600includes communications framework1602, which provides communications between processor unit1604, memory1606, persistent storage1608, communications unit1610, input/output (I/O) unit1612, and display1614. In this example, communications framework1602takes the form of a bus system.

Processor unit1604serves to execute instructions for software that can be loaded into memory1606. Processor unit1604includes one or more processors. For example, processor unit1604can be selected from at least one of a multicore processor, a central processing unit (CPU), a graphics processing unit (GPU), a physics processing unit (PPU), a digital signal processor (DSP), a network processor, or some other suitable type of processor. Further, processor unit1604can may be implemented using one or more heterogeneous processor systems in which a main processor is present with secondary processors on a single chip. As another illustrative example, processor unit1604can be a symmetric multi-processor system containing multiple processors of the same type on a single chip.

Memory1606and persistent storage1608are examples of storage devices1616. A storage device is any piece of hardware that is capable of storing information, such as, for example, without limitation, at least one of data, program code in functional form, or other suitable information either on a temporary basis, a permanent basis, or both on a temporary basis and a permanent basis. Storage devices1616may also be referred to as computer-readable storage devices in these illustrative examples. Memory1606, in these examples, can be, for example, a random-access memory or any other suitable volatile or non-volatile storage device. Persistent storage1608can take various forms, depending on the particular implementation.

For example, persistent storage1608may contain one or more components or devices. For example, persistent storage1608can be a hard drive, a solid-state drive (SSD), a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination of the above. The media used by persistent storage1608also can be removable. For example, a removable hard drive can be used for persistent storage1608.

Communications unit1610, in these illustrative examples, provides for communications with other data processing systems or devices. In these illustrative examples, communications unit1610is a network interface card.

Input/output unit1612allows for input and output of data with other devices that can be connected to data processing system1600. For example, input/output unit1612can provide a connection for user input through at least one of a keyboard, a mouse, or some other suitable input device. Further, input/output unit1612can send output to a printer. Display1614provides a mechanism to display information to a user.

Instructions for at least one of the operating system, applications, or programs can be located in storage devices1616, which are in communication with processor unit1604through communications framework1602. The processes of the different examples can be performed by processor unit1604using computer-implemented instructions, which can be located in a memory, such as memory1606.

These instructions are program instructions and are also referred to as program code, computer usable program code, or computer-readable program code that can be read and executed by a processor in processor unit1604. The program code in the different examples can be embodied on different physical or computer-readable storage media, such as memory1606or persistent storage1608.

Program code1618is located in a functional form on computer-readable media1620that is selectively removable and can be loaded onto or transferred to data processing system1600for execution by processor unit1604. Program code1618and computer-readable media1620form computer program product1622in these illustrative examples. In the illustrative example, computer-readable media1620is computer-readable storage media1624.

Computer-readable storage media1624is a physical or tangible storage device used to store program code1618rather than a media that propagates or transmits program code1618. Computer-readable storage media1624, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Alternatively, program code1618can be transferred to data processing system1600using a computer-readable signal media. The computer-readable signal media are signals and can be, for example, a propagated data signal containing program code1618. For example, the computer-readable signal media can be at least one of an electromagnetic signal, an optical signal, or any other suitable type of signal. These signals can be transmitted over connections, such as wireless connections, optical fiber cable, coaxial cable, a wire, or any other suitable type of connection.

Further, as used herein, “computer-readable media1620” can be singular or plural. For example, program code1618can be located in computer-readable media1620in the form of a single storage device or system. In another example, program code1618can be located in computer-readable media1620that is distributed in multiple data processing systems. In other words, some instructions in program code1618can be located in one data processing system while other instructions in program code1618can be located in one data processing system. For example, a portion of program code1618can be located in computer-readable media1620in a server computer while another portion of program code1618can be located in computer-readable media1620located in a set of client computers.

The different components illustrated for data processing system1600are not meant to provide architectural limitations to the manner in which different examples can be implemented. In some illustrative examples, one or more of the components may be incorporated in or otherwise form a portion of, another component. For example, memory1606, or portions thereof, can be incorporated in processor unit1604in some illustrative examples. The different illustrative examples can be implemented in a data processing system including components in addition to or in place of those illustrated for data processing system1600. Other components shown inFIG.16can be varied from the illustrative examples shown. The different examples can be implemented using any hardware device or system capable of running program code1618.

Illustrative examples of the disclosure may be described in the context of aircraft manufacturing and service method1700as shown inFIG.17and aircraft1800as shown inFIG.18. Turning first toFIG.17, an illustration of an aircraft manufacturing and service method is depicted in accordance with an illustrative example. During pre-production, aircraft manufacturing and service method1700may include specification and design1702of aircraft1800inFIG.18and material procurement1704.

During production, component and subassembly manufacturing1706and system integration1708of aircraft1800inFIG.18takes place. Thereafter, aircraft1800inFIG.18can go through certification and delivery1710in order to be placed in service1712. While in service1712by a customer, aircraft1800inFIG.18is scheduled for routine maintenance and service1714, which may include modification, reconfiguration, refurbishment, and other maintenance or service.

Each of the processes of aircraft manufacturing and service method1700may be performed or carried out by a system integrator, a third party, an operator, or some combination thereof. In these examples, the operator may be a customer. For the purposes of this description, a system integrator may include, without limitation, any number of aircraft manufacturers and major-system subcontractors; a third party may include, without limitation, any number of vendors, subcontractors, and suppliers; and an operator may be an airline, a leasing company, a military entity, a service organization, and so on.

With reference now toFIG.18, an illustration of an aircraft is depicted in which an illustrative example may be implemented. In this example, aircraft1800is produced by aircraft manufacturing and service method1700inFIG.17and may include airframe1802with plurality of systems1804and interior1806. Examples of systems1804include one or more of propulsion system1808, electrical system1810, hydraulic system1812, and environmental system1814. Any number of other systems may be included. Although an aerospace example is shown, different illustrative examples may be applied to other industries, such as the automotive industry.

Apparatuses and methods embodied herein may be employed during at least one of the stages of aircraft manufacturing and service method1700inFIG.17.

In one illustrative example, components or subassemblies produced in component and subassembly manufacturing1706inFIG.17can be fabricated or manufactured in a manner similar to components or subassemblies produced while aircraft1800is in service1712inFIG.17. As yet another illustrative example, one or more apparatus examples, method examples, or a combination thereof can be utilized during production stages, such as component and subassembly manufacturing1706and system integration1708inFIG.17. One or more apparatus examples, method examples, or a combination thereof may be utilized while aircraft1800is in service1712, during maintenance and service1714inFIG.17, or both. The use of a number of the different illustrative examples may substantially expedite the assembly of aircraft1800, reduce the cost of aircraft1800, or both expedite the assembly of aircraft1800and reduce the cost of aircraft1800.

The use of data manager210to identify outliers, causality for outliers, and perform retesting can reduce the time needed to certify parts are systems during certification and delivery1710. Further, the identification of outliers and causality for outliers can also be used to improve the quality of components during component and subassembly manufacturing1706.

Turning now toFIG.19, an illustration of a block diagram of a product management system is depicted in accordance with an illustrative example. Product management system1900is a physical hardware system. In this illustrative example, product management system1900includes at least one of manufacturing system1902or maintenance system1904.

Manufacturing system1902is configured to manufacture products, such as aircraft1800inFIG.18. As depicted, manufacturing system1902includes manufacturing equipment1906. Manufacturing equipment1906includes at least one of fabrication equipment1908or assembly equipment1910.

Fabrication equipment1908is equipment that used to fabricate components for parts used to form aircraft1800inFIG.18. For example, fabrication equipment1908can include machines and tools. These machines and tools can be at least one of a drill, a hydraulic press, a furnace, an autoclave, a mold, a composite tape laying machine, an automated fiber placement (AFP) machine, a vacuum system, a robotic pick and place system, a flatbed cutting machine, a laser cutter, a computer numerical control (CNC) cutting machine, a lathe, or other suitable types of equipment. Fabrication equipment1908can be used to fabricate at least one of metal parts, composite parts, semiconductors, circuits, fasteners, ribs, skin panels, spars, antennas, or other suitable types of parts.

Assembly equipment1910is equipment used to assemble parts to form aircraft1800inFIG.18. In particular, assembly equipment1910is used to assemble components and parts to form aircraft1800inFIG.18. Assembly equipment1910also can include machines and tools. These machines and tools may be at least one of a robotic arm, a crawler, a faster installation system, a rail-based drilling system, or a robot. Assembly equipment1910can be used to assemble parts such as seats, horizontal stabilizers, wings, engines, engine housings, landing gear systems, and other parts for aircraft1800inFIG.18.

In this illustrative example, maintenance system1904includes maintenance equipment1912. Maintenance equipment1912can include any equipment needed to perform maintenance on aircraft1800inFIG.18. Maintenance equipment1912may include tools for performing different operations on parts on aircraft1800inFIG.18. These operations can include at least one of disassembling parts, refurbishing parts, inspecting parts, reworking parts, manufacturing replacement parts, or other operations for performing maintenance on aircraft1800inFIG.18. These operations can be for routine maintenance, inspections, upgrades, refurbishment, or other types of maintenance operations.

In the illustrative example, maintenance equipment1912may include ultrasonic inspection devices, x-ray imaging systems, vision systems, drills, crawlers, and other suitable devices. In some cases, maintenance equipment1912can include fabrication equipment1908, assembly equipment1910, or both to produce and assemble parts that needed for maintenance.

Product management system1900also includes control system1914. Control system1914is a hardware system and may also include software or other types of components. Control system1914is configured to control the operation of at least one of manufacturing system1902or maintenance system1904. In particular, control system1914can control the operation of at least one of fabrication equipment1908, assembly equipment1910, or maintenance equipment1912.

The hardware in control system1914can be implemented using hardware that may include computers, circuits, networks, and other types of equipment. The control may take the form of direct control of manufacturing equipment1906. For example, robots, computer-controlled machines, and other equipment can be controlled by control system1914. In other illustrative examples, control system1914can manage operations performed by human operators1916in manufacturing or performing maintenance on aircraft1800. For example, control system1914can assign tasks, provide instructions, display models, or perform other operations to manage operations performed by human operators1916.

In these illustrative examples, data manager210fromFIG.2can be implemented in control system1914to manage at least one of the manufacturing or maintenance of aircraft1800inFIG.18. For example, data manager210can perform analysis of test data obtained during at least one of manufacturing or maintenance of a product. The test data can be for the product or parts forming product. This test data can be analyzed to determine whether outliers are present and the causes of those outliers. When the causes of the outliers of current through manufacturing or maintenance processes, changes can be made and the product can be retested. This retesting can include using changes to a measurement process or remanufacturing the product or part.

As result, the use of data manager210can provide increased quality in the product. Further, data manager210can also provide test data that can be used for various certification or compliance processes to meet regulations or manufacturing requirements.

In the different illustrative examples, human operators1916can operate or interact with at least one of manufacturing equipment1906, maintenance equipment1912, or control system1914. This interaction can occur to manufacture aircraft1800inFIG.18.

Of course, product management system1900may be configured to manage other products other than aircraft1800inFIG.18. Although product management system1900has been described with respect to manufacturing in the aerospace industry, product management system1900can be configured to manage products for other industries. For example, product management system1900can be configured to manufacture products for the automotive industry as well as any other suitable industries.

Some features of the illustrative examples are described in the following clauses. These clauses are examples of features not intended to limit other illustrative examples.

Clause 1

A method for managing a set of outliers in test data, the method comprising:analyzing, by a computer system, a set of features derived from the test data using different outlier detection methods to generate a result of the set of outliers identified by the different outlier detection methods, wherein the test data is obtained from testing a physical structure;determining, by the computer system, a causality for the set of outliers in the result; andretesting the physical structure with a set of changes determined using the causality identified for the set of outliers, wherein the retesting generates new test data for the physical structure.
Clause 2

The method according to clause 1 further comprising:detecting, by the computer system, outliers for the set of features using the different outlier detection methods; andgenerating, by the computer system, the result of the set of outliers detected by each outlier detection method in the different outlier detection methods.
Clause 3

The method according to one of clauses 1 or 2, wherein determining the causality for the set of outliers in the result comprises:identifying, by the computer system, a set of outlier types for an outlier in the result; anddetermining, by the computer system, the causality for the outlier using the set of outlier types identified for the outlier.
Clause 4

The method according to one of clauses 1, 2, or 3 further comprising:identifying, by the computer system, the set of features from the test data, wherein the set of features is selected from at least one of a standard feature, a transformed feature, or a selected feature.
Clause 5

The method according to one of clauses 1, 2, 3, or 4 wherein the different outlier detection methods are selected from two or more of a cosine similarity, a correlation analysis, a principal component analysis, a Sprague-Geers analysis, or a robust principal component analysis.

Clause 6

The method according to one of clauses according to one of clauses 1, 2, 3, 4, or 5, wherein retesting the physical structure with the set of changes comprises:retesting, by the computer system, the physical structure using a change to a measurement process in response to the causality indicating that the measurement process was a cause of an outlier in the set of outliers.
Clause 7

The method according to one of clauses 1, 2, 3, 4, 5, or 6, wherein retesting the physical structure with the set of changes comprises:manufacturing a new physical structure with the set of changes identified; andretesting the new physical structure.
Clause 8

The method according to clause 7, wherein the set of changes comprises at least one of, a measurement process change, a geometry change, a manufacturing parameter change, or a manufacturing process change.

Clause 9

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, or 8 further comprising:removing, by the computer system, noise from the set of features using at least one of removing the noise prior to analyzing the set of features derived from the test data using the different outlier detection methods or using an outlier detection method in the different outlier detection methods that removes the noise.
Clause 10

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, 8, or 9, wherein the result is a result matrix identifying outliers and outlier detection methods identifying the outliers.

Clause 11

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10, wherein the physical structure is selected from a group comprising a composite part, a test coupon, an assembly, a system, an alloy part, and a metal structure.

Clause 12

An outlier management system comprising:a computer system; anda data manager in the computer system, wherein the data manager is configured to:

analyze a set of features derived from test data using different outlier detection methods to generate a result of a set of outliers identified by the different outlier detection methods, wherein the test data is obtained from testing a physical structure;determine a causality for the set of outliers in the result; andretest the physical structure with a set of changes determined using the causality identified for the set of outliers, wherein the retesting generates new test data for the physical structure.
Clause 13

The outlier management system according to clause 12, wherein the data manager is configured to:detect outliers for the set of features using the different outlier detection methods; andgenerate the result of the set of outliers identified by each outlier detection method in the different outlier detection methods.
Clause 14

The outlier management system according to one of clauses 12 or 13, wherein in determining the causality for the set of outliers in the result, the data manager is configured to:identify a set of outlier types for an outlier in the result; anddetermine the causality for the outlier using the set of outlier types identified for the outlier.
Clause 15

The outlier management system according to one of clauses 12, 13, or 14, wherein the data manager is configured to:identify, by the computer system, the set of features from the test data, wherein the set of features is selected from at least one of a standard feature, a transformed feature, or a selected feature.
Clause 16

The outlier management system according to one of clauses 12, 13, 14, or 15, wherein the different outlier detection methods are selected from two or more of a cosine similarity, a correlation analysis, a principal component analysis, a Sprague-Geers analysis, or a robust principal component analysis.

Clause 17

The outlier management system according to one of clauses 12, 13, 14, 15, or 16, wherein in retesting the physical structure with the set of changes, the data manager is configured to:

retest the physical structure using a change to a measurement process in response to the causality indicating that the measurement process was a cause of an outlier in the set of outliers.

Clause 18

The outlier management system according to one of clauses 12, 13, 14, 15, 16, or 17, wherein in retesting the physical structure with the set of changes, the data manager is configured to:manufacture a new physical structure with the set of changes identified; andretest the new physical structure.
Clause 19

The outlier management system according to clause 18, wherein the set of changes comprises at least one of a measurement process change, a geometry change, a manufacturing parameter change, or a manufacturing process change.

Clause 20

The outlier management system according to one of clauses 12, 13, 14, 15, 16, 17, 18, or 19, wherein data manager is configured to:remove noise from the set of features using at least one of removing the noise prior to analyzing the set of features derived from the test data using the different outlier detection methods or using an outlier detection method in the different outlier detection methods that removes the noise.
Clause 21

The outlier management system according to one of clauses 12, 13, 14, 15, 16, 17, 18, 19, or 20, wherein the result is a result matrix identifying outliers and outlier detection methods identifying the outliers.

Clause 22

The outlier management system according to one of clauses 12, 13, 14, 15, 16, 17, 18, 19, 20, or 21, wherein the physical structure is selected from a group comprising a composite part, an assembly, a test coupon, a system, an alloy part, and a metal structure.

Clause 23

A method for managing a set of outliers in test data, the method comprising:analyzing, by a computer system, a set of features derived from the test data using different outlier detection methods to generate a result of the set of outliers identified by the different outlier detection methods, wherein the test data is obtained from testing a physical structure;removing, by the computer system, the set of outliers from the features; andvalidating, by the computer system, a simulation model with the outlier removed from the set of features.
Clause 24

A computer program product for managing a set of outliers in test data, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer system to cause the computer system to perform a method of:analyzing, by a computer system, a set of features derived from the test data using different outlier detection methods to generate a result of the set of outliers identified by the different outlier detection methods, wherein the test data is obtained from testing a physical structure;determining, by the computer system, a causality for the set of outliers in the result; andretesting the physical structure with a set of changes determined using the causality identified for the set of outliers, wherein the retesting generates new test data for the physical structure.

Thus, a method, apparatus, system, and computer program product for managing outliers is provided. In one illustrative example, a computer system analyzes a set of features derived from the test data using different outlier detection methods to generate a result of a set of outliers identified by the different outlier detection methods. The test data is obtained from testing a physical structure. The computer system determines a causality for the set of outliers in the result. The physical structure is retested with a set of changes determined using the causality identified for the set of outliers. The retesting generates new test data for the physical structure.

The different processes illustrated in the flowcharts can be used to analyze test data in smaller amounts as compared to currently available outlier detection systems. By analyzing the test data using different outlier detection methods, the likelihood of more outliers or outlier types can be identified.

Further, in the different illustrative examples the original test data can be tested as standard features without changes, selected features in which portions of the test data selected or is transformed data. With transformed data, the original test data is transformed based on the theory governing physics of issues or problems identified for underlying features in physical structures. Different outlier detection methods may detect outliers using different types of test data selected from at least one of standard features, transform features, were selected features. The selected features can also include selected features from transformed test data.

As a result, these features in the different illustrative examples can more efficiently detect more types of outliers as compared to current systems. Further, noise can also be removed. As a result, the test data processed using the illustrative example can reveal the underlying true response of a physical structure more easily as compared to current techniques.

The description of the different illustrative examples has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the examples in the form disclosed. The different illustrative examples describe components that perform actions or operations. In an illustrative example, a component can be configured to perform the action or operation described. For example, the component can have a configuration or design for a structure that provides the component an ability to perform the action or operation that is described in the illustrative examples as being performed by the component. Further, To the extent that terms “includes”, “including”, “has”, “contains”, and variants thereof are used herein, such terms are intended to be inclusive in a manner similar to the term “comprises” as an open transition word without precluding any additional or other elements.

Many modifications and variations will be apparent to those of ordinary skill in the art. Further, different illustrative examples may provide different features as compared to other desirable examples. The example or examples selected are chosen and described in order to best explain the principles of the examples, the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various examples with various modifications as are suited to the particular use contemplated.