Patent Description:
There is a desire within any manufacturing environment to reduce the risk associated with manufacturing processes, especially processes that involve human steps or interactions. Such a reduction in the risk should result in a reduction in defects and non-conformances in the products produced by the manufacturing processes.

For example, in the manufacture of circuit card assemblies (CCAs), it is desirable to reduce the percentage of defects in the finished product. This is presently accomplished in several ways. A trained operator can examine the product and determine if there are any non-conformances in the finished product. The trained operator can use one or more pieces of equipment to aid in this examination, such as an x-ray machine. However, some non-conformances can be missed because the amount of CCAs produced by a production line in a day can be quite large, and the operator can become fatigued. Additionally, current examination procedures do not have any aggregation of data relating to failures that can be used by the operator.

<CIT> discloses a computer implemented method for surface defect inspection that includes recording an optical image of a surface including a defect; converting the optical image including the defect into a heat map; extracting a region of interest including the defect from the heat map; and comparing the region of interest including the defect from the heat map to a binary classification model using a sliding window based voting mechanism to determine if the defect is greater than or less than a threshold failure value.

<CIT> discloses a method, apparatus, and system for visualizing nonconformance data for a physical object. An augmented reality application in a portable computing device plots, in a defined coordinate cube, points corresponding to nonconformance locations on the physical object. The augmented reality application determines a sub-set of the points plotted that correspond to a region of the physical object visible in an image of the region of the physical object acquired by the portable computing device at a position of the portable computing device, where the sub-set of the points exclude nonconformance locations occluded from view by a physical object structure of the physical object in the image. The augmented reality application displays the nonconformance data for the sub-set of the points visible in the image in association with a sub-set of the nonconformance locations for the physical object in the image displayed on a display system in the portable computing device.

In an aspect, the present disclosure provides a process comprising: receiving into an augmented reality, AR, device an image of an object, wherein the object comprises a manufactured good, and wherein the manufactured good comprises a printed circuit board; receiving information for identifying the object or a component associated with the object; accessing a database to retrieve data relating to the object or the component associated with the object, wherein the data retrieved from the database relating to the object or the component associated with the object comprises a failure history of the object or the component associated with the object; creating a heat map as a function of the data relating to the object or the component associated with the object, wherein the heat map comprises visual non-conformances on the printed circuit board; displaying the heat map on a computer display device; and locating one or more fiducials associated with the object or the component associated with the object, wherein the one or more fiducials comprise an other object that is placed in a field of view of the AR device such that the other object comprises a point of reference or measure in the field of view of the AR device.

In another aspect, the present disclosure provides a non-transitory computer readable medium comprising instructions that when executed by a computer processor executes a process comprising: receiving into an augmented reality, AR, device an image of an object, wherein the object comprises a manufactured good, and wherein the manufactured good comprises a printed circuit board; receiving information for identifying the object or a component associated with the object; accessing a database to retrieve data relating to the object or the component associated with the object, wherein the data retrieved from the database relating to the object or the component associated with the object comprises a failure history of the object or the component associated with the object; creating a heat map as a function of the data relating to the object or the component associated with the object, wherein the heat map comprises visual non-conformances on the printed circuit board; displaying the heat map on a computer display device; and locating one or more fiducials associated with the object or the component associated with the object, wherein the one or more fiducials comprise an other object that is placed in a field of view of the AR device such that the other object comprises a point of reference or measure in the field of view of the AR device.

In another aspect, the present disclosure provides a system comprising: an augmented reality device; and a computer database coupled to the augmented reality device; wherein the system is operable for: receiving into the augmented reality device an image of an object, wherein the object comprises a manufactured good, and wherein the manufactured good comprises a printed circuit board; receiving information for identifying the object or a component associated with the object; accessing the computer database to retrieve data relating to the object or the component associated with the object, wherein the data retrieved from the database relating to the object or the component associated with the object comprises a failure history of the object or the component associated with the object; creating a heat map as a function of the data relating to the object or the component associated with the object, wherein the heat map comprises visual non-conformances on the printed circuit board; displaying the heat map on a computer display device; and locating one or more fiducials associated with the object or the component associated with the object, wherein the one or more fiducials comprise an other object that is placed in a field of view of the AR device such that the other object comprises a point of reference or measure in the field of view of the AR device.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various aspects of different embodiments of the present invention. It will be evident, however, to one skilled in the art, that the present invention may be practiced without all the specific details and/or with variations, permutations, and combinations of the various features and elements described herein.

An embodiment of the present disclosure relates to leveraging object character recognition (OCR) capabilities in supporting manufacturing operations. The embodiment can be implemented on a mobile device. The embodiment can be linked with risk and opportunity management processes (risk-probability matrix, waterfall charts, etc.). A full digital make-up of the manufacturing operations can be created, and past, present, and future data on manufacturing (assembly, inspection, test procedures) can be displayed to a person involved in the manufacturing operation on any device. The embodiment can leverage visualization techniques to provide heat maps, and risk and opportunity waterfall charts, to assist in manufacturing or maintenance procedures. These heat maps can illustrate high probability areas for failure in the manufacturing process. Additionally, an embodiment can provide overlays or display high level data for manufacturing and maintenance environments.

In real time, data aggregation techniques (waterfall and probability maps) can be visually represented on computer card assemblies (CCA) and other hardware and manufactured products (such as automobiles, planes, cabinets, and additive manufactured products) to assist in augmentation in an augmented reality (AR) environment for work instructions, test assessments/troubleshooting, and inspection operations. Manufacturing real time data and historical data can be leveraged to provide heat maps of risks and opportunities within the non-conformances. For example, an inspection task can highlight high risk areas that have failed within the most recently manufactured units. A test engineer can then access data relating to the most recently failed components or parts to help provide real updates on past data. Visualization of these data can be shown on a mobile device or other devices.

<FIG> is a block diagram of an embodiment of a system for determining non-conformance of an object. An AR system <NUM> works in conjunction with a manufacturing system <NUM>. The system includes a database <NUM>, which includes data such as a history of the manufacturing processes and the products manufactured by the manufacturing processes. As noted above, these data are used in the AR system to identify historical non-conformances in the manufacturing process. The AR system <NUM> can further include an Internet of Things (IOT) platform <NUM>. For example, the manufactured good itself can have Internet-access capabilities, and these capabilities can be used to retrieve data over the Internet and share data over the Internet. At <NUM>, using the database <NUM> and/or the IOT platform <NUM>, information is provided about the manufactured good such as maintenance procedures, installation procedures, replacement procedures, assembly procedures, inspection procedures, and test procedures. As indicated at <NUM>, this information can be provided to manufacturing personnel via an AR headset <NUM>, a mobile device <NUM>, and/or some type of computer display device <NUM>. At <NUM>, information relating to the actual maintenance procedures, installation procedures, replacement procedures, assembly procedures, inspection procedures, and test procedures that were performed on the manufactured good are captured at <NUM> and provided as feedback in the AR system <NUM>.

<FIG> illustrates an example of a visualization. In <FIG>, the system displays components <NUM> and <NUM>, and several fasteners <NUM> to couple the components <NUM> and <NUM>. The visualization can further include instructions such as a fastening pattern and the amount of torque to be applied to the fasteners. As indicated in <FIG>, the system <NUM> permits manufacturing personnel to consider captured data at <NUM>, to visualize the maintenance procedures, installation procedures, replacement procedures, assembly procedures, inspection procedures, and test procedures at <NUM>, and to make any needed adjustments via the maintenance procedures, installation procedures, replacement procedures, assembly procedures, inspection procedures, and test procedures at <NUM>.

<FIG> is an example of a risk waterfall chart <NUM>. As time proceeds and the AR system <NUM> is used to identify and react to non-conformances, the severity levels of the non-conformances fail from highly severe <NUM>, to moderately severe <NUM>, to a low severity <NUM>. The severities can be color-coded, such as red for highly severe <NUM>, yellow for moderately severe <NUM>, and green for the low severity <NUM>. <FIG> is an example of an opportunity waterfall chart <NUM>. As time proceeds and the AR system <NUM> is used to identify and react to non-conformances, new opportunities are taken. However, with these new opportunities, over time, the severity level rises from a low severity level at <NUM>, to a moderate severity level at <NUM>, to a high severity level at <NUM>. Like with the risk waterfall chart <NUM>, the severities can be color-coded in the opportunity waterfall chart <NUM>, such as red for highly severe <NUM>, yellow for moderately severe <NUM>, and green for the low severity <NUM>.

<FIG> is an example of a probability-consequence chart <NUM>. The probability-consequence chart is used to determine the risk factor for the risk waterfall chart <NUM>. For example, if there is a low probability of occurrence of an event, e.g., a <NUM> at <NUM>, and a low consequence if that event occurs, e.g., a <NUM> at <NUM>, then the risk factor is a relatively low <NUM> (i.e., the product of the two). However, if there is a high probability of occurrence of an event, e.g., a <NUM> at <NUM>, and a high consequence if that event occurs, e.g., a <NUM> at <NUM>, then the risk factor is a relatively high <NUM>. <FIG> illustrates examples of ranges that can be used to determine these risk factors.

<FIG> is an example of a benefit-feasibility chart. The benefits-feasibility chart <NUM> is used to determine the opportunity factor for the opportunity waterfall chart <NUM>. For example, if there is a low benefit associated with the occurrence of an event, e.g., a <NUM> at <NUM>, and a low consequence if that event occurs, e.g., a <NUM> at <NUM>, then the opportunity factor is a relatively low <NUM>. However, if there is a high benefit associated with the occurrence of an event, e.g., a <NUM> at <NUM>, and a high consequence if that event occurs, e.g., a <NUM> at <NUM>, then the opportunity factor is a relatively high <NUM>. <FIG> illustrates examples of ranges that can be used to determine these opportunity factors.

<FIG> are examples of heat maps <NUM> and <NUM> that are associated with a printed circuit board. The heat maps <NUM> and <NUM> are a way to visualize trends in manufacturing data historically. For example, if manufacturing personnel are actively inspecting a particular type of board over a time period, they may be able to see as the color shifts from red to yellow indicating the manufacturing quality is moving in the right direction for that particular board or product line. But if the manufacturing person starts up a new run from the day before or takes over from a prior shift, that person, without the benefits of the heat map, won't have any reference for how that part has performed up to that point in time. With an embodiment, the person has the ability of showcasing the index/count that they are inspecting, and they can scroll through data based on different snapshots of the board window, and they can understand how the heat map of the board has progressed. This provides opportunities for manufacturing leadership to use heat maps to understand the trend rates of their hardware, to understand and use learning curves, and to better train their employees.

Referring specifically to <FIG>, in <FIG>, component <NUM> is outlined in two boxes, indicating a color of yellow, which indicates a severity history of moderate, and <FIG> illustrates that component <NUM> still has a moderate severity because it is still enclosed in a yellow box (as indicated by the two boxes). In <FIG>, component <NUM> is outlined in three boxes, indicating a color of red, which indicates a severity history of high, and <FIG> illustrates that component <NUM> has progressed in the proper direction as it is now outlined in a yellow box (two boxes) indicating moderate severity. In <FIG>, component <NUM> is outlined in two boxes, indicating a color of yellow, which indicates a severity history of moderate, and <FIG> illustrates that component <NUM> is now outlined in a single box, indicating a color of green, which indicates that component <NUM> has improved to a low severity. These progressions indicating a worsening severity and a lessening severity will catch the attention of manufacturing personnel. <FIG> further illustrate an input device <NUM>, such as a slider, which permits a user such as manufacturing personnel to select a particular object or a particular component of the object on a computer display device. This selection can be used to access a database and retrieve the information about the particular object or the particular component. The input device <NUM> can further be used to select, for example, the part number of the manufactured good and a specific quantity or date range of when the good was manufactured.

<FIG> is a flowchart of an example embodiment of a process for generating a heat map for an object. <FIG> includes a number of process blocks <NUM> - <NUM>. Though arranged substantially serially in the example of <FIG>, other examples may reorder the blocks, omit one or more blocks, and/or execute two or more blocks in parallel using multiple processors or a single processor organized as two or more virtual machines or sub-processors. Moreover, still other examples can implement the blocks as one or more specific interconnected hardware or integrated circuit modules with related control and data signals communicated between and through the modules. Thus, any process flow is applicable to software, firmware, hardware, and hybrid implementations.

Referring specifically now to <FIG>at <NUM>, an image of an object is received into an augmented reality device. As indicated at <NUM>, the augmented reality device can be an off the shelf head set, a computer tablet, or some other mobile device. As further indicated at <NUM>, the object can be a manufactured good, and more particularly, the object can be a printed circuit board (<NUM>). When the object is a printed circuit board, the heat map can include visual non-conformances on the printed circuit board.

At <NUM>, information is received for identifying the object or a component associated with the object. At <NUM>, one or more fiducials that are associated with the object or the component of the object are located using OCR. In this context, a fiducial is another object that is placed in the field of view of the AR device, and this other object appears in the view of the AR device. This other object can be used as a point of reference or measure. As indicated at <NUM>, the information is received via one or more of object character recognition (OCR), text, barcode, quick response (QR) code, human input, or radio frequency identification (RFID). And as further indicated at <NUM>, the AR device can sense radio frequency identification (RFID) information associated with the object or the component associated with the object.

At <NUM>, a database is accessed to retrieve data relating to the object or the component that is associated with the object. As indicated at <NUM>, the data retrieved from the database relating to the object or the component associated with the object is a failure history of the object or the component associated with the object. The data retrieved from the database can also relate to such things as a maintenance procedure for the object or the component associated with the object, an installation procedure for the object or the component associated with the object, a replacement procedure for the object or the component associated with the object, an assembly procedure for the object or the component associated with the object, an inspection procedure for the object or the component associated with the object, and a test procedure for the object or the component associated with the object (<NUM>).

At <NUM>, a heat map is created as a function of the data relating to the object or the component associated with the object, and at <NUM>, the heat map is displayed on a computer display device. As noted above, such a computer display device can be an AR headset, a tablet, a laptop computer, or some other computer device. The display can be an overlay of the heat map onto the object (<NUM>). For example, as discussed above, the overlay of the heat map on a printed circuit board can identify components that have recently been experiencing problems, and the overlay can further indicate in which direction the problems associated with the component are trending. In addition to a heat map overlay, the display on the computer display device can include a risk chart, an opportunity waterfall chart, and a probability matrix relating to the object or the component associated with the object (<NUM>).

At <NUM>, a selection of a particular object or a particular component of the object is received from manufacturing personnel. This selection is used to access the database and retrieve the information about the particular object or the particular component. In an embodiment, this feature can be implemented using a slider device on a computer display unit. The slider is used to select, for example, the part number of the manufactured good and a specific quantity or date range of when the good was manufactured.

At <NUM>, a selection of one or more colors is received from manufacturing personnel for use on the heat map. This feature is particularly useful for persons who are color-blind to certain colors. This feature, like the feature of operation <NUM>, can be implemented via a slider device on a computer display unit. In another embodiment, color is not used as an indicator on the computer display unit, but different shapes such as circles, triangles, and squares.

Claim 1:
A process comprising:
receiving (<NUM>) into an augmented reality, AR, device (<NUM>, <NUM>, <NUM>) an image of an object, wherein the object comprises a manufactured good, and wherein the manufactured good comprises a printed circuit board;
receiving (<NUM>) information for identifying the object or a component associated with the object;
accessing (<NUM>) a database (<NUM>) to retrieve data relating to the object or the component associated with the object, wherein the data retrieved from the database relating to the object or the component associated with the object comprises a failure history of the object or the component associated with the object;
creating (<NUM>) a heat map as a function of the data relating to the object or the component associated with the object, wherein the heat map comprises visual non-conformances on the printed circuit board;
displaying (<NUM>) the heat map on a computer display device; and
locating (<NUM>) one or more fiducials associated with the object or the component associated with the object, wherein the one or more fiducials comprise an other object that is placed in a field of view of the AR device such that the other object comprises a point of reference or measure in the field of view of the AR device.