Patent Publication Number: US-2023135476-A1

Title: Defect detection of a component in an assembly

Description:
RELATED APPLICATION 
     The present application is a continuation application of U.S. Nonprovisional Pat. Application No. 16/882,198, filed May 22, 2020, issued as U.S. Pat. No. 11,538,148 on Dec. 27, 2022, and entitled “Defect Detection of a Component in an Assembly,” the entire disclosure of which application is incorporated herein by reference. 
    
    
     FIELD OF THE TECHNOLOGY 
     At least some embodiments disclosed herein relate generally to component installation validation. More particularly, the embodiments relate to artificial intelligence systems and methods for computer-aided validation of the installation of a component in an assembly such as a computing device. 
     BACKGROUND 
     Device manufacturing, such as mobile devices (e.g., smartphones, tablets, smartwatches, or the like) utilize several components assembled together. The assembly process can include, for example, securing the components together via fasteners (e.g., screws or the like). The assembly process, if not completed properly (e.g., missing screws, incorrect screws, improperly tightened screws, or the like), can cause quality control issues. 
     Improved methods and systems for validating the proper installation of the various components are desirable. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       References are made to the accompanying drawings that form a part of this disclosure and illustrate embodiments in which the systems and methods described in this Specification can be practiced. 
         FIG.  1    shows a system for validation of installation of a component in an assembly, according to an embodiment. 
         FIG.  2    shows a portion of the system for validation of installation of a component of an assembly of  FIG.  1   , according to an embodiment. 
         FIG.  3    shows a schematic architecture for the system of  FIG.  1   , according to an embodiment. 
         FIG.  4    shows a flowchart for a method of validating installation of a component in an assembly, according to an embodiment. 
         FIG.  5    shows a first graphical user interface (GUI), according to an embodiment. 
         FIG.  6    shows a second GUI, according to an embodiment. 
         FIG.  7    shows a third GUI, according to an embodiment. 
       Like reference numbers represent like parts throughout. 
     
    
    
     DETAILED DESCRIPTION 
     Computing devices such as, but not limited to, smartphones, tablets, laptops, smartwatches, and the like, include numerous components that are assembled together. The assembly process can include fasteners (e.g., screws or the like) that keep the various components secured. It is important that these fasteners be installed correctly (e.g., all screws installed (e.g., no missing screws), proper screws installed, screws properly tightened, or the like) as part of the quality control process. 
     The embodiments disclosed herein are directed to a system and method for inspecting components (e.g., fasteners such as screws or the like) of an assembly (e.g., a computing device such as, but not limited to, a smartphone, a tablet, a laptop, a smartwatch, a cellphone, or the like) during the manufacturing process. The system includes a platform that holds the test assembly (i.e., the assembly being inspected) in a defined position. The defined position is established relative to a plurality of cameras that are arranged to capture different views of the assembly. 
     When an assembly (e.g., a computing device) is placed onto the platform, an image of the assembly is captured from each of the plurality of cameras. The plurality of cameras are triggered to capture each of the images at the different viewing angles in response to a combination of proximity and motion sensors that identify when the assembly is in place. “Profile images” (i.e., images of a model assembly (e.g., a model assembly (i.e., screws properly installed)) can be captured in a calibration process. The platform maintains the assembly in a generally fixed location. As a result, each image of an assembly being validated (i.e., a test assembly) is taken in the same coordinate system or with a predetermined relationship maintained by the platform relative to the corresponding profile image. 
     Each of the captured images is compared against the corresponding profile image to determine a matching score representative of whether the components were correctly installed. 
     In an embodiment, the different viewing angles can provide a more accurate response, as the installation defect may be noticeable in one of the views but appear correct (or not be noticeable) in another of the views. The matching score is based on the comparison of each view. In response to the determination, an indicator (correct, incorrect, requires further checking) including one of the profile images is then displayed on an interface of the system. 
     A system is disclosed. The system includes a housing, enclosing a platform having a tiered surface. The tiered surface includes a first surface and a second surface parallel to the first surface and disposed a height from the first surface. The tiered surface forms an abutment surface configured as a stop against which a test assembly is abutted. A plurality of cameras are enclosed in the housing, each of the plurality of cameras positioned to capture a different view of the test assembly. A processing device is configured to execute instructions to capture an image from each of the plurality of cameras of the test assembly that includes a plurality of components. Each of the plurality of components within each image of the plurality of images is analyzed. A matching score is determined between a corresponding profile image of each of the plurality of components within each of the plurality of images of the test assembly. The installation of each of the plurality of components is classified based on the matching score. An indication of whether the plurality of components was correctly installed in the test assembly is generated based on the matching score. A display device is configured to display a graphical user interface (GUI) based on the indication as generated. 
     A method for validating correctness of installation of a component in an assembly is also disclosed. The method includes capturing a plurality of images using a plurality of cameras. The plurality of cameras are oriented to capture different views of a test assembly including a component. A processing device analyzes the component within each image of the plurality of images. The processing device determines a matching score between a profile image of the component within each corresponding profile image and the plurality of captured images. The component installation is classified based on the matching score. An indication of whether the component was correctly installed in the test assembly based on the matching score is generated. 
       FIG.  1    shows a system  10  for validation of installation of a component in an assembly  15 , according to an embodiment. The system  10  can generally be used to, for example, validate whether a fastener (e.g., a screw or the like) or other component is properly installed in the assembly  15 . In an embodiment, the validation can be part of a quality control process during manufacturing. In an embodiment, the system  10  can be used to validate whether the fastener or other component is properly installed at a point after manufacturing (e.g., during refabricating, maintenance, or the like). 
     In the illustrated embodiment, the assembly  15  is a smartphone. It is to be appreciated that the smartphone is an example, and the assembly  15  can vary beyond a smartphone. Examples of other assemblies  15  include, but are not limited to, a tablet, a smartwatch, a mobile phone other than a smartphone, a personal digital assistant (PDA), a laptop computing device, or the like. Furthermore, the maker or manufacturer of the assembly  15  is not limited. That is, the system  10  can be used to validate the installation correctness of components in assemblies  15  from different manufacturers so long as a calibration procedure is performed to create a profile image for the corresponding assembly  15 . 
     The system  10  includes a display  20  for displaying results of the validation to the user. In an embodiment, the display  20  can be a combined display and input (e.g., a touchscreen). In an embodiment, the display  20  can be a display of a tablet or the like. In such an embodiment, a memory of the tablet can store one or more programs to be executed by a processing device of the tablet for validating the correctness of the installation of the component in the assembly  15 . 
     In the illustrated embodiment, the display  20  is secured to housing  25  of the system  10 . In an embodiment, the display  20  can be separate from the housing  25  (i.e., not secured to the housing  25 , but positioned near the system  10  and electronically connected to the system  10 ). However, it may be beneficial to secure the display  20  to the housing  25  to reduce a footprint of the system  10 . 
     A platform  30  is utilized to position the assembly  15  within the system  10  for validation. The platform  30  enables each assembly  15  placed into the system  10  for validation to be placed in substantially the same location. As a result, an amount of effort in determining whether the profile image and the assembly  15  under test (test assembly) is in a same location relative to cameras of the system  10  can be reduced. The platform  30  is shown and described in additional detail in accordance with  FIG.  2    below. 
     In an embodiment, the system  10  can be portable. For example, the illustrated embodiment shows system  10  with a handle  35  for carrying the system  10 . It is to be appreciated that portability of the system  10  is optional, and accordingly, the handle  35  is optional. In an embodiment, the system  10  may be sized differently based on the type of assembly  15  to be validated. 
       FIG.  2    shows the platform  30  of the system  10  of  FIG.  1    for validation of installation of a component in an assembly  15 , according to an embodiment. 
     The platform  30  includes a tiered surface having a first surface  40  and a second surface  45 . A step is thus formed between the first surface  40  and the second surface  45 . A plane of the first surface  40  and a plane of the second surface  45  are parallel. In the illustrated embodiment, the second surface  40  is L-shaped when viewed from a top view. 
     The second surface  45  is positioned a height H from the first surface  40 . The height H between the first surface  40  and the second surface  45  creates an abutment surface  50 . 
     The height H is selected such that the abutment surface  50  serves as a stop for the assembly  15  when placed within the system  10 . The abutment surface  50  is configured to provide a stop for the assembly  15  on two sides of the assembly  15  (i.e., a major dimension of the assembly  15  and a minor dimension of the assembly  15 ). 
     The height H is selected to be smaller than a thickness T of the assembly  15  being validated in the system  10 . The height H is selected to be smaller than the thickness T of the assembly  15  to not hinder side views of the assembly  15 . The height H is selected to be large enough that an operator inserting the assembly  15  can abut the assembly  15  with the abutment surface  50 . In this manner, the abutment surface  50  serves as a stop for the operator when inserting the assembly  15  into the system  10 . In an embodiment, the height H can be substantially the same as the thickness T of the assembly  15 . 
     The configuration of the platform  30  is helpful in establishing the location of the assembly  15 . By including the platform  30 , the system  10  can be calibrated to generate the profile images using a single assembly since the coordinate system is generally fixed. The platform  30  can, as a result, be used to account for minor variations in placement of the assembly  15  by the operator as the offset from the expected coordinated system can be determined based on the location of the assembly  15  relative to a calibration assembly  15 . 
       FIG.  3    shows a schematic architecture for the system  10  of  FIG.  1   , according to an embodiment. 
     The system  10  generally includes a plurality of cameras  100 ; a motion sensor  105 ; a proximity sensor  110 ; a processing device  115 , memory  120 , a network input/output (I/O)  125 , user I/O  130 , storage  135 , and an interconnect  140 . The processing device  115 , memory  120 , network input/output (I/O)  125 , user I/O  130 , storage  135 , and interconnect  140  can be within the housing  25  in an embodiment. In an embodiment, the processing device  115 , memory  120 , network input/output (I/O)  125 , user I/O  130 , storage  135 , and interconnect  140  can be external from the housing  25 . 
     The plurality of cameras  100  are arranged in the system  10  to capture different views of the assembly  15 . In an embodiment, the cameras  100  are digital cameras. For example, in an embodiment the system  10  includes three cameras  100  arranged to capture a top view, an up-front view, and an up-side view. In an embodiment, the system  10  includes four cameras  100  arranged to capture a top view, an up-front view, a first up-side view, and a second (opposite) up-side view. It will be appreciated that a single camera  100  could be used, although accuracy may be improved when a plurality of cameras  100  are used as a component may appear to be correctly installed in a first view but be determined to be incorrectly installed in a second view. 
     The motion sensor  105  can be, for example, a laser sensor that can be triggered when an object (i.e., assembly  15 ) breaks the laser signal. The motion sensor  105  can be installed at the opening to the housing  25 . In an embodiment, the motion sensor  105  may not be included. 
     The proximity sensor  110  can be a sensor to determine when an object is placed near it. The proximity sensor  110  can be placed in the platform  30  of the system  10 . In an embodiment, when the motion sensor  105  is triggered and the proximity sensor  110  detects an object, the cameras  100  can capture images of the assembly  15  on the platform  30 . In an embodiment, the proximity sensor  110  can be included regardless of whether the motion sensor  105  is present. In an embodiment with both motion sensor  105  and proximity sensor  110 , the image capturing may be performed after the proximity sensor  110  detects the assembly  15 . 
     In an embodiment, automatically causing the image capturing and subsequent validation to be performed using the proximity sensor  110 , or a combination of the proximity sensor  110  and the motion sensor  105 , can increase a number of assemblies  15  that can be validated in a set period. That is, reducing effort of a human operator, or even allowing for a robotic arm to load the assembly  15  into the system  10  for validation, can reduce an amount of time and effort needed to review the quality of the manufacturing process. 
     The processing device  115  can retrieve and execute programming instructions stored in the memory  120 , the storage  135 , or combinations thereof. The processing device  115  can also store and retrieve application data residing in the memory  120 . The programming instructions can perform the method described in accordance with  FIG.  4    below to determine whether the components of the assembly  15  are properly installed, and additionally, cause display of one of the graphical user interfaces (GUIs) shown and described in accordance with  FIGS.  5 - 7    below. 
     The interconnect  140  is used to transmit programming instructions and/or application data between the processing device  115 , the user I/O  130 , the memory  120 , the storage  135 , and the network I/O  125 . The interconnect  140  can, for example, be one or more busses or the like. The processing device  115  can be a single processing device, multiple processing devices, or a single processing device having multiple processing cores. In an embodiment, the processing device  115  can be a single-threaded processing device. In an embodiment, the processing device  115  can be a multi-threaded processing device. 
     The memory  120  is generally included to be representative of a random-access memory such as, but not limited to, Static Random-Access Memory (SRAM), Dynamic Random-Access Memory (DRAM), or Flash. In an embodiment, the memory  120  can be a volatile memory. In an embodiment, the memory  120  can be a non-volatile memory. In an embodiment, at least a portion of the memory  120  can be virtual memory. 
     The storage  135  is generally included to be representative of a non-volatile memory such as, but not limited to, a hard disk drive, a solid-state device, removable memory cards, optical storage, flash memory devices, network attached storage (NAS), or connections to storage area network (SAN) devices, or other similar devices that may store non-volatile data. In an embodiment, the storage  135  is a computer readable medium. In an embodiment, the storage  135  can include storage that is external to the user device, such as in a cloud. 
       FIG.  4    shows a flowchart for a method  200  of validating installation of a component in a test assembly  15  using artificial intelligence, according to an embodiment. 
     At block  205  a test assembly  15  is loaded into the system  10 . This includes abutting the assembly  15  with the abutment surface  50  of the platform  30 . In an embodiment, the test assembly  15  can be loaded by a human operator. In an embodiment, a robotic or mechanical arm can be automated to place the test assembly  15  onto the platform  30 . The placement of the test assembly  15  can cause the motion sensor  105 , the proximity sensor  110 , or a combination thereof, to generate a signal indicative of the assembly  15  being in place. 
     At block  210 , in response to the signal generated by the motion sensor  105 , the proximity sensor  110 , or a combination thereof, the plurality of cameras  100  each capture an image. As discussed above, the cameras  100  are oriented such that the captured images are of different views of the test assembly  15 . 
     At block  215 , each component within each of the captured images is compared against the corresponding component in the corresponding profile image. In an embodiment, the comparison can include, for each component location, comparing multiple portions of the captured image around the component. For example, if a center of the component is a point (x, y) in the profile image, a first portion would be at (x, y), another at (x+1, y+1), another at (x-1, y-1), etc. In this manner, a matching score can be computed for each portion of the captured image relative to the profile image. In an embodiment, the matching can be computed using a similarity metric such as, but not limited to, correlation coefficient score. A sampling of 50-500 points around (x, y) can be performed to provide a higher accuracy in making the validation. This improvement in accuracy can, for example, be because the multiple samples can compensate for minor displacements (e.g., 0-20 pixels or 1-2 mm in physical space) of the components being tested relative to the components in the profile image. Thus, the sampling can account for minor variations in the placement of the assembly  15  at block  205 . 
     The highest matching score among the portions of the captured image can be considered the score for the component being examined. It is to be appreciated that other statistical values could be used for the matching score, such as, but not limited to, the mean, median, or mode. 
     In performing the similarity comparison, several bases for determining a component is incorrect can be captured. For example, the validation can identify missing components, improperly installed components (e.g., wrong screw or the like), or installations that are not complete (e.g., a screw is not tightened or is over-tightened) based on orientation of the component. It is to be appreciated that the matching score will be impacted more heavily by a missing or incorrect component than by a misoriented component. The lowest matching score for each component is compared across the plurality of images and is retained as the matching score for that component. 
     At block  220  an output is generated indicative of the results of the validation (e.g., correct, incorrect, needs checking). The output can be based on a range of the matching score. That is, if the matching score is greater than a first value, then the output can be that the installation is correct; between the first value and a lower second value, the installation may need checking; and between the lower second value and a third value that is lower than the second value, the installation may be incorrect. 
     At block  225 , the output is displayed on the display  20  of the system  10 . In an embodiment, the display can include a notation of which components are incorrect or need checking. In an embodiment, to reduce processing effort and increase a speed of the output, the display can include one of the profile images (instead of one of the captured images for the test assembly  15 ). 
       FIG.  5    shows a first graphical user interface (GUI)  250 , according to an embodiment. The first GUI  250  is generally representative of a GUI displayed when the manufacturing component was validated and determined to be correctly installed. 
       FIG.  6    shows a second GUI  300 , according to an embodiment. The second GUI  300  is generally representative of a GUI displayed when the manufacturing component was incorrectly installed. 
       FIG.  7    shows a third GUI  350 , according to an embodiment. The third GUI  350  is generally representative of a GUI displayed when the manufacturing component was determined to have a potential issue, or an appropriate confidence factor was not met. 
     The GUIs  250 - 350  in  FIGS.  5 - 7    generally include an indicator  255 . In the GUIs  300  and  350 , because the GUIs are representative of situations where the installation of the component was not validated as being correct, a profile image  260  is displayed showing the correct assembly. It is to be appreciated that the GUI  250  could also display the profile image  260 . The indicator  255  can vary based on the GUI to visually represent to the operator whether the component was correctly installed (GUI  250 ), incorrectly installed (GUI  300 ), or needs checking (GUI  350 ). The indicators  255  in the illustrated embodiments are representative and are not intended to be limiting. The image  260  in each of the GUIs  300  and  350  is the same and is a top view of the profile image. It is to be appreciated that the image  260  can vary. That is, a different view can be shown in an embodiment. 
     Examples of computer-readable storage media include, but are not limited to, any tangible medium capable of storing a computer program for use by a programmable processing device to perform functions described herein by operating on input data and generating an output. A computer program is a set of instructions that can be used, directly or indirectly, in a computer system to perform a certain function or determine a certain result. Examples of computer-readable storage media include, but are not limited to, a floppy disk; a hard disk; a random access memory (RAM); a read-only memory (ROM); a semiconductor memory device such as, but not limited to, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), Flash memory, or the like; a portable compact disk read-only memory (CD-ROM); an optical storage device; a magnetic storage device; other similar device; or suitable combinations of the foregoing. 
     In some embodiments, hardwired circuitry may be used in combination with software instructions. Thus, the description is not limited to any specific combination of hardware circuitry and software instructions, nor to any source for the instructions executed by the data processing system. 
     The terminology used herein is intended to describe embodiments and is not intended to be limiting. The terms “a,” “an,” and “the” include the plural forms as well, unless clearly indicated otherwise. The terms “comprises” and/or “comprising,” when used in this Specification, specify the presence of the stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and/or components. 
     It is to be understood that changes may be made in detail, especially in matters of the construction materials employed and the shape, size, and arrangement of parts without departing from the scope of the present disclosure. This Specification and the embodiments described are examples, with the true scope and spirit of the disclosure being indicated by the claims that follow.