Inspection of circuit boards for unauthorized modifications

A target image of a target circuit board and a gold image of a gold circuit board are taken by an image acquisition system. Fiducial points are located on the target image and on the gold image. Perspective transformation is performed on the target image using the fiducial points on the target image for reference and on the gold image using the fiducial points on the gold image for reference. After perspective transformation, an anomalous section of the target image is identified by identifying pixels that have different intensities between the target image and the gold image, the anomalous section being indicative of an unauthorized modification to the target circuit board.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to circuit board manufacturing, and more particularly but not exclusively to inspection of circuit boards.

2. Description of the Background Art

Electrical circuits may be implemented on a circuit board. Depending on the electrical circuit, a circuit board may have many components, including integrated circuit chips, discrete components (e.g., capacitors, resistors, inductors), switches, jumpers, traces, etc. Examples of circuit boards include motherboards (also referred to as main circuit boards) and daughter boards that are used in computer systems and other electronic devices. It is critical not just for functionality but also for security reasons that a circuit board is manufactured to design specifications. For example, an unauthorized modification (e.g., adding a hacking chip) to a circuit board of a network device may allow a cyber criminal to infiltrate a computer network.

SUMMARY

In one embodiment, a target image of a target circuit board and a gold image of a gold circuit board are taken by an image acquisition system. Fiducial points are located on the target image and on the gold image. Perspective transformation is performed on the target image using the fiducial points on the target image for reference and on the gold image using the fiducial points on the gold image for reference. After perspective transformation, an anomalous section of the target image is identified by identifying pixels that have different intensities between the target image and the gold image, the anomalous section being indicative of an unauthorized modification to the target circuit board.

DETAILED DESCRIPTION

FIG.1shows a logical diagram of an inspection system100in accordance with an embodiment of the present invention. The inspection system100is configured to detect unauthorized modifications to a target (i.e., being inspected) circuit board102. The inspection system100may comprise an image acquisition system110and a computer120. In the example ofFIG.1, the inspection system100is configured to detect one or more differences between the target circuit board102and a gold circuit board101based on their respective images. The gold circuit board101is “gold” in that it is a known-good circuit board, i.e., verified to have been manufactured as per design specifications. The target circuit board102is expected to be manufactured to the same design specifications as the gold circuit board101. A feature (e.g., component, trace) that is present in the target circuit board102but is not present in the gold circuit board101is deemed to be an unauthorized modification of the target circuit board102.

In the example ofFIG.1, an image acquisition system110is configured to take an image of a subject, such as a circuit board. In one embodiment, the image acquisition system110comprises an X-ray imaging machine, and each of the gold image103and the target image104is an X-ray image. An X-ray imagining machine advantageously captures relatively small features (e.g., traces) of circuit boards. Depending on the application, other suitable imaging machines may also be used, including Automated Optical Inspection (AOI) machines.

In the example ofFIG.1, the image acquisition system110takes an image of the gold circuit board101to generate a gold image103and takes an image of the target circuit board102to generate a target image104. In one embodiment, the gold image103and the target image104are x-ray images in grayscale. The gold image103and target image104may be represented in TIFF or another image file format.

The computer120may comprise a workstation, server computer, a desktop computer, or other computing device. The computer120includes a processor121(e.g., central processing unit), a memory122(e.g., random access memory, solid state drive), and other computer components that are not shown (e.g., keyboard, network interface card, display screen, I/O ports). In the example ofFIG.1, the memory122stores instructions of an image processing library123and of a modification detector124. The instructions stored in the memory122, when executed by the processor121, cause the computer120to perform its functions as described below.

The image processing library123comprises functions or other program code for image processing. In one embodiment, the image processing library123comprises the OpenCV library. Other suitable image processing libraries or program code for image processing may also be employed. As can be appreciated, the use of the image processing library123is largely a matter of convenience. One can always write his or her own program code to perform an image processing step.

The modification detector124is configured to detect unauthorized modifications to a target circuit board. In the example ofFIG.1, the gold image103and the target image104are received in the computer120. The gold image103and the target image104may be received as image files, whose contents are subsequently loaded in the memory122for processing by the modification detector124. The modification detector124may perform image processing algorithms described below, in conjunction with the image processing library123, to detect unauthorized modifications on the target image104relative to the gold image103.

FIG.2shows a flow diagram of a method200of inspecting circuit boards for unauthorized modifications in accordance with an embodiment of the present invention. The method200may be performed by the modification detector124in the computer120. In the example ofFIG.2, the steps of the method200are performed on the gold image103and the target image104in sequential order. For example, images being processed in step206have already been set to the same canvas size (as in step202), perspective transformed (as in step204) using located fiducial points for reference (as in step203), and blurred (as in step205).

In step201, the gold image103and the target image104are loaded in the memory122for processing. For example, the gold image103and the target image104may be loaded from their respective files using the “imread” function of the OpenCV library.

In step202, the gold image103and the target image104are set to have the same canvas size. In one embodiment, canvas size refers to the width and height of an image in units of pixels. More particularly, the gold image103and/or the target image104may be scaled up or down, such that both images have the same width and height. Setting the two images to have the same canvas size includes adjusting one or both dimensions (i.e., width and/or height) of one or both images. An image may have surrounding white portions at the edges, which show portions that are not part of the circuit board. The white portions may be enlarged so that the gold image103and the target image have the same canvas size.

In step203, fiducial points are located on each of the gold image103and the target image104. A fiducial point is a point of reference on an image. In one embodiment, four fiducial points, one on each corner, are located on each image. The fiducial points may be mounting holes (e.g., screw holes) on each corner of a circuit board, for example.

In step204, perspective transformation is performed on each of the gold image103and the target image104. The gold circuit board101and the target circuit board102are three-dimensional objects that are represented in two dimensions on their respective images. Perspective transformation corrects or minimizes errors caused by projecting a three-dimensional circuit board to a two-dimensional image. The fiducial points found in step204may be used to calibrate and align the gold image103and target image104to have the same size and aspect ratio by perspective transformation. That is, perspective transformation of an image may be performed using the located fiducial points on the image for reference. For example, the four fiducial points located in the step203may be employed in conjunction with the “getPerspectiveTransform” function of the OpenCV library to perform perspective transformation on the gold image103and the target image104.

In step205, the gold image103and the target image104are blurred to reduce the noise level of the images. The blurring step smooths the boundary edges of dual in-line packaged (DIP) components, such as Dual in-Line Memory Module (DIMM) sockets, heatsinks, etc. The blurring step advantageously reduces unnecessary displacement tolerances of these components as they appear on the image.

In step206, the intensities of corresponding pixels of the gold and target images are compared. In one embodiment, the target image104is subtracted from the gold image103to generate a difference image. In a grayscale image, each pixel has a matrix location and an intensity. In the case of an 8-bit intensity, the intensity may be in the range of 0 to 255. In one embodiment, an intensity of zero is designated as minimum intensity and an intensity of 255 is designated as maximum intensity, with an intensity of 1 to 254 being a shade between minimum and maximum intensities. In one embodiment, a pixel with minimum intensity appears as black on the image, whereas a pixel with maximum intensity appears as white on the image.

For each pixel, the subtraction step subtracts the intensity of a pixel of the target image104from the intensity of a corresponding pixel (i.e., same matrix location) of the gold image103. A pixel of the difference image has a zero intensity when corresponding pixels of the gold image103and the target image104have the same intensity. In one embodiment, a pixel of the difference image with a zero intensity indicates no modification and will appear as fully black, i.e., minimum intensity, on the difference image. In one embodiment, a pixel of the difference image with a negative intensity is ignored for purposes of detecting unauthorized modifications, because negative intensity is indicative of assembly errors (e.g., missing components) rather than unauthorized modifications, which typically involve adding components (e.g., hacking chips) to the target circuit board102.

In one embodiment, a pixel of the difference image will have a non-zero intensity when corresponding pixels of the gold image103and the target image104have different intensities. In that case, the non-zero intensity is compared to an intensity threshold, and the non-zero intensity is converted to maximum intensity when the non-zero intensity exceeds the intensity threshold. The intensity threshold may be adjusted to emphasize or de-emphasize sections of pixels that have different intensities on the target image104and the gold image103(step207). That is, after the thresholding, pixels of the difference image with intensities that meet (i.e., equal to or greater than) the intensity threshold will appear more prominent, making them easier to distinguish from pixels with intensities below the intensity threshold.

Sections of pixels with intensities that meet the intensity threshold indicate differences between the target image104and the gold image103and are thus indicative of unauthorized modifications. These sections of pixels, which are also referred to herein as anomalous sections, may be identified with markings (step208). The markings may be superimposed on the target image104(step209) to locate the anomalous sections on the target image104(step210). A portion of the target image104, e.g., quadrant, where a marking identifies an unauthorized modification may be enhanced to facilitate identification of the unauthorized modification (step211). The enhancement may involve enlarging the section with the unauthorized modification.

In response to detecting an unauthorized modification, an alert may be raised (step212) to notify inspection personnel of the detection of the unauthorized modification on the target circuit board102. The alert may be visual (e.g., flashing light, colored button or icon on an interface), acoustical (e.g., alarm sound), textual (e.g., text message, email, log entry, message window, characters on an interface), or in some other form.

FIG.3shows a flow diagram of a method250of locating fiducial points on an image in accordance with an embodiment of the present invention. The method250may be performed as the step203of the method200(seeFIG.2) to locate fiducial points on each of the gold image103and the target image104. The method250is performed after both images have been set to have the same canvas size (i.e., after step202of the method200). In the example ofFIG.3, the steps of the method250are performed in sequential order.

In step251, an image is converted from a grayscale image to a binary image. A grayscale image has a spectrum distribution of pixel intensities, which range from minimum intensity (e.g., “0”) to maximum intensity (e.g., “255”). In the spectrum distribution, each intensity has an associated number of pixels that have that intensity. An accumulation of intensities, from maximum intensity down to a cutoff intensity, contributes a number of pixels that is greater than a percentage of the total pixel count of the entire image. The cutoff intensity may be used as a limit for converting the grayscale image to the binary image. For example, an intensity that is below the cutoff intensity may be converted to minimum intensity, and an intensity that is equal to or greater than the cutoff intensity may be converted to maximum intensity. Each pixel on the binary image will result in having one of two possible intensities, i.e., either minimum intensity or maximum intensity. The binary image is thereafter inverted to generate an inverted binary image (step252). The inverted binary image may be generated by inverting the intensities of each pixel of the binary image, i.e., from maximum intensity to minimum intensity and vice-versa.

In step253, the image is dilated to expand the white portions of the image by a predetermined factor. The dilation step causes the white portions of the image to get larger, thereby removing some noise from the image. That is, the expansion of the white portions of the image removes smaller black portions on the image. The dilation step may be performed using the “dilate” function of the OpenCV library, for example.

In step254, the white portions of the image are eroded to enhance the contrast between white and black portions of the image. In one embodiment, mounting holes on a circuit board are used as fiducial points. The erosion step makes the mounting holes appear more prominent with enhanced contrast on the image. The erosion step may be performed using the “erode” function of the OpenCV library, for example.

In step255, the contours of the image are found. A contour is a curve that joins continuous points along a boundary that have the same intensity. The contours of the image show the features on the image. For example, the “findContours” function of the OpenCV library may be used to find all contours on the image.

In step256, each contour of the image is bound by a rectangle. A contour has a center point, which is also referred to herein as “contour center point”. For each contour, a bounding rectangle may be found by finding the contour center point and generating a rectangle with a perimeter that covers the contour. A minimum area of a bounding rectangle may be specified such that contours with bounding rectangles that do not meet (i.e., less than) the minimum area may be removed from inspection consideration. The contours of the image may be bounded by rectangles using the “boundingRect” function of the OpenCV library, for example.

In step257, a main center point of the entire image is found. In one embodiment, the main center point of the entire image is the contour center point of the largest contour on the image.

In step258, fiducial points are located relative to the main center point. In one embodiment, an image is divided into four quadrants with respect to the main center point. For each quadrant, the distances from the main center point to contour center points are calculated (e.g., using the Pythagorean Theorem) and the contour center point that has the maximum distance from the main center point is selected as the fiducial point on that quadrant. Step258may be performed to locate the fiducial point on each corner of the image.

In step259, each located fiducial point on the image is assigned a designation. In one embodiment, the fiducial points are assigned numerical designations from1to4, in a clockwise direction starting from the upper left corner of the image.

FIGS.4-11pictorially illustrate a method of inspecting circuit boards for unauthorized modifications in accordance with an embodiment of the present invention.

Referring to the example ofFIG.4, there are shown a gold image103and a target image104as received from the image acquisition system110. The canvas sizes of the gold image103and the target image104are set to the same canvas size (see arrow301). A canvas size may be in terms of width and height in units of pixels. In one embodiment, a width of the canvas size is set to the maximum canvas width between the gold image103and the target image104and a height of the canvas size is the maximum canvas height between the gold image103and the target image104. In the example ofFIG.4, the gold image103has a canvas size of 100×100 pixels and the target image104has a canvas size of 200×150 pixels. Accordingly, the gold image103is scaled up by increasing its width (see arrow271) and height (see arrow272) to 200×150 pixels. It is to be noted that the aforementioned canvas sizes are given for ease of illustration. In general, one can use the highest resolution images available. The gold image103is relabeled as “103-1” and the target image104is relabeled as “104-1” to indicate that they have been scaled to the same canvas size.

FIGS.5-7pictorially illustrate locating fiducial points on an image andFIG.8pictorially illustrates performing perspective transformation on the image using the located fiducial points for reference in accordance with an embodiment of the present invention. The image processing steps illustrated inFIGS.5-8may be performed on each of the gold image103-1and target image104-1ofFIG.4.

Referring to the example ofFIG.5, a grayscale image351-1is converted to an inverted binary image351-2(see arrow302). The grayscale image351-1may be the gold image103-1or the target image104-1shown inFIG.4. The grayscale image351-1may be converted to a binary image, and the binary image may be inverted to generate the inverted binary image351-2. It is to be noted that a mounting hole (e.g., see dashed-circle352), being a void, will appear as white in the binary image (not shown). Inverting the binary image makes the mounting hole appear as black (e.g., see dashed-circle353) on the inverted binary image351-2. The inverted binary image351-2thus facilitates location of mounting holes used as fiducial points.

In the example ofFIG.6, the inverted binary image351-2is dilated to expand the white portions on the inverted binary image351-2(see arrow303). The dilation step removes small black spots from the inverted binary image351-2resulting in the dilated image351-3. Thereafter, the white portions of the dilated image351-3are eroded to increase the visibility and contrast of the black portions (see arrow304), resulting in the eroded image351-4. The contours on the eroded image351-4are found (see arrow305). The contours are bounded with rectangles, with the smaller bounding rectangles and their contours being removed from inspection consideration (see arrow306). In one embodiment, the threshold for removing smaller bounding rectangles is based on the size of the fiducial points. As a particular example, when mounting holes are used as fiducial points, bounding rectangles with an area smaller than that of a mounting hole are removed from inspection consideration.

In the example ofFIG.6, the image351-5is the eroded image351-4shown with bounding rectangles (which are illustrated as small squares) and the image351-6is the eroded image351-4shown with the smaller bounding rectangles removed. It is to be noted that bounding rectangles do not have to be shown on an image but are shown on the image351-5and351-6for illustration purposes.

In the example ofFIG.7, fiducial points are located on each quadrant of the image351-6(see arrow307). The main center point375of the image351-6, which is also labeled as origin (0,0), is in the middle of the image for illustration purposes only. It is to be noted that, in one embodiment, the main center point375is the contour center point of the largest contour, which is not necessarily located in the middle of the image. The quadrants I, II, III, and IV of the image351-6are relative to the main center point375. The fiducial points371,372,373, and374are located as contour center points that are farthest away from the main center point375relative to other contour center points on the same quadrant. The located fiducial points371,372,373, and374are assigned numerical designations (see arrow308) in a clockwise direction starting from the upper left corner of the image351-6.

In the example ofFIG.8, perspective transformation is performed on the image351-1(also shown inFIG.5). As previously noted, the image351-1may be the gold image103-1or the target image104-1shown inFIG.4. In the example ofFIG.8, the perspective error of the image351-1is exaggerated for illustration purposes. The previously located fiducial points371,372,373, and374are identified on the image351-1(see arrow401). A quadrilateral421, whose corners are on the fiducial points371,372,373, and374, illustrates the perspective error on the image351-1before the perspective transformation. Using the fiducial points371,372,373, and374for reference, perspective transformation is performed on the image (see arrow402) to generate the perspective-transformed image451-1. A quadrilateral422, whose corners are on the fiducial points371,372,373, and374, illustrates the perspective transformation relative to the image351-1(compare quadrilateral422to quadrilateral421). The image451-1may be blurred for noise reduction purposes.

In the example ofFIG.9, a gold image502and a target image501represent the gold image103-1and the target image104-1ofFIG.4, respectively, after perspective transformation and blurring. The target image501is subtracted from the gold image502to generate a difference image503(see arrow403). Each pixel of the difference image503has an intensity that is the difference between intensities of corresponding pixels of the gold image502and the target image501. In the example ofFIG.9, a pixel with zero intensity appears as black. Accordingly, on the difference image503, black sections of pixels indicate no modification, whereas lighter sections of pixels are indicative of a modification. A thresholding step may be performed to identify pixels that meet an intensity threshold, such that the intensities of those identified pixels may be increased (e.g., to maximum intensity) to enhance visibility. In the example ofFIG.9, a white section of pixels is an anomalous section that identifies a location of an unauthorized modification. As illustrated inFIG.10, the anomalous section of pixels is marked with a marking505, which is superimposed on the target image501.

The portion of the target image501in the vicinity of the marking505may be enhanced to increase visibility of the unauthorized modification. In the example ofFIG.11, an enhanced view506is an enlargement of the area of the target image501in the vicinity of the marking505. As is more evident in the enhanced view506, a feature507, which is not present on the gold image502, is identified. The feature507may be a jumper wire, surface mount component, or other feature that is not authorized to be on the target circuit board102.

System and method for inspecting circuit boards for unauthorized modifications have been disclosed. While specific embodiments of the present invention have been provided, it is to be understood that these embodiments are for illustration purposes and not limiting. Many additional embodiments will be apparent to persons of ordinary skill in the art reading this disclosure.