# Plenoptic Inspection System for Automatic Quality Control of MEMS and Microsystems

Moritz A. Kirschmann, Jörg Pierer, Alexander Steinecker, Philipp Schmid, Arne Erdmann

► To cite this version:

Moritz A. Kirschmann, Jörg Pierer, Alexander Steinecker, Philipp Schmid, Arne Erdmann. Plenoptic Inspection System for Automatic Quality Control of MEMS and Microsystems. 9th International Precision Assembly Seminar (IPAS), Dec 2020, Held virtually, Unknown Region. pp.220-232, 10.1007/978-3-030-72632-4_16. hal-03520396

HAL Id: hal-03520396

https://inria.hal.science/hal-03520396

Submitted on 11 Jan 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Plenoptic Inspection System for Automatic Quality Control of MEMS and Microsystems

Moritz A. Kirschmann¹, Jörg Pierer¹, Alexander Steinecker¹, Philipp Schmid¹, Arne Erdmann²

¹ CSEM SA, Gründlistrasse 1, 6055 Alpnach Dorf, Switzerland

² Raytrix GmbH, Schauenburgerstr. 116, 24118 Kiel, Germany

moritz.kirschmann@csem.ch

**Abstract.** Optical quality control of MEMS and microsystems is challenging as these structures are micro-scale and three dimensional. Here we lay out different optical systems that can be used for 3D optical quality control in general and for such structures in particular. We further investigate one of these technologies – plenoptic cameras and characterize them for the described task, showing advantages and disadvantages. Key advantages are a huge increase in depth of field compared to conventional microscope camera systems allowing for fast acquisition of non-flat systems and secondly the resulting total focus images and depth maps. Finally we conclude that these advantages render plenoptic cameras a valuable technology for the application of quality control.

**Keywords:** Optical inspection, lightfield camera, plenoptic camera, automatic defect recognition, 3D inspection

# 1 Introduction

## 1.1 MEMS and typical defects

Advancements in manufacturing techniques of micro-electro-mechanical systems (MEMS) and microsystems such as deep reactive ion-etching further increased the range of possible applications of these systems. Bridging the world of electronics and mechanics, MEMS are used for applications as micro actuators, generating sound, vibration and liquid flow in microfluidic pumps, while micro sensors turn acceleration, gyroscopic forces, inertia, pressure, sound waves et cetera into electric signals. These actuators and sensor are used in many industries such as mobile computing, automotive, VR/AR, electronic entertainment, pharma and medical. To achieve high production yield and quality of the final products quality control is key. Functional testing of MEMS is often only possible in a further integrated assembly. Yet it is desirable to
assess product quality as early as possible to avoid assembly of defect components into integrated systems rendering a whole assembly worthless. To assess quality in an early stage avoiding further assembly, optical quality control is one possibility. However, MEMS have a key property that makes optical quality control specially challenging: functional structures of MEMS are intrinsically extended in depth (See Fig. 1, Fig. 2 and Fig. 3).

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Schematic cross-section of a Capacitive Micromachined Ultrasonic Transducer (CMUT) device (top-left), optical microphotographs of CMUT devices in top view (lower-left) and (right) a col-or-coded depth map acquired by holographic microscopy (image curtesy of Philips Electronics Nederland B.V.).</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Microphotograph of CMUTs with typical defects highlighted by white arrows: Missing CMUT disks (left) and dust, resist or other particles (right); (image curtesy of Philips Electronics Nederland B.V.).</figcaption></figure>

An optical quality control system for MEMS must therefore fulfill three requirements: (1) Micrometer-scale features must be resolved in all three spatial dimensions. Production defects such as a displaced transducer bridge or wire bond may only be detectable by utilizing height data captured by the optical system. (2) The relevant depth
of the product must be acquired at sufficient resolution. (3) The system's throughput must be high enough to allow for real-time inspection of produced units.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** A selection of failures and defects that can occur during the fabrication of the Flex-to-Rigid structures (image courtesy of Philips Electronics Nederland B.V.).</figcaption></figure>
## 1.2 State of the art (3D) inspection technology

Different optical acquisition systems can be used for quality control in general but only a few are suitable for MEMS inspection. The straightforward approach is the application of conventional wide field microscopy. Considering a typical microscope objective with sub-micrometer resolution a Field of View (FoV) in the range of 0.3 mm² might be captured with one image. Capturing the area of an eight-inch wafer (30'000 mm²) would already require 100'000 images. The typical Depth of Field (DoF) of the same lens covers maximally 2 in height [1]. For MEMS with 3D structures covering a thickness ranging from sub-micrometer up to several 100 µm, a technique called “focus stacking” may be used to expand the DoF and acquire depth data [2]. For very common structures 100 µm in height the eight-inch wafer required an amount of 5 million images. Even with a fast camera enabling 100 fps (frames per second) and a manipulation system matching this speed for repositioning the wafer it would take 14 hours to capture the 3D data, not yet considering any kind of image analysis. While 2D inspection is a very common technique, 3D by focus stacking is not employed in industry for MEMS inspection.

Another possibility to obtain 3D depth information are stereo camera images by analyzing shift of parallax of image features. However, parts of an image that are occluded from one view lack parallax information. This renders inference of the depth impossible at these places. Systems with more cameras mitigate this issue as the chance of occluding parts of the image for all but one camera decreases with the number of cameras [3]. However, the complexity and bulkiness of the system also increases with the number of cameras. The need for two lenses and cameras imaging the same FoV limits the application to macroscopic lenses. Thus the limited achievable resolution, laterally as well as in height, so far prohibits the application of stereo camera systems for MEMS inspection.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Stereo camera system offered by Ensenso GmbH [4].</figcaption></figure>

Laser Line Triangulation (LLT) illuminates the sample by swiping a projected laser line from an angle to the viewing direction [5]. Depth displacement in the imaged sample shifts the apparent laser line position. On uniform samples such systems deliver precise depth maps quickly [6] but samples with strong differences in reflectivity and
absorption can cause artefacts. The lateral resolution of such systems is mostly limited
by diffraction properties of the projected laser beam (several µm) while the depth res-
olution is a function of incidence angle, sensor pixel size and lens magnification. Com-
mon systems range from µm to several 10 micrometers. LLT systems do not generate
photo-like images and therefore need to be combined with microscope cameras in order
to deliver image and depth map.

A very similar technology probably in direct competition to plenoptic cameras is structured illumination microscopy. It probably also is the most versatile technology ranging from moiré fringe exploitation [7] to various kind of pattern projection and 3D information extraction [8]. In the first case the superposition of a well-defined illumination pattern with the structure of the object generates a beat pattern otherwise known as moiré fringes. The scene has to be captured several times with slightly re-oriented or phase shifted pattern. The simpler pattern projection relies on known geometric relations between camera and projector with the advantage to need only one image per scene, depending on the actual pattern. In both cases subsequent processing of the image(s) enables the generation of 3D data. With proper computational hardware and a fast camera several 3D images might be obtained per second. In terms of DoF, FoV and resolution the same constraints as for any other optical system apply.

<figure><img src="image_6.png" /><figcaption>Fig. 5. Schematics of (left) laser line triangulation [5] and (right) structured light illumination [8].</figcaption></figure>

Imaging based on holography employing laser interference (DHM Digital Holo-
graphic Microscopy) offers outstanding depth resolution around 20 nm and simultane-
ously a considerable DoF of up to a few 100 µm [9], [10], [11]. The latter one, however,
is only true for rather continuous structures. As soon as steps and sharp edges are in-
volved those numbers get down to sub-um up to a few um. The lateral resolution as
well as the FoV are limited by laws of physics just as for the other optical imaging
technologies as well. Even so this technology has some drawbacks, it offers very high
performance in terms of 3D imaging. Unfortunately, this comes literally at costs at least an order of magnitude higher than any of the other systems.

<figure><img src="image_7.png" /><figcaption>Fig. 6. Typical outline of a Digital Holographic Microscopy system [11].</figcaption></figure>

OCT (Optical Coherence Tomography) as a 3D imaging technology shall be men-
tioned as well. Although an interferometric principle as holography it relies on a short
coherence length of broadband light sources such as superluminescent LEDs [12]. Typ-
ical depth resolution achieved goes down to the sub-micrometer range. The DoF of an
OCT system depends on either a mechanical movement of a mirror in the reference arm
(Time Domain OCT) or the wavelength sweeping of the source (Frequency Domain
OCT) and is for TD-OCT at least in theory only limited by size constraints. Lateral
resolution again is limited by physics, depending on the lens used. Besides the need for
depth scanning which increases the acquisitions time the small number of pixels of
available cameras with smart pixels (e.g. 280 x 292 [13]) limits the FoV and therewith
further increases the acquisition time. It shall be mentioned that standard cameras may
be used as well but demand a way more complex setup [14].

## 2 3D real-time imaging with plenoptic camera

In the following the hard- and software technology of plenoptic cameras and their ap-
plications in our context of inspection systems are described.

2.1 Principle of plenoptic camera technology

The general principle of a plenoptic, also known as light field camera dates back to
Leonardo Da Vinci and he even described the principle of a pinhole camera to capture
the light field [15]. The first multi-lens concept and actual camera was described and
developed by physicist Gabriel Lippmann in 1908 [16] although the term “light field”
was not used at this time. In 1992 Edward Adelson and John Wang published an article
describing the plenoptic camera concept as, in its basic principle, is used until to-
day [17].
Conventional 2D digital cameras use objectives which images an object onto a 2-dimensional array of photosensitive elements. In a plenoptic camera, as Adelson and Wang described, a microlens array is placed between main lens and the photosensitive camera chip. Hereby the main lens generates an intermediate image which subsequently is imaged by the microlenses onto subsegments of the photosensitive part of the chip [18] (see Fig. 7).

<figure><img src="image_8.png" /><figcaption>Fig. 7. Schematic representation of imaging with standard (left) and plenoptic camera (right).</figcaption></figure>

As depicted by the small red respectively yellow arrow on the right side of Fig. 7, the object point will be imaged on a different camera pixel with respect to the optical axis of the corresponding microlens. Based on that not only the position of the object point in space can be determined but as well the direction wherefrom the light rays originated. Adjacent microlenses are producing sub-images with are partially overlapping FoVs. Knowing the configuration of main lens, microlens array and camera chip it is possible to computationally reconstruct a 2D image and a depth map [19] (Fig. 8) which together are used to form the 3D image as depicted in Fig. 9.
<figure><img src="image_9.png" /><figcaption>**Fig. 8.** Computational reconstructed image (bottom middle) from raw image (left). The microlens sub-images are visible in the enlarged image (top middle). The image on the top right shows the color-coded depth map for high contrast areas while below the same map is filled.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Fig. 9.** Real-time 3D image as obtained by Raytrix light field camera.</figcaption></figure>

## 2.2 Application of plenoptic cameras for MEMS and microsystems inspection

As summarized above many 3D imaging technologies exist, each having its strengths and weaknesses. The same is true for plenoptic cameras. Due to redundant imaging of object points on several microlens sub-images the reconstructed 2D image loses planar resolution by a factor of 2-times in every direction. This may be countered by use of
large pixel count camera chips. However, this increases the amount of data to be han-
dled by the camera interface, the computational hardware and finally by means of stor-
age. Actually these are the main reasons why plenoptic cameras became available for
real world applications only during the last decade. Modern GPU hardware made real-
time reconstruction of high pixel count light field camera images manageable.

Plenoptic cameras of Raytrix offer unique advantages for industrial applications. As
only one camera with a standard main lens is needed, the entire setup becomes rather
lean and consumes little space. Assuming a fast interface, e.g. Camera Link, acquisition
of full 3D images at 80 frames per second is feasible. As a side effect of the special
Raytrix technology using microlenses with different focal lengths on each of the arrays
[19], the DoF is increased by a factor of 6 compared to a standard cameras. The proba-
bly biggest advantage compared to many of the other 3D imaging technologies is the
almost total lack of occlusion. As can be seen in Fig. 10 this enables inspection of de-
vises with high aspect ratios and steep edges.
<figure><img src="image_11.png" /><figcaption>**Fig. 10.** Upper left: Raw image from a plenoptic camera, Lower left: Detail crop from the area marked by the red rectangle. Round sub-images created by the microlenses are visible. Note the duplication of features in neighboring sub-images. Lower right: Algorithmically reconstructed sharp image. Right: 3D rendering of the image.</figcaption></figure>

Within a European project “CITCOM”<fnref n="1" /> we are investigating in detail the 3D MEMS inspection using light field cameras. We used and assessed two different plenoptic camera-main lens combinations during development of an automatic optical quality-control system. Specs of those are provided in Table 1. Shown in Fig. 11 is the test setup (no cabling) with a Raytrix R26 camera and 3x telecentric lens on the left side and a R12 in microscope configuration employing a 20x main lens.

<table><caption>**Table 1.** Main specifications of used plenoptic cameras</caption><thead><tr><th></th><th>Raytrix R26</th><th>Raytrix R12</th></tr></thead><tbody><tr><td>Working distance</td><td>48 mm</td><td>20 mm</td></tr><tr><td>Field of View</td><td>7.8 x 7.8 mm²</td><td>0.65 x 0.44 mm²</td></tr></tbody></table>

<footnote n="1">¹ www.citcom.eu; This project has received funding from the European Union's HORIZON 2020 research and innovation program under Grant Agreement no. 768883.</footnote>
<table><tr><td>Lateral resolution</td><td>3 µm</td><td>0.7 µm</td></tr><tr><td>Depth of Field</td><td>1.1 mm</td><td>60 µm</td></tr><tr><td>Depth resolution</td><td>2.5 µm</td><td>0.4 µm</td></tr><tr><td>Frame rate (max)</td><td>81 fps</td><td>30 fps</td></tr></table>

<figure><img src="image_12.png" /><figcaption>Fig. 11. Development setup with two plenoptic cameras Raytrix R26 (left) and R12 (right).</figcaption></figure>

The R26 setup enables very fast capturing of an entire eight-inch wafer. Considering an overlap of 10% to enable stitching of the images only about 600 images are needed which might be acquired in less than 10 seconds. In practice, however, not the speed of the camera is limiting the acquisition time but the movement of the stages. While the 3D reconstruction of a single image can be obtained at the maximal frame rate on PCs employing 4 GPUs (Graphic Processing Units), image stitching and automated recognition of defects and anomalies have to be considered as well.

While most of the typical defects and anomalies might be found with this setup, some demand for higher resolution in both, lateral as well as vertical direction. For those cases a second camera, an R12 in combination with a microscope objective, are used on specific areas of interest. Scanning the entire wafer with this camera is neither useful considering the necessary 130'000 images at 30 fps (frames per second) taking more than one hour at best, nor is it needed. With a FoV covering less than 0.3 mm², most of the wafer or substrate area appears as a plain, feature-less surface, while in specific areas a closer look might be highly appreciated.

Images taken with the setup described above are presented in Fig. 12 allowing for metrology in all spatial dimensions on a sub-micrometer scale.
<figure><img src="image_13.png" /><figcaption>**Fig. 12.** Left: Total focus image of PCB captured with Raytrix R26 and a 3x tele-centric lens. Right: 3D rendered image (bottom) with color coded depth map overly (top).</figcaption></figure>

An actual disadvantage that accounts for other techniques as well is the inability to computationally reconstruct the depth map at locations with missing contrast. However this can be mitigated by illuminating with a light patterned as depicted in Fig. 13. Those images were generated within a Bachelor thesis [20] initiated and supervised by CSEM. This structured illumination reveals the surface topology and allows a 3D reconstruction even if contrast is lacking.

<figure><img src="image_14.png" /><figcaption>**Fig. 13.** Imaging of component with partially low contrast resulting in lack of depth information (left). Patterned illumination projection mitigates the effect of contrast-arm surfaces (right). Colored images depict the color-coded depth maps of the 3D objects. Images by [20].</figcaption></figure>

As the reconstructed total focus images and depth maps can be exported as 2D image data, existing software for quality control, fault detection and anomaly detection can directly process and analyze these images. In order to achieve a proper performance of
the system, parameters for reconstruction of total focus and depth maps need to be
adapted for different inspection scenarios.

3 Conclusion

Real-time 3D inspection is not needed for every application. However, where it is the case multiple technologies might be considered ranging from simple stacking of 2D images over stereo camera configuration all the way to interferometric systems. As usual complexity and cost are increasing with more demanding needs. Plenoptic cameras might not be perfect and certainly will not replace every other inspection technology. However, as is the case for MEMS and other microsystems in general, plenoptic cameras have a huge potential whenever the following criteria seem to be advantageous:

* 6-times larger Depth of Field compared to standard camera

* occlusion-free 3D imaging

* real-time 3D image acquisition up to 80 frames per second

* simple implementation due to single camera/lens configuration

* cost efficient compared to competing technologies with similar performance

* simple combination with other means of inspection, e.g. electrical probing possible

Plenoptic cameras are a valuable instrument for the inspections of MEMS because
of its unique advantages at their competitive price.

References

[1] Edmund Optics Ltd, "20X Mitutoyo Plan Apo Infinity Corrected Long WD Objective," Edmund Optics Ltd, 2019. [Online]. Available: https://www.edmundoptics.de/p/20x-mitutoyo-plan-apo-infinity-corrected-long-wd-objective/6625/. [Accessed 16 September 2019].

[2] J. Ens and P. Lawrence, "An Investigation of Methods for Determining Depth from Focus," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 15, no. 2, pp. 97-108, 1993.

[3] B. Wilburn, N. Joshi, M. Levoy, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams and M. Horowitz, "High Performance Imaging Using Large Camera Arrays," *ACM Transactions on Graphics*, vol. 24, no. 3, pp. 765-776, 2005.

[4] Ensenso GmbH, "www.ensenso.com," Ensenso GmbH, 2018. [Online]. Available: https://www.ensenso.com/. [Accessed 12 September 2019].

[5] D. Callen, "Vision-Spectra.com," Coherent Inc., May 2017. [Online]. Available: https://www.photonics.com/Article.aspx?AID=62061. [Accessed 11 September 2019].
[6] Micro-Epsilon, "Micro Epsilon Laser Triangulation system," MICRO-EPSILON UK & Ireland Ltd., 2019. [Online]. Available: https://www.micro-epsilon.co.uk/service/glossar/Laser-Triangulation.html.

[7] M. G. L. Gustafson, "Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy," *Journal of Microscopy*, vol. 198, no. 2, pp. 82-87, 2000.

[8] J. Geng, "Structured-light 3D surface imaging: a tutorial," *Advances in Optics and Photonics*, vol. 3, no. 2, pp. 128-160, 2011.

[9] Lyncée Tec, "www.lynceetec.com," [Online]. Available: https://www.lynceetec.com/. [Accessed 11 September 2019].

[10] V. Pagliarulo, T. Russo, L. Miccio and P. Ferraro, "Numerical tools for the characterization of microelectromechanical systems by digital holographic microscopy," *Journal of Micro/Nanolithography, MEMS, and MOEMS*, vol. 14, no. 4, p. 041314, 2015.

[11] W. contributors, "Digital holographic microscopy," Wikipedia, The Free Encyclopedia, 28 August 2019. [Online]. Available: https://en.wikipedia.org/w/index.php?title=Digital_holographic_microscopy&oldid=912841452. [Accessed 12 September 2019].

[12] W. contributors, "Optical coherence tomography," Wikipedia, The Free Encyclopedia, 11 July 2019. [Online]. Available: https://en.wikipedia.org/w/index.php?title=Optical_coherence_tomography&oldid=905735980. [Accessed 12 September 2019].

[13] Heliotis AG, "www.heliotis.ch," [Online]. Available: https://www.heliotis.ch/index.html. [Accessed 12 September 2019].

[14] C. Dunsby, Y. Gu and P. M. W. French, "Single-shot phase-stepped wide-field coherence-gated imaging," *Optics Express*, vol. 11, no. 2, pp. 105-115, 2003.

[15] LightField Forum, "History of light field photography ( timeline)," [Online]. Available: http://lightfield-forum.com/what-is-the-lightfield/history-of-light-field-photography-timeline/. [Accessed 13 September 2019].

[16] G. Lippmann, "Épreuves réversibles donnant la sensation du relief," *J. Phys. Theor. Appl.*, vol. 7, no. 1, pp. 821-825, 1908.

[17] E. H. Adelson and J. Y. A. Wang, "Single Lens Stereo with a Plenoptic Camera," *IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE*, vol. 14, no. 2, pp. 99-106, 1992.

[18] A. Lumsdaine and T. Georgiev, "Full resolution lightfield rendering.," Indiana University and Adobe Systems, 2008.

[19] C. Perwass and L. Wietzke, "Single lens 3D-camera with extended depth-of-field," *Proc. SPIE*, vol. 8291, pp. 4-, 2012.

[20] S. Ammann and O. Gilson, *Pattern Illumination for Lightfield Camera*, Buchs: Interstaatliche Hochschule für Technik Buchs, 2019.