NEURAL NETWORK TRAINING METHOD

A method of training a neural network for use in surface metrology includes providing height image data comprising a series of height measurements of a sample, the height image data comprising a plurality of features; obtaining, from the height image data, a plurality of height image patches, each height image patch containing at least a portion of a feature. The method also includes applying one or more effects to each of the height image patches to obtain a corresponding modified height image patch for each height image patch and inputting one or more of the modified height image patches into the neural network. The method further includes using the neural network to identify a feature in each of the one or more modified height image patches and training the neural network based on the identification.

FIELD OF THE INVENTION

The present invention relates to a method, apparatus and computer program for training a neural network

BACKGROUND OF THE INVENTION

When measuring samples using a scanning probe microscope, it is often important to accurately measure features of these samples, such as the height of a protrusion or the depth of a depression. To measure height for example, it is necessary to measure a point of the feature relative to a “ground” level of the sample. To do this efficiently, and accurately on an automated basis, it is important to be able to reliably identify the feature, as well as suitable “ground” points in the image (i.e. those positions not relating to a feature or abnormality). This can be done using a neural network. There is therefore a need to train the neural network to be able to perform this function.

SUMMARY OF THE INVENTION

A first aspect of the invention provides a method of training a neural network for use in surface metrology, comprising: providing height image data comprising a series of height measurements of a sample, the height image data comprising a plurality of features; obtaining, from the height image data, a plurality of height image patches, each height image patch containing at least a portion of a feature; applying one or more effects to each of the height image patches to obtain a corresponding modified height image patch for each height image patch; inputting one or more of the modified height image patches into the neural network; using the neural network to identify a feature in each of the one or more modified height image patches; and training the neural network based on the identification.

The height image data may comprise real data obtained from a real sample. For example the height image data may comprise real data obtained from a real sample by scanning the real sample with a probe microscope. Additionally or alternatively, the height image data may comprise simulated data.

The one or more effects may comprise at least one of: a rotation; a reflection; applying noise; raising or lowering brightness; raising or lowering contrast; zooming in; and zooming out.

The plurality of height image patches may comprise 10000 height image patches. The plurality of height image patches may comprise 15000 height image patches.

The neural network may be trained based on each of the modified height image patches.

Each height image patch may comprise at least a corner, edge or central portion of a feature.

The method may further comprise: obtaining, from the height image data, a plurality of additional height image patches that do not contain a feature or a portion of a feature; applying one or more effects to each of the additional height image patches to obtain a corresponding modified additional height image patch for each additional height image patch; inputting one or more of the modified additional height image patches into the neural network; using the neural network to determine that there are no features in each of the one or more modified additional height image patches; and training the neural network based on the determination.

Optionally each height image patch comprises height measurements of an area of a surface of the sample.

Optionally the method further comprises: obtaining, from the height image data, height image patches which do not contain a portion of a feature.

Optionally a plurality of the features have at least a portion of the feature contained in at least one of the height image patches.

Optionally some of the height image patches contain an entire feature and some of the height image patches contain a portion of a feature.

A second aspect of the invention provides apparatus for training a neural network for use in surface metrology, comprising a processor configured to perform the method of the first aspect.

A third aspect of the invention provides a method of performing surface metrology of a sample, the method comprising: training a neural network by a method according to the first aspect, thereby generating a trained neural network; scanning a new sample to obtain a series of height measurements of the new sample; and operating the trained neural network to identify feature data in the series of height measurements of the new sample.

A fourth aspect of the invention provides a computer-readable medium that, when read by a computer, causes the computer to perform the method of the first aspect.

DETAILED DESCRIPTION OF EMBODIMENT(S)

A scanning probe microscopy system according to an embodiment of the invention is shown inFIG.1. The system comprises a piezoelectric driver4and a probe comprising a cantilever2and a probe tip3. The bottom of the piezoelectric driver4provides a cantilever mount, with the cantilever2extending from the cantilever mount from a proximal end or base to a distal free end. The probe tip3is carried by the free end of the cantilever2.

The probe tip3comprises a conical or pyramidal structure that tapers from its base to a point at its distal end that is its closest point of interaction with a sample7on a sample stage11a. The sample comprises a sample surface which defines a sample surface axis which is normal to the sample surface and inFIG.1also extends vertically. The cantilever2comprises a single beam with a rectangular profile extending from the cantilever mount13. The cantilever2has a length of about 20 micron, a width of about 10 micron, and a thickness of about 200 nm.

In this example the probe tip3tapers to a point, but in other embodiments the probe tip3may be specially adapted for measuring sidewalls. For instance the probe tip3may have a flared shape.

The cantilever2is a thermal bimorph structure composed of two (or more) materials, with differing thermal expansion coefficients—typically a silicon or silicon nitride base with a gold or aluminium coating. The coating extends the length of the cantilever and covers the reverse side from the tip3. An illumination system (in the form of a laser30) under the control of photothermal (PT) drive33is arranged to illuminate the cantilever on its upper coated side with an intensity-modulated radiation spot.

The cantilever2is formed from a monolithic structure with uniform thickness. For example the monolithic structure may be formed by selectively etching a thin film of SiO2or SiN4as described in Albrecht T., Akamine, S., Carver, T. E., Quate, C. F. J., Microfabrication of cantilever styli for the atomic force microscope, Vac. Sci. Technol. A 1990, 8, 3386 (hereinafter referred to as “Albrecht et al.”). The tip3may be formed integrally with the cantilever, as described in Albrecht et al., it may be formed by an additive process such as electron beam deposition, or it may be formed separately and attached by adhesive or some other attachment method.

The wavelength of the actuation beam32output by the laser30is selected for good absorption by the coating, so that the cantilever2bends along its length and moves the probe tip3. In this example the coating is on the reverse side from the sample so the cantilever2bends down towards the sample when heated, but alternatively the coating may be on the same side as the sample so the cantilever2bends away from the sample when heated.

The piezoelectric driver4expands and contracts up and down in the Z-direction in accordance with a piezo drive signal5at a piezo driver input. As described further below, the piezo drive signal5causes the piezoelectric driver4to move the probe repeatedly towards and away from the sample7in a series of cycles. The piezo drive signal5is generated by a piezo controller (not shown). Typically the piezoelectric driver4is mechanically guided by flexures (not shown).

A measurement system80is arranged to detect a height of the free end of the cantilever2directly opposite to the probe tip3. The measurement system80includes an interferometer and a quadrant photodiode (QPD).FIG.1only shows the measurement system80schematically andFIG.2gives a more detailed view. Light100from a laser101is split by a beam splitter102into a sensing beam103and a reference beam104. The reference beam104is directed onto a suitably positioned retro-reflector120and thereafter back to the beam splitter102. The retro-reflector120is aligned such that it provides a fixed optical path length relative to the vertical (Z) position of the sample7. The beam splitter102has an energy absorbing coating and splits both the incident103and reference104beams to produce first and second interferograms with a relative phase shift of 90 degrees. The two interferograms are detected respectively at first121and second122photodetectors.

Ideally, the outputs from the photodetectors121,122are complementary sine and cosine signals with a phase difference of 90 degrees. Further, they should have no dc offset, have equal amplitudes and only depend on the position of the cantilever and wavelength of the laser101. Known methods are used to monitor the outputs of the photodetectors121,122while changing the optical path difference in order to determine and to apply corrections for errors arising as a result of the two photodetector outputs not being perfectly harmonic, with equal amplitude and in phase quadrature. Similarly, dc offset levels are also corrected in accordance with methods known in the art.

These photodetector outputs are suitable for use with a conventional interferometer reversible fringe counting apparatus and fringe subdividing apparatus123, which may be provided as dedicated hardware, FPGA, DSP or as a programmed computer. Phase quadrature fringe counting apparatus is capable of measuring displacements in the position of the cantilever to an accuracy of λ/8. That is, to 66 nm for 532 nm light.

Known fringe subdividing techniques, based on the arc tangent of the signals, permit an improvement in accuracy to the nanometre scale or less. In the embodiment described above, the reference beam104is arranged to have a fixed optical path length relative to the Z position of the sample7. It could accordingly be reflected from the surface of the stage11aon which the sample7is mounted or from a retro-reflector whose position is linked to that of the stage. The reference path length may be greater than or smaller than the length of the path followed by the beam103reflected from the probe. Alternatively, the relationship between reflector and sample Z position does not have to be fixed. In such an embodiment the reference beam may be reflected from a fixed point, the fixed point having a known (but varying) relationship with the Z position of the sample. The height of the tip is therefore deduced from the interferometrically measured path difference and the Z position of the sample with respect to the fixed point.

The interferometer detector is one example of a homodyne system. The particular system described offers a number of advantages to this application. The use of two phase quadrature interferograms enables the measurement of cantilever displacement over multiple fringes, and hence over a large displacement range. Examples of an interferometer based on these principles are described in U.S. Pat. No. 6,678,056 and WO2010/067129. Alternative interferometer systems capable of measuring a change in optical path length may also be employed. A suitable homodyne polarisation interferometer is described in EP 1 892 727 and a suitable heterodyne interferometer is described in U.S. Pat. No. 5,144,150.

Returning toFIG.1, the output of the interferometer is a height signal on a height detection line20which is input to a surface height calculator (not shown) and a surface detection unit (not shown). The surface detection unit is arranged to generate a surface signal on a surface detector output line for each cycle when it detects an interaction of the probe tip3with the sample7.

The reflected beam is also split by a beam splitter106into first and second components107,110. The first component107is directed to a segmented quadrant photodiode108via a lens109, and the second component110is split by the beam splitter102and directed to the photodiodes121,122for generation of the height signal on the output line20. The photodiode108generates angle data124which is indicative of the position of the first component107of the reflected beam on the photodiode108, and varies in accordance with the angle of inclination of the cantilever relative to the sensing beam103.

The angle data124comprises a deflection/bending signal which indicates a flexural angle of the cantilever—i.e. an angle which changes as the cantilever bends along its length. Thus the deflection/bending signal is indicative of the flexural shape of the cantilever. The deflection/bending signal may be determined in accordance with a difference between the signals from the top and bottom halves of the quadrant photodiode108.

The angle data124also comprises a lateral/twisting signal which indicates a torsion angle of the cantilever—i.e. an angle which changes as the cantilever twists. Thus the lateral/twisting signal is indicative of the torsional shape of the cantilever. The lateral/twisting signal may be determined in accordance with a difference between the signals from the left and right halves of the quadrant photodiode108.

The scanning probe microscopy system described above is used to scan a sample to obtain probe microscope data in the form of a series of height image data. The sample comprises a plurality of features. A feature in the sample is simply an abnormality of the sample, optionally having a height dimension that notably deviates from a mean or mode height dimension across the sample. The features may be columns or wells, for example. Features correspond to feature data within height image data.

Turning toFIG.3A, height image data201obtained from the sample scanning is shown. The height image data201comprises height measurements of an area of a surface of the sample, each height measurement being represented by a respective pixel.

The height image data201is divided into a plurality of height image patches202. Each height image patch comprises height measurements of an area of the surface of the sample, each height measurement being represented by a respective pixel.

InFIG.3A, five such height image patches can be seen. Each of the height image patches shown contains at least a portion of a feature. However, it is possible that some patches may not contain even a portion of a feature, i.e. these patches would show a featureless portion of the sample. Two of the height images patches202overlap and hence both contain a shared portion of the height image data201. It is possible for more patches to overlap, or for no patches to overlap. Typically, fifty patches may be obtained per image.

A plurality of the features may have at least a portion of the feature contained in at least one of the height image patches. In the case ofFIG.3Atwenty features are shown in the image data201, and thirteen of these features have at least a portion of the feature contained in at least one of the height image patches202.

Some of the height image patches202inFIG.3A(in this case two of the height image patches202) contain an entire feature. All of the five height image patches202inFIG.3Acontain a portion of a feature. Some of the height image patches202inFIG.3A(in this case two of the height image patches202) contain an entire feature and also a portion of another feature.

FIG.3Bshows a number of height image patches202obtained from the height image data201. Some of the height image patches inFIG.3B(in this case three of the height image patches202) contain an entire feature. Some of the height image patches inFIG.3B(in this case six of the height image patches202) contain a portion of a feature.

One or more effects are applied to each of the height image patches to obtain a corresponding modified height image patch203for each height image patch, as can be seen inFIG.3C. The one or more effects may include at least one of: a rotation; a reflection; applying noise; raising or lowering brightness; raising or lowering contrast; zooming in; and zooming out.

FIG.4shows a number of modified height images patches203. As discussed above, each modified height image patch derives from a height image patch that has had one or more visual effects applied to it. The modified height image patches are used to train a neural network for the purpose of identifying feature data, and hence for segmenting height image data to identify feature data within the height image data.

Each modified height image patch is input into the neural network, and the neural network is used to identify any features or portions of features in the modified height image patch. The patches in which features have been identified (“identified patches”303) are shown alongside their respective modified height image patches. In the identified patches303, the feature data appears as white, while the non-feature data appears as black. For each identified patch303, the neural network is scored based on how accurately the features are identified. The neural network may also be provided with the “answer”, i.e. the correct feature identification—the correct features are manually labelled if real data is being used, or are already known and automatically output if simulated data is being used (discussed further below). The neural network is trained using this scoring. For example, if the neural network scores well for a particular patch, it will “learn” that its identification for that patch was good, and that it should reinforce the identification patterns used for that patch. Conversely, if the neural network scores poorly for a given patch, its algorithm/code may be altered so as to more accurately identify features in the future. This alteration may be based on the correct feature identification.

Alternatively or additionally, the neural network may simply be provided with both the modified height image patches and the corresponding identified patches at the same time, in order to learn to correctly identify the features without a scoring system.

Although this embodiment has been described using real height image data that was obtained from a scan performed by a scanning probe microscope, it will be appreciated that the neural network could equally or additionally be trained using height image data that has been simulated by a computer. For example, the height image patches could be obtained using data generated by a computer simulation. In other words, simulated data is created using a computer model of the physical probe interaction against samples. Samples and structures are defined in code, and the scanning probe microscope interactions are simulated on the virtual sample. Alternatively, the height image patches could be a combination of real height image data and simulated height image data.

Turning toFIG.5A, an image401formed from the height image data is shown. The height image data comprises feature data403corresponding to the one or more features of the sample. The features are columns that protrude from the sample surface, having a substantially square base when viewed from above. The height image data also comprises first region data402corresponding to a first region of the sample (i.e. non-feature data).

Once trained, the neural network segments the height image data so as to identify the feature data. In other words, the trained neural network receives the height image data as an input, and determines the portions of the image that correspond to the feature data403and the portions of the image that do not correspond to the feature data403. These determinations mean that the neural network can identify the feature data403.

FIG.5Bshows the result of the segmentation by the trained neural network. The neural network has identified the portions of the image that correspond to feature data403. The portions corresponding to identified feature data403aare shown inFIG.5B.

The neural network can output a mask, using the identified feature data403a. As shown inFIG.5C, this can be overlayed over the height image data, such that the feature data403is masked and the first region data402remains unmasked. Following this, one or more processing steps can take place. For example, if the first region data402is known to be substantially flat, the image401can be masked as discussed above. A transformation can be determined based on the unmasked first region data and the knowledge of the flatness of the first region. This may be necessary if the first region data is not substantially flat due to an error in the scanning or the scanning equipment for example. The transformation can then be applied to the whole dataset in order to obtain corrected height image data.

The transformation may be a first order transformation, for example, in which an angled plane is transformed to a horizontal plane. Alternatively, the transformation may be a second order transformation or higher order transformation, in which a curved surface is transformed to a horizontal plane.

FIG.6Ashows an image resulting from a series of height image data obtained with a scanning probe microscope. As with the data shown inFIG.5A, the height image data ofFIG.6Acomprises first region data502and feature data503corresponding to a plurality of features. However, in this Figure the features in question are domes that protrude from the sample surface, rather than columns. The domes have a substantially circular base when viewed from above.

As with the previous embodiment, the trained neural network segments the height image data so as to identify the feature data. The result of this identification503ais shown inFIG.6B. An optional masking process is shown inFIG.6C, as is discussed above.

Turning now toFIG.7, the training and use of the neural network601is shown. As has been described above, the neural network601receives images602a,602bas training inputs along with corresponding identifications603a,603bof any features within the images. These training inputs are used to train the neural network to identify feature data within height image data. As can be seen, both column features and dome features are provided as training inputs. Equally, further types of features/structures may be provided to the neural network as training inputs.

Specifically, though not shown in this Figure, the images602a,602bare provided to the neural network as modified height image patches202, and the correctly identified images603a,603bmay be provided at the same time as the modified height image patches202, or afterwards, for scoring purposes or alongside a scoring result.

Following the training of the neural network to generate a trained neural network, a new sample is scanned with a probe microscope to obtain a series of height measurements of the new sample, as represented by a new image604. The trained neural network is then operated to identify feature data in the series of height measurements of the new sample. For example the trained neural network601may segment the new image604to identify any feature data in the manner described above. This identification may be used to generate and output a mask605.

FIGS.8A-Cshow the effectiveness of the segmentation technique when it is carried out by a trained neural network compared with when it is carried out using a more classical approach. The original image701shown inFIG.8Adisplays a number of features, specifically columns, as with the image shown inFIG.5A. However, in this example the image is highly obscured by noise. As such, the columns are more difficult to discern in this image than in the image shown inFIG.5A.

FIG.8Bshows the result702of a classical method of segmenting the image. Such a classical method may involve identifying features by identifying portions of the height image data that exceed a given height threshold. Such a method be somewhat effective for height image data that does not have substantial levels of noise. However, given the high level of noise associated with the image ofFIG.8A, the result of the classical identification method is not reflective of the actual image, as can be seen inFIG.8B.

In contrast, the segmentation703performed by the trained neural network using the method described above is shown inFIG.8C. As can be seen this identification is a lot more accurate than the classical result.

Turning now toFIG.9, a flow chart illustrating the overall process of training the neural network is shown. Height image data comprising a series of height measurements of a sample is provided 801, the data comprising a plurality of features. From the height image data, a plurality of height image patches are obtained802. At least some of the height image patches contain a portion of a feature. Modified height image patches are obtained by applying one or more effects to the height image patches803, such that a corresponding modified height image patch is generated for each height image patch. The modified height image patches are input into the neural network804. The neural network identifies any features in each of the inputted modified height image patches, and is trained based on this identification805.