Patent Description:
In microscopy, an area of interest in a specimen to be imaged is often larger than can be displayed by taking a single image with the microscope. Thus scanning techniques are employed to image an entire desired area. In automated scanning, the specimen is moved under the objective lens of the microscope by an XY translation stage so that the microscope can scan across the desired area, with multiple images being collected and then aggregated or stitched to form a single larger image. This stitching can be accomplished using standard software techniques or by ensuring that images are taken at specific locations with very precise stage movement feedback such that, when a first image is taken, the stage moves exactly the distance equal to the width of the first image (and without movement in the height direction) and a second image is taken to them be joined at the common border. If precise enough, the left edge of first image will then exactly mate and compliment the right edge of for the second image.

It is often advantageous to align the camera pixels to a specific orientation relative to the specimen and then scan that specimen in a specific desired direction and in a manner that maintains the desired orientation. For example, components on silicon wafers (e.g., micro electronic devices or patterned films such as through photolithography) are often oriented in rows (x-direction) or columns (y-direction), and it is helpful to align the camera pixel rows parallel to a component row or align the camera pixel columns parallel to a component column to then accurately scan along a desired row or column while maintaining the parallel relationship there between.

In the current state of the art the camera pixel orientation is often manually aligned to the XY travel of the stage by visible observation, which, in light of the size scales typically involved, does not provide a suitable level of accuracy for many imaging needs. The chances of accurately aligning the pixel rows with the x-direction of the stage and the pixel columns with the y-direction of the stage are very low. After this likely inaccurate alignment, the specimen is rotated relative to the stage in an attempt to align the pixel rows of the image sensor with a desired x'-direction for scanning the specimen and/or to align the pixel columns with a desired y'-direction for scanning the specimen. That is, the specimen is rotated relative to the XY translation stage in order to position a desired x' scanning direction of the specimen parallel to the x-direction movement of the stage and/or position a desired y' scanning direction of the specimen parallel to the y-direction movement of the stage (i.e., the x'-direction and x-direction are intended to be the same and the y'-direction and y-direction are intended to be the same). The x-direction of the XY stage and the rows of pixels of the camera having been previously visually aligned (as are, axiomatically, the y-direction of the stage and the columns of pixels), the movement of the stage in the desired x-direction or desired y-direction maintains the desired alignment but only to the extent the manually alignment of the image sensor pixels to the XY travel of the stage is highly accurate and precise.

Returning to the patterned silicon wafer example, a desired x-direction might be a row of micro-circuits with this row being aligned parallel to the x-direction movement of the XY translation stage, and, thus parallel to the rows of pixels of the camera. The row of micro-circuits can thus be scanned simply by moving the XY translation stage in the x-direction, while the parallel relationship between the rows of pixel and the row of microcircuits is maintained, and the image sensor is not shifted in the y'-direction, such that accurate recording and stitching is facilitated.

Thus, accurate results depend upon highly accurate alignment of the camera pixels, the XY travel of the stage, and the desired x' and/or y' scanning directions of the specimen. This is difficult to achieve given normal tolerance in machining and errors inherent in mere visual observation alignment. If even slightly out of alignment, the image sensor will be shifted in the x'-direction and/or y'-direction to an unacceptable degree as the specimen is moved by the translation stage, thus frustrating the ease by which images can be analyzed and/or stitched together. Additionally, it is often desired that a specimen be analyzed with a minimal amount of handling of the specimen. Thus, there is a need in the art for new methods for aligning and scanning that do not rely on specimen movement and ensure accurate alignment between the image sensor and the specimen.

<CIT> discloses a magnification observation device, comprising: an imaging part that images an observation object, to acquire imaging data of an imaging region; a display part that displays an image of the observation object as an observed image based on the image data acquired by the imaging part; a stage that has a placement surface on which the observation object is placed, and is provided relatively rotatably with respect to the imaging part around a rotational axis substantially vertical to the placement surface, and movably along first and second axes intersecting with each other within a plane substantially parallel to the placement surface; a rotational angle detecting part that detects a rotational angle of the stage; a stage driving part that moves the stage relatively with respect to the imaging part along the first and second axes; an instruction accepting part that accepts an instruction for a moving direction of the stage; and a control part that provides the stage driving part with moving amounts of the stage along the first and second axes, to control movement of the stage, wherein the first and second axes of the stage rotate around the rotational axis integrally with the stage, and the control part is configured to control the moving amounts along the first and second axes to be provided to the stage driving part such that a moving direction of the observation object in the imaging region of the imaging part agrees with the direction accepted by the instruction accepting part, based on the rotational angle detected by the rotational angle detecting part. <CIT>, <CIT> and <CIT> disclose further microscopy systems and methods dealing with roatational misalignment between camera, specimen and stage.

In a first embodiment, the present invention provides a microscopy method for imaging a specimen comprising two alignment marks along a desired x'-direction of the specimen, the desired x'-direction defined by the alignment marks of the specimen. The specimen is placed on an XY translation stage and movable by the XY translation stage so as to have a portion of the specimen placed within the field of view of an image sensor. The XY translation stage is movable in an x-direction and a y-direction to move the specimen relative to the image sensor, the image sensor having a multitude of pixels arranged to define pixel rows and pixel columns, the desired x'-direction of the specimen being angularly offset from the x-direction of the XY translation stage so as to define a slope and angle of offset relative thereto, the image sensor viewing only a discrete segment of the specimen at a time. The method comprising the steps of: rotating the image sensor such that the pixel rows are substantially parallel with the desired x'-direction of the specimen, without rotating the XY translation stage; determining the angle of offset of the desired x'-direction as compared to the x-direction of the XY translation stage; establishing a first position for the specimen relative to the image sensor as rotated in said step of rotating, said first position placing at least a portion of the specimen within the field of view of the image sensor; and, after said step of determining and said step of establishing, moving the specimen with the XY translation stage to a second position along the desired x'-direction by moving the XY translation stage in the x-direction, wherein the second position places at least a second portion of the specimen within the field of view of the image sensor, and the second position is not substantially shifted in a y'-direction of the specimen, the y'-direction being orthogonal to the x'-direction of the specimen, wherein said step of moving is based upon the angle of offset determined in said step of determining.

In a second embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein said step of determining an angle of offset includes: measuring the x distance and y distance between a first focal feature and a second focal feature on the specimen and aligned along and thus defining the desired x'-direction of the specimen, the x distance and y distance being measured relative to the x-direction and y-directions of the translation stage.

In a third embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein said step of measuring the x distance and y distance includes: placing the first focal feature so as to overlap with one or more target pixels of the image sensor, and thereafter moving the specimen to place the second focal feature so as to overlap with the same one or more target pixels, said step of measuring the x distance and y distance being the magnitude of x and y movement of the translation stage (ΔX, ΔY) necessary to achieve said step of moving the specimen to place the second focal feature so as to overlap with the same one or more target pixels.

In a fourth embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein said target pixels encompass the center of the image sensor.

In a fifth embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein said step of rotating the image sensor includes identifying an axis-defining feature on the specimen running in the x'-direction, and using computer vision to align the pixel rows substantially parallel to the detectable direction of the specimen.

In a sixth embodiment, the present invention provides a microscopy method as in any of the embodiments above wherein said step of rotating the image sensor is performed before said step of measuring the x distance and y distance.

In a seventh embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein said step of rotating the image sensor includes taking a mosaic of images suitable for calculating a reference line between the first focal feature and the second focal feature and using computer vision to align the pixel rows of the image sensor with the reference line.

In an eighth embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein said step of taking a mosaic of images is carried out while carrying out said step of measuring the x distance and y distance.

In a ninth embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein, before said step of rotating the image sensor, the method includes the step of aligning the pixel rows substantially parallel to the x-direction of the XY translation stage.

In a tenth embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein said step of rotating the image sensor includes: identifying an axis-defining feature on the specimen, the axis-defining feature having a detectable shape running in the desired x'-direction; and using computer vision to align the pixel rows substantially parallel to the detectable shape, and said step of determining the angle of offset includes: measuring the degrees of rotation of the image sensor from its position after said step of aligning the pixel rows substantially parallel to the x-direction of the XY translation stage to its position after said step of rotating the image sensor.

In an eleventh embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein said XY translation stage provides a specimen chuck to hold the specimen, wherein either the specimen chuck or a specimen placed thereon includes a reference mark, and said step of aligning the pixel rows substantially parallel with the x-direction of the XY translation stage includes: placing the reference mark at a first position within in the field of view of the image sensor and taking image data to determine a first pixel row number for the position of the reference mark relative to the pixel rows, moving the specimen chuck along only the x-direction of the XY translation stage to place the reference mark at a second position within the field of view of the image sensor and taking image data to determine a second pixel row number for the position of the reference mark relative to the pixel rows, and, after said steps of placing and moving, rotating the image sensor to place the reference mark at a third position having a third pixel row number that is between said first pixel row number and said second pixel row number.

In a twelfth embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein, after said step of rotating the image sensor to place the reference mark at a third position, said steps of (i) placing the reference mark at a first position, (ii) moving the specimen chuck along only the x-direction, and (iii) rotating the image sensor to place the mark at a third position are repeated until the pixel rows are substantially parallel with the x-direction of the XY translation stage.

In a thirteenth embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein said step of determining is carried out after said step of aligning the pixel rows substantially parallel with the x-direction of the XY translation stage.

In a fourteenth embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein said step of rotating the image sensor includes: identifying an axis-defining feature on the specimen, the axis-defining feature having a detectable shape running in the desired x'-direction; and using computer vision to align the pixel rows substantially parallel to the detectable shape, and said step of determining the angle of offset includes: measuring the degrees of rotation of the image sensor from its position after said step of aligning the pixel rows substantially parallel to the x-direction of the XY translation stage to its position after said step of rotating the image sensor.

In a fifteenth embodiment, the present invention provides a microscopy method as in any of the embodiments above, wherein said step of measuring the degrees of rotation of the image sensor includes obtaining a signal output from an instrument rotating the image sensor.

In a sixteenth embodiment, the present invention provides a microscope system comprising: a microscope; an image sensor recording image data, said image sensor including pixel rows and pixel columns; an XY translation stage; a specimen on said XY translation stage and viewed by said image sensor, wherein the XY translation stage is movable in an x-direction and a y-direction to move the specimen relative to the image sensor, the image sensor having a multitude of pixels arranged to define pixel rows and pixel columns, wherein the specimen presents features along a x'-direction that is angularly offset from the x-direction of the XY translation stage so as to define an angle of offset relative thereto, the specimen further including a first focal feature and a second focal feature, a processor serving to: rotate the image sensor relative to the specimen such that the pixel rows are parallel with the x'-direction of specimen, without rotating the XY translation stage, move the XY translation stage in the x-direction after the image sensor is rotated; determine the angle of offset of the x'-direction as compared to the x-direction of the XY translation stage; and scanning across the specimen in the desired x' direction by: establishing a first position for the specimen relative to the image sensor when the pixel rows are parallel with the x'-direction, the first position placing at least a portion of the specimen within the field of view of the image sensor; and, moving the specimen with the XY translation stage in the x-direction to a second position along the desired x'-direction, wherein the second position places at least a second portion of the specimen within the field of view of the image sensor, and the second position is not substantially shifted in a y'-direction of the specimen, the y'-direction being orthogonal to the x'-direction of the specimen, wherein the movement is based upon the angle of offset determined by the processor.

In a seventeenth embodiment, the present invention provides a method for aligning pixel rows of an image sensor with the x-direction of an XY translation stage, wherein said XY translation stage provides a specimen chuck to hold the specimen, the specimen chuck being moved in a x-direction and a y-direction by the XY translation stage, the method comprising the steps of: providing a reference mark on the specimen chuck or on a specimen placed on the specimen chuck; placing the reference mark at a first position within the field of view of the image sensor and taking image data to determine a first pixel row number for the position of the reference mark relative to the pixel rows of the image sensor, moving the specimen chuck along only the x-direction of the XY translation stage to place the reference mark at a second position within the field of view of the image sensor and taking image data to determine a second pixel row number for the position of the reference mark relative to the pixel rows, and, after said steps of placing and moving, rotating the image sensor to place the reference mark at a third position having a third pixel row number that is between said first pixel row number and said second pixel row number, without rotating the XY translation stage; wherein, after said step of rotating the image sensor to place the reference mark at a third position, said steps of (i) placing the reference mark at a first position, (ii) moving the specimen chuck along only the x-direction, and (iii) rotating the image sensor to place the mark at a third position are repeated until the pixel rows are substantially parallel with the x-direction of the XY translation stage.

Contrary to the prior art, the present invention does not seek to align a desired x'-direction with the x-direction of the XY stage, but instead permits a desired x'-direction for scanning across the specimen to be out of alignment with the x-direction of movement of the XY translation stage. The present invention rotates the image sensor to achieve the desired alignment, thereby creating an offset between the XY directions of the translation stage and the pixel rows and columns of the image sensor. The present invention also determines the angle of offset or slope of the desired x'-direction relative to the x-direction of the XY translation stage. With the angle of offset/slope known, and the pixel rows aligned with the desired scanning direction, i.e., the x'-direction, the XY translation stage can be controlled to move the specimen relative to the image sensor to different positions along the desired x'-direction without a substantial shift of the image sensor relative to the specimen in a y'-direction, the y'-direction being orthogonal to the x' direction of the specimen.

The present invention also provides a method to accurately and precisely align the pixel rows of an image sensor with the x-direction of the XY translation stage. This alignment then leads to a method for determining the angle of offset/slope of a desired x'-direction of a specimen relative to the x-direction of the XY translation stage.

The general processes of the present invention are disclosed in various embodiments herein and, once the general physical conditions of the process are established, the process can be carried out in an automated manner with an appropriately configured microscopy system and related computer processing and microscopy techniques such as computer vision, motion control and measurement, and the like. As used herein "computer vision" is to be understood as covering algorithms for image processing, pattern recognition, computer vision or other known techniques for image analysis. First, aspects of the general microscope apparatus are disclosed, with methods of the present invention being disclosed thereafter.

<FIG> show an embodiment of a reflected light microscope system <NUM> as an example of the present invention, noting that the invention is equally applicable to other types of microscopes such as transmitted light microscopes, inverted microscopes, electron microscopes and others. Standard parts of the typical microscope system <NUM> include a stand <NUM>, a vertical illuminator <NUM>, a camera <NUM>, a lens tube <NUM>, a nosepiece <NUM>, an objective lens <NUM>, a z-axis focus arm <NUM> and an XY translation stage <NUM>. These components are well known to those familiar with the art. A camera rotator <NUM> works together with the XY translation stage <NUM> to achieve unique methods of scanning and/or imaging in accordance with this invention.

XY translation stages are well known in the art. They can be driven by stepper, servo, or linear motors, among others. The configuration of an XY translation stage is typically that of affixing one single axis stage to the z-axis focus arm <NUM> and affixing a second single axis stage to the first stage with axis of translation being <NUM> degrees to each other, though minor errors in orthogonal alignment of the X and Y stage are experienced in practice. Orthogonal alignment is a generally known term and addresses the fact that, for the two stages to travel precisely along the x and y axes, the line of travel for the y-axis must be orthogonal to the line of travel of the x-axis. If the two travel lines are not orthogonal, x-axis travel creates a position error in the y-direction. An orthogonality error can be expressed as the degrees of offset between the theoretical x-axis direction and the direction of empirical x-axis travel in light of the position error that occurs in the y-direction. It can also be expressed as a y position offset per x-direction travel length (e.g., <NUM> micron y position shift per <NUM> x-direction travel).

<FIG> provide an example of a typical configuration of an XY translation stage. <FIG> represents the front view of a single axis stage 18a comprising a drive motor <NUM>, a drive screw <NUM>, and a stand <NUM> with cushion blocks 26a and 26b. The cushion blocks 26a and 26b contain bearings and retainers (not shown) to support the drive screw <NUM>, which is attached to the drive motor <NUM> by a coupling (not shown). <FIG> is a top down view of the single axis stage 18a. In this embodiment, guide rods 28a and 28b are present to provide stability and guidance to the travel of specimen chuck <NUM>. The specimen chuck <NUM> contains a linear bearing (not shown) through which the guide rods travel, and a ball nut <NUM> that propels the chuck <NUM> along the drive screw <NUM> depending upon the direction of rotation of the drive screw <NUM>. The stand <NUM> is provided with mounting holes 34a and 34b to attach to the z-axis focus arm. A first single axis stage 18a and a second single axis stage 18b can be joined to create an XY translation stage as generally known in the art and shown in <FIG>, wherein stage 18b holds stage 18a, which holds the specimen chuck <NUM>.

This particular XY translation stage is provided as an example only, and it will be appreciated that other configurations of XY translations stages existing or hereafter created may be found useful in the present invention. As will be appreciated from the disclosure herein, it is only necessary that a translation stage used for the present invention be capable of allowing for precise control of the XY stage and providing precise position information of the X and Y stages. For example, in translation stages utilizing screw drives, feedback may be in the form of rotary encoders providing a signal that is directly proportional to the distance traveled. In other translation stages, a linear encoder may be used to provide direct feedback of the position of the stage.

<FIG> provide more details of a camera rotator <NUM>, such as that shown in <FIG>. It should be noted that this configuration is shown for example only and that other configurations resulting in the rotation of the camera with respect to the XY stage are possible. Here a rotator housing <NUM> is provided with flange <NUM> so that it may be attached to lens tube <NUM> and held in place by locking screws 40a and 40b. A rotational camera mount <NUM> is held in the housing <NUM> and free to rotate within bearings <NUM>. A connector <NUM> serves to attach the camera <NUM> to the camera mount <NUM>. A drive motor <NUM> together with a pulley <NUM>, a drive belt <NUM> and camera rotator pulley <NUM> provide means to rotate the camera <NUM>. A rotary encoder, potentiometer, stepper motor control or other similar device can be used so that the angular rotation α is known. These would provide a signal output (for example to processor <NUM>) that is proportional to the degree of rotation, such that a highly accurate angle of offset is determined from that signal. In some embodiments, a rotary encoder is provided within the motor assembly to provide the exact degree of rotation of the camera.

It should be appreciated that all adjustable parts of the microscope can be controlled by additional appropriate hardware and one or more processors. A processor <NUM> is shown here as controlling the camera rotator <NUM>, the camera <NUM>, and the XY translation stage <NUM> (and the z-axis focus arm <NUM>), and appropriate hardware and software is implicated and generally denoted by processor <NUM>. It will be appreciated that multiple processors could be employed, and the same in encompassed by the simple use of processor <NUM> in the Figures. Other aspects of the microscope system <NUM> can also be so controlled. The software is programmed to control X, Y and Z movement of the XY translation stage <NUM>, as well as rotation of the camera <NUM> and the activation of the camera to record image data through the image sensor <NUM>. Focusing can be automated by known methods as well. As is known in the art, the stage travel can be precisely known and controlled with rotary or linear encoders. The camera rotation can be precisely known by rotary encoders, potentiometers, stepper motor control or other. These precise positioning devices are used to provide input to software assisting in carrying out the invention. A computer, programmable controller or other processor, such as processor <NUM>, is employed to analyze the inputs and provide control output to the stage and rotator.

The camera <NUM> may contain an image sensor <NUM> (such as a CCD or CMOS sensor) used to capture images (or image data) of a specimen S. As used herein, an "image" does not require the actual production of an image viewable by an observer, and it simply requires the taking of digital data that can be used to create the desired image. Thus "image" and "image data" are used herein without significant distinction. An image sensor is comprised of pixels. A typical sensor may have between <NUM> and <NUM> megapixels or more. The size of the pixels vary and are typically between <NUM> and <NUM> micrometers (um) and a typical sensor size may be between less than <NUM> square mm to greater than <NUM> square mm. A representation of the image sensor <NUM> is shown in <FIG>. It is important to realize that the image sensor's field of view is a function of the magnification of the microscope objective lens <NUM>. An image sensor that is <NUM> x <NUM> will have a field of view that is <NUM> x <NUM> at 100x magnification. It can be appreciated that the magnified field of view is much smaller than the specimen to be examined.

As mentioned, the process carried out includes two major steps, a step of rotating the image sensor such that the pixel rows of the image sensor are substantially parallel with the desired x'-direction (i.e., the desired scanning direction) of the specimen, and a step of determining the angle of offset of the desired x'-direction as compared to the x-direction of the XY translation stage. In different embodiments herein, sometimes these steps are separate and distinct, and sometimes they overlap. In accordance with one embodiment, a process for aligning the pixel rows of the image sensor with the X-direction of the XY translation stage is first practiced to provide an accurate reference position of the image sensor prior to rotating it to have pixel rows aligned with the x'-direction of the specimen.

With the pixel rows substantially parallel to the x'-direction, and with the angle of offset known, advantageous scanning across the specimen can be achieved by establishing a first position for the specimen relative to the image sensor, the first position placing at least a portion of the specimen within the field of view of the image sensor; and moving the specimen with the XY translation stage to a second position along the desired x'-direction, wherein the second position places at least a second portion of the specimen within the field of view of the image sensor, and the second position is not substantially shifted in a y'-direction of the specimen, the y'-direction being orthogonal to the x'-direction of the specimen. In the step of moving, the movement is based upon the angle of offset determined in said step of determining.

In some embodiments, it is precise to state that an angle of offset is employed, but it will be appreciated that a slope of the desired x'-direction as compared to the x'-direction of the XY translation stage could instead be employed, and, for purposes herein the "angle of offset" can be expressed or conceptualized as either a slope (m) or an angle (degrees), the angle of a line of slope m relative to a base line being tan-<NUM>(m). That is, knowing slope, the angle can be calculated, and vice versa.

A first embodiment of the invention is described with reference to <FIG>, wherein relevant portions of a microscope system <NUM> are shown and help explain the general scanning conditions being addressed by the present embodiment. A specimen S is positioned on a specimen chuck <NUM> positioned in proximity of an image sensor <NUM> of camera <NUM>. The image sensor <NUM> is in a fixed position rotatable about its center axis so that the XY translation stage <NUM>, represented by the guide rods, and the specimen S, is moveable to position the specimen S under the image sensor <NUM>. In some embodiments, the camera <NUM> and image sensor <NUM> are part of a microscope, and it will be appreciated that the image sensor <NUM> can be employed to record image data reaching the image sensor <NUM> through the objective lens <NUM> and other well-known microscope components. Those of ordinary skill in the art do not require further disclosures beyond the schematics presented here in order to appreciate how the use of microscopes are implicated.

In this particular example of <FIG>, it is desired to scan the specimen S along a desired x'-direction of the specimen. Notably, the desired x'-direction of the specimen is angularly offset from the x and y movement directions of specimen chuck <NUM> (as represented by the x and y arrows), and, therefore, the desired x'-direction defines a slope relative to the x- and y-directions of the stage <NUM>. In this example the direction of x' travel is indicated by the line drawn between alignment mark <NUM> and alignment mark <NUM>. For purposes of disclosure, the angle of offset is shown to be quite significant, and it can be, but it should also be appreciated that the present invention will often be employed where the angle of offset is very small, as angles of offset of even a tiny fraction of a degree can cause significant problems for scanning and/or stitching images at high magnification.

As seen in <FIG>, and as generally known in the art, the image sensor <NUM> includes pixel rows and pixel columns, which are schematically represented as pixel rows P(<NUM>,<NUM>) to P(j,<NUM>) and pixel columns P(<NUM>, <NUM>) through P(<NUM>, i). As noted in the background, the camera <NUM> and its image sensor <NUM> are typically mounted in an attempt to place the pixel rows parallel to the x-direction of the translation stage <NUM>/specimen chuck <NUM>, thus also placing the pixel columns parallel to the y-direction of the translation stage <NUM> and specimen chuck <NUM>. However due to machining tolerances and limitations in achieving perfect alignment whether through automated or, more typically, visual methods, the rows and columns of the image sensor <NUM> are usually out of alignment with the x- and y-directions of the translation stage <NUM>. Thus, in this embodiment, the sensor rows and columns are not parallel to either the x' or y' desired direction of scanning.

In contradistinction to the prior art, the present invention rotates the camera <NUM> and thus the image sensor <NUM> contained within the camera <NUM> so as to place the pixel rows in a position parallel with the desired x'-direction of the specimen. In some embodiments, relative rotation can be accomplished by rotating the image sensor, the camera holding the image sensor, or the microscope holding the camera or through any other appropriate manipulation of a component of the system.

In the embodiment of <FIG>, the angle of offset is determined by assessing a slope between two reference marks, herein alignment marks, on the specimen S. The rotation of the image sensor to place pixel rows substantially parallel to the desired x'-direction can be practiced before or after assessing the slope, and various methods for such rotation can be practiced.

In <FIG>, specimen S is seen to have two alignment marks <NUM> and <NUM> inscribed. The direction of the line through marks <NUM> and <NUM> defines the desired direction of scanning x'. For reference, orthogonal center lines are shown through image sensor <NUM>, and their intersection marks the image sensor center C. <FIG> illustrates moving specimen chuck <NUM> to a position such that alignment mark <NUM> is positioned in the center of image sensor <NUM>. This movement and positioning is determined using computer vision and standard translation stage motion control. The coordinates of the XY translation stage <NUM> are recorded and set as the origin for future movements. For example, the processor <NUM> can record and analyze position and movement data in accordance with the teaching herein. Using computer vision and motion control (e.g., via processor <NUM>) the XY stage <NUM> is repositioned so that alignment mark <NUM> is centered in the image sensor <NUM> as shown in <FIG>. The distance to move the stage in the x-direction and y-direction is determined and recorded or otherwise retained for further processing in accordance with the teaching herein. <FIG> illustrates this movement and is marked with the change in the x-direction as ΔX and the change in the y-direction as ΔY.

Although the alignment marks <NUM>, <NUM> are focused onto the center C of the image sensor <NUM> in order to assess ΔX and ΔY, it will be appreciated that it is possible to designate any pixel or set of pixels as the target pixel(s) for placing the alignment marks and assessing ΔX and ΔY. Thus it is sufficient to place the alignment marks <NUM> so as to overlap with one or more target pixels of the image sensor <NUM>, and thereafter move the specimen to place the alignment mark <NUM> so as to overlap with the same one or more target pixels; thereafter assessing the x and y movement to obtain ΔX and ΔY.

In some embodiments, the alignment marks are smaller than a pixel, and are thus targeted on a single pixel to assess ΔX and ΔY. In other embodiments, the alignment marks encompass multiple pixels. In some embodiments, the alignment marks encompass multiple pixels, and the center of the alignment mark is calculated and used for positioning in a target pixel (such as the center C used in the example). The center can be calculated through computer vision.

Instead of using alignment marks purposefully placed on the specimen, in some embodiments it is possible to employ component features on the specimen S such as micro circuitry components (in some embodiments) or photolithographic features (in other, non- limiting embodiments) that extend in the desired x'-direction. Identifiable component features would be used in the same manner as alignment marks.

Knowing ΔX and ΔY provides the slope (m), which is ΔY/ΔX. With the slope, the desired direction of movement x' is defined as compared to the x-direction and y-direction of the XY translation stage. The line defined by slope ΔY/ΔX and going through alignment marks <NUM>, <NUM> forms an angle α relative to the line extending in the x-direction and extending through the alignment mark <NUM>. Referring to <FIG>, the angle α can be calculated as tan-<NUM>(m). Knowing the angle α, any movement in the x'-direction from any point of origin (e.g., a point encompassed by alignment mark <NUM>) can be calculated. For example, in <FIG>, to move a distance of d' in the x'-direction from the point of origin at alignment mark <NUM> the specimen chuck can move: ΔX = d'(cos(α)); ΔY = d'(sin(α)). The origin can be set at any location within the plane defined by the maximum travel of the translation stage <NUM> in the x-direction and the y-direction. It can be appreciated that the same procedures can be used to calculate movement in the y'-direction. It can also be appreciated that other mathematical techniques may be used to calculate travel in the x'-and y'-directions.

Being able to move precisely to different positions along the x'-direction without a substantial shift in the y'-direction allows for accurate scanning in the x'-directions and facilitates accurate stitching of multiple images or image data recorded by the image sensor, particularly when the rows of pixels of the image sensor are substantially parallel to the x'-direction. Thus, in the present embodiment, either before or after determining the slope/angle of offset as noted above, the image sensor is rotated to orient the rows of pixels substantially parallel to the x'-direction, and some methods for doing so are next disclosed.

In the particular method represented in the drawings, and particularly <FIG>, the sensor is rotated by rotation Ra so as to align the pixel rows with desired scanning direction x'. In this embodiment, the rotation occurs after the determination of ΔX and ΔY, but it will be appreciated that, in other embodiments, the image sensor could first be aligned and then ΔX and ΔY can be determined through the movements described above. Rotation techniques are disclosed herein below.

In some embodiments, as generally represented in <FIG>, the rotating of the image sensor includes identifying an axis-defining feature <NUM> on the specimen S, the axis-defining feature <NUM> having detectable shape running in a desired x'-direction. The axis-defining feature <NUM> is so named because it is provided to define the desired scanning direction x'. Computer vision is used to orient the pixel rows parallel to the x'-direction of the specimen based on the detectable shape of the axis-defining feature <NUM>. A rectangular shape is shown as the axis-defining feature <NUM>.

Another rotation technique is shown in <FIG>, and includes forming a mosaic of overlapping images (m1, m2, m3, m4) encompassing both first alignment mark <NUM> and second alignment mark <NUM>, and using computer vision to calculate a reference line <NUM> between the two alignment marks (this line also being the desired x'-direction) and align the pixel rows of the image sensor with the reference line <NUM>. In <FIG>, the specimen has been moved relative to the image sensor <NUM> to take a plurality of images (m1, m2, m3, m4), and these images (i.e., image data) are capable of being processed by a computer to define the reference line <NUM> between the alignment mark <NUM> and the alignment mark <NUM>. With this composite of image data, computer vision is employed to align the pixel rows of the image sensor with the reference line <NUM> as calculated by the computer (e.g., processor <NUM>).

For example, from the position of image m1, taking into account the position of alignment mark <NUM>, the specimen is moved in the x-direction in incremental distances less than the width of the field of view of the image sensor. Here the distance is <NUM>% of the width (i.e., moved <NUM> pixels in a total of <NUM>) just for ease of depicting the concept in drawings. However, in some embodiments, these increments (including y increment movements described below) can be between <NUM> and <NUM>% of the (width or height dimension of the) field of view. In other embodiments, the increments are between <NUM>% and <NUM>% of the field of view, and in other embodiments, between <NUM> and <NUM>% of the field of view. The specimen is moved at such increments in the x-direction until it has been moved a distance suitable for aligning the image sensor <NUM> under the alignment mark <NUM>. At each incremental movement an image is taken (e.g., m1, m2, m3, m4). The specimen is then moved in the y-direction until alignment mark <NUM> is within the field of view of the sensor. An image is taken at each increment (e.g., m5, m6). Using standard image stitching techniques, a composite image is obtained showing alignment marks <NUM> and <NUM> in the composite image. Again, with this composite of image data, computer vision is employed to align the pixel rows of the image sensor with the reference line <NUM>.

With the pixel rows aligned (regardless of the method employed to do so) the slope/angle of offset can be employed as noted with respect to <FIG> to accurately move/scan alone the x'-direction.

With respect to orienting pixel rows parallel to the desired x'-direction, it will be appreciated that a perfectly parallel relationship is likely theoretical only, especially when considering the potential for working at high magnification (where small angular offsets are more easily appreciated). The present invention seeks to align the pixel rows with the desired x'-direction so that there is extremely low or no degree of offset between the x'-direction of the specimen and the direction the pixel rows extend. This is similar to the concerns of "orthogonality error" described above with respect to XY translation stages. In some embodiments, it is sufficient herein that the pixel rows be less than <NUM> degrees off of the desired x'-direction. In some embodiments, the pixel rows are less than <NUM> degrees off of the desired x'-direction, in other embodiments, less than <NUM> degrees, in other embodiments, less than <NUM> degrees, in other embodiments, less than <NUM> degrees, in other embodiments, less than <NUM> degrees, in other embodiments, less than <NUM> degrees, in other embodiments, less than <NUM> degree, and, in other embodiments, less than <NUM> degrees. In sum, the recitations herein regarding alignment do not require absolute perfect alignment, but rather a substantial alignment (or substantially parallel relationship) suitable for the purpose for which the present invention is provided. In some embodiments, the invention serves to substantially reduce orthogonality error evident at high magnifications, even on the order of 100x or higher. In particular, the present invention provides a highly accurate alignment suitable for scanning along a desired x'-direction without a significant shift in the y'-direction, even at high levels of magnification.

A second embodiment of the invention is shown in <FIG>, and relates first to a method of aligning the image sensor <NUM> with the translation stage <NUM> so that the pixel rows are substantially parallel to the x-direction. A first step of this embodiment thus requires the sensor <NUM> be aligned with the movement of the translation stage <NUM> (i.e., pixel rows and columns parallel to the x-direction travel and the y-direction travel). <FIG> shows the translation stage with the specimen chuck <NUM> at a position left of the image sensor <NUM>. In some embodiments, the specimen chuck 30a has a reference mark <NUM> imprinted thereon provided to be viewed and located by computer vision. In other embodiments, a specimen S placed on the specimen chuck 30a has a reference mark thereon, such that this method envisions either the specimen chuck or a specimen placed thereon including a reference mark, but the method is specifically disclosed with respect to images showing the reference mark <NUM> on the specimen chuck 30a. When implementing this method with a reference mark on a specimen placed on the specimen chuck, the relative positions between the chuck and the specimen should not be allowed to change during movement of the specimen chuck.

Per <FIG>, the specimen chuck <NUM> is moved in the x-direction until reference mark <NUM> is in the field of view of the image sensor <NUM>. As noted previously, each pixel has a unique location designated by P(j,i). In <FIG> that mark <NUM> has been imaged and, for the purpose of example, is located in P(<NUM>,<NUM>). As seen in <FIG>, the specimen chuck <NUM> is moved in the x-direction by the XY translation stage <NUM> so that reference mark <NUM> is laterally moved (x-direction) relative to the image sensor <NUM> but still in the image sensor field of view. The reference mark <NUM> is now position in pixel P(<NUM>,<NUM>). It should be noted that the example shown is not to actual scale. Image sensors can have <NUM>,<NUM> of rows of pixels and <NUM>,<NUM> of columns of pixels.

After this first lateral movement and imaging, the goal is to rotate the camera so that when the image sensor <NUM> scans across the field of view of the specimen chuck <NUM>, the reference mark <NUM> is imaged in the same pixel row or rows as it passes across the sensor, i.e., there is no substantial change in relative positions in the y-direction. <FIG> shows the image sensor <NUM> and the relative locations of the reference mark <NUM> in the image of <FIG> (mark at P(<NUM>,<NUM>)) and the image of <FIG> (mark at P(<NUM>,<NUM>)). Center lines shows the x- and y-direction of movement of the specimen chuck <NUM> (as defined by the translation stage <NUM>). With the image sensor <NUM> oriented as in <FIG>, there is a very large y-direction shift during relative movement of the image sensor <NUM> and the specimen chuck <NUM>, and this shift can be reduced and effectively eliminated by a method involving averaging row numbers of the reference mark <NUM> at the two positions. In this example, the reference mark <NUM> is in row <NUM> in the image of <FIG>, and in row <NUM> in the image of <FIG>. A line which represents the average (i.e., (<NUM>+<NUM>) ÷ <NUM> = <NUM>) is shown across row <NUM> in <FIG>. Assuming that the specimen chuck <NUM> is still in the second position as indicated in <FIG>, the camera <NUM> and image sensor <NUM> is rotated by camera rotator <NUM> by rotation Rb (see <FIG>) so the reference mark <NUM> is in row <NUM>. As seen in <FIG>, this step places the rows of pixels in the image sensor <NUM> closer to parallel with the x-direction of the specimen chuck <NUM>. These steps of imaging the mark across the image sensor at two locations and then rotating the camera is repeated until the desired substantial alignment is reached between the pixel rows of the image sensor and the x-direction of the translation stage. In some embodiments, the steps are repeated until the center of the reference mark <NUM> (as determined by computer vision) remains in a single row of pixels when scanned across the entire width of the image sensor <NUM>. Notably, averaging row numbers of the reference mark <NUM> at the two positions is only one way to iteratively arrive at a highly precise alignment. It is sufficient in other embodiments, that the rotation represented by Rb simply place the reference mark <NUM> in a row number between the two row numbers for the reference mark upon movement of the image sensor between the two positions.

In some embodiments, the reference mark <NUM> is smaller than a pixel, and is thus targeted on a single pixel to perform this alignment process. In other embodiments, the reference mark <NUM> encompasses multiple pixels. In some embodiments, the reference mark <NUM> encompasses multiple pixels, and the center of the reference mark <NUM> is calculated and used for positioning in a target pixel (such as the center C used in the example). The center can be calculated through computer vision.

In some embodiments, the reference mark <NUM> is positioned close to the edge of the field of view, but still within the field of view of the image sensor, in each placement. This provides a more accurate assessment because it employs a longer x-direction travel by which to assess the y-direction shift. For example, per the figures, the reference mark <NUM> is positioned so that, in the image shown in <FIG>, the mark is positioned to the left of center close to the edge of the field of view but still within the field of view, while the reference mark <NUM> is then re-positioned so that, in <FIG>, the reference mark <NUM> is positioned to the right of center close to the edge of the field of view but still within the field of view.

After aligning the image sensor <NUM> to the x movement of the specimen chuck <NUM>, the image sensor <NUM> is rotated to align image sensor <NUM> with a desired x'-direction for scanning across the specimen. In this embodiment specimen S1 represents a specimen that with axis-defining feature <NUM> imprinted on it with distinct axis-defining characteristics which define the desired x'- and/or y'-directions for scanning across the specimen S1. <FIG> shows the specimen S1 place on specimen chuck <NUM> prior to scanning, and line <NUM> is provided to visually represent that the image sensor <NUM> is aligned with the x and y travel of the specimen chuck <NUM>. As seen in <FIG>, the specimen chuck <NUM> is moved so that feature <NUM> is within the field of view of the sensor <NUM>. In <FIG> the image sensor <NUM> is rotate by a measured number of degrees, Rc, which is also α, so as to align the pixel rows of the image sensor with the x'-direction defined by feature <NUM>. As already noted, the image sensor rotation can be precisely known by rotary encoders, potentiometer, stepper motor control or other, such that the angular rotation α is known.

Alternatively, after aligning the image sensor <NUM> to the x movement of the specimen chuck <NUM>, alignment marks and a mosaic may be employed to identify the desired x'-direction and rotate the image sensor (as in <FIG>).

Notably, because the reference mark <NUM> is associated with the XY translation stage, the aligning of the pixel rows of the image sensor with the x-direction of the XY translation stage needs only be performed once, and the aligned position can be recorded for future use. Thus, a different specimen can be placed on the chuck with a different orientation and different axis-defining feature, and the image sensor can be positioned with pixel rows aligned to the x-direction (per the process above) and then rotated to the axis-defining feature to find the angle of offset of desired x'-direction of the new specimen.

Starting with the image sensor pixel rows aligned with the x-direction of the XY stage, the subsequent angular rotation of the camera Rc (to place the pixel rows in the desired x'-direction) is equivalent to α described with respect to <FIG>, and any movement d' in a desired x'-direction can now be determined by: ΔX = d'(cos(α)); ΔY = d'(sin(α)).

Regardless of the methods herein employed to align the pixel rows with a desired x'-direction and to determine the slope/angle of offset of that x'-direction relative to the x-direction of the XY translation stage, once the pixels are so aligned and the slope/angle is determined, the specimen S can be moved in such a manner that a left border of a first image can be accurately stitched to a right border of a second image, where "left" and "right" are defined in the x'-direction. For example, in <FIG>, a first imaging position is shown by the positioning of image sensor <NUM> at p1 and a second imaging position is shown by the positioning of image sensor <NUM> at p2, wherein the left border of imaging sensor <NUM> at p1 is aligned with the right border of image sensor <NUM> at p2, again without any significant shifting in the y' direction. It should also be appreciated that it is acceptable to take images p1 and p2 with an overlapping region, i.e., where columns of pixels at the left side of the image at p1 overlap with columns of pixels at the right side of the image at p2. The overlapping portions can assist in the accurate stitching, as known. The present invention, however, by causing an alignment of pixel rows with the desired x'-direction, facilitates an accurate border-to-border stitching of images, which can decrease the entire imaging process by requiring less images and less computation.

It should be appreciated that the various steps herein can, and preferably are performed automatically by the microscope system <NUM>. The movement of moveable and rotatable components would be handled by appropriate software and hardware, and computer vision can be employed to identify detectable features such as the alignment marks <NUM>, <NUM>, the reference mark <NUM>, and the axis-defining features <NUM> that govern the orientation of the image sensor <NUM> relative to the specimen. The focusing and taking of image data can also be automated. This all is represented in the figures by processor <NUM>. Thus, the present invention allows the specimen and XY translation stage to be out of alignment (i.e., desired x'- and y'-directions of the specimen are out of alignment with the x- and y-directions of the translation stage) and does not require manipulation of the specimen to remedy this lack of alignment. The system self-calibrates, and, knowing the width of an image sensor, can progressively scan the specimen and take discreet images having aligned borders to then be stitched together to form the complete desired image of the specimen.

The general concepts of the present invention are adequately disclosed to those of ordinary skill in the art by the figures and description herein. The detailed disclosure is provided to broadly disclose those general concepts, but is not necessary for those of ordinary skill in the art to fully implement the concepts of the present invention. This is true even though the drawings are schematic.

Claim 1:
A microscopy method for imaging a specimen (S) comprising two alignment marks (<NUM>, <NUM>) along a desired x'-direction of the specimen (S), the desired x'-direction defined by the alignment marks (<NUM>, <NUM>) of the specimen (S), the specimen (S) being placed on an XY translation stage (<NUM>) and movable by the XY translation stage (<NUM>) so as to have portion thereof placed within the field of view of an image sensor (<NUM>), wherein the XY translation stage (<NUM>) is movable in an x-direction and a y-direction to move the specimen (S) relative to the image sensor (<NUM>), the image sensor having a multitude of pixels arranged to define pixel rows and pixel columns, the desired x'-direction of the specimen (S) being angularly offset from the x-direction of the XY translation stage (<NUM>) so as to define a slope and angle of offset relative thereto, the image sensor (<NUM>) viewing only a discrete segment of the specimen (S) at a time, the method comprising the steps of:
rotating the image sensor (<NUM>) such that the pixel rows are substantially parallel with the desired x'-direction of the specimen (S), without rotating the XY translation stage (<NUM>);
determining the angle of offset of the desired x'-direction as compared to the x-direction of the XY translation stage (<NUM>);
establishing a first position for the specimen (S) relative to the image sensor (<NUM>) as rotated in said step of rotating, said first position placing at least a portion of the specimen (S) within the field of view of the image sensor (<NUM>); and,
after said step of determining and said step of establishing, moving the specimen (S) with the XY translation stage (<NUM>) to a second position along the desired x'-direction by moving the XY translation stage (<NUM>) in the x-direction, wherein the second position places at least a second portion of the specimen (S) within the field of view of the image sensor (<NUM>), and the second position is not substantially shifted in a y'-direction of the specimen (S), the y'-direction being orthogonal to the x'-direction of the specimen, wherein said step of moving is based upon the angle of offset determined in said step of determining.