Patent Description:
The present application generally relates to whole-slide imaging, and, more particularly, to sub-pixel scanning of samples on microscope slides.

Current pixel-shifting technology enhances resolution by moving an image sensor in increments that are smaller in length than the width of a pixel. For example, to sample a subpixel that is a quarter of the size of a pixel, a first image including the subpixel is imaged, the image sensor is moved to the left by half a pixel and a second image including the subpixel is imaged, the image sensor is moved up by half a pixel and a third image including the subpixel is imaged, and the image sensor is moved to the right by half a pixel and a fourth image including the subpixel is imaged. The intensities in the four images are then combined to generate the subpixel. <CIT> refers to infrared imaging microscope, particularly of the type used to carry out FT-IR measurement, having a detector in the form of a small detector array of individual detector elements. The outputs of the detector elements are fed in parallel to processing means which process the output signals. The use of a small array means that the outputs can be processed without the need for complex multiplexing or perhaps no multiplexing at all thus avoiding the reduction in signal to noise ratio which is associated with large scale multiplexing. The small detector array will generally have between <NUM> and <NUM> detector elements. Typically the upper limit will be <NUM> and a preferred arrangement has <NUM> detector elements. <CIT> relates to a photosensor array including two adjacent and parallel rows of photosensors deposited on a common substrate, the photosensors being spaced apart uniformly, with the two rows being staggered longitudinally by half a spacing so that light which would fall on the boundary between two sensors in one row falls directly on a sensor in the other row.

In an embodiment, a slide scanning device is disclosed that comprises: a stage which supports a microscope slide having a sample; a plurality of line sensors, wherein each of the plurality of line sensors comprises a plurality of pixel sensors, and wherein each of the plurality of line sensors, in a longitudinal direction of the line sensor, is offset from an adjacent one of the plurality of line sensors by a fraction of a length of each pixel sensor; an objective lens that provides a same field of view of the sample to each of the plurality of line sensors, successively, such that, for each of a plurality of positions on the sample, each of the plurality of line sensors senses the same field of view of the position and generates a line image of the same field of view of the position at the line sensor's respective offset; and at least one hardware processor that, for each of the plurality of positions on the sample, combines the line images of the same field of view, generated by the plurality of line sensors at their respective offsets, to produce a plurality of subpixels for each of at least a subset of pixels within the line images of the same field of view, and generates an up-sampled line image of the position comprising the produced plurality of subpixels, and combines the up-sampled line images of each of the plurality of positions on the sample into an image. The plurality of line sensors may consist of N line sensors, wherein the fraction of the length of each pixel sensor, by which each of the plurality of line sensors is offset from an adjacent line sensor, is <NUM>/Nth. For each of the plurality of positions on the sample, the line images of the same field of view may comprise a line image for N offsets from zero to (N-<NUM>)/N. For each of the plurality of positions on the sample, combining the line images of the same field of view may comprise, for each of the at least a subset of pixels, summing intensity values for the pixel from each of the line images at their respective offsets. The at least one hardware processor may: combine the up-sampled line images of each of the plurality of positions on the sample into a plurality of image stripes; and align each of the plurality of images stripes with at least one adjacent one of the plurality of image stripes into a contiguous digital image.

In an embodiment, a method is disclosed that comprises using at least one hardware processor to: receive image data from a plurality of line sensors, wherein each of the plurality of line sensors comprises a plurality of pixel sensors, and wherein each of the plurality of line sensors, in a longitudinal direction of the line sensor, is offset from an adjacent one of the plurality of line sensors by a fraction of a length of each pixel sensor; control an objective lens that provides a same field of view of a sample to each of the plurality of line sensors, successively, such that, for each of a plurality of positions on the sample, each of the plurality of line sensors senses the same field of view of the position and generates a line image of the same field of view of the position at the line sensor's respective offset; for each of the plurality of positions on the sample, combine the line images of the same field of view, generated by the plurality of line sensors at their respective offsets, to produce a plurality of subpixels for each of at least a subset of pixels within the line images of the same field of view, and generate an up-sampled line image of the position comprising the produced plurality of subpixels; and combine the up-sampled line images of each of the plurality of positions on the sample into an image.

In an embodiment, a non-transitory computer-readable is disclosed having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to: receive image data from a plurality of line sensors, wherein each of the plurality of line sensors comprises a plurality of pixel sensors, and wherein each of the plurality of line sensors, in a longitudinal direction of the line sensor, is offset from an adjacent one of the plurality of line sensors by a fraction of a length of each pixel sensor; control an objective lens that provides a same field of view of a sample to each of the plurality of line sensors, successively, such that, for each of a plurality of positions on the sample, each of the plurality of line sensors senses the same field of view of the position and generates a line image of the same field of view of the position at the line sensor's respective offset; for each of the plurality of positions on the sample, combine the line images of the same field of view, generated by the plurality of line sensors at their respective offsets, to produce a plurality of subpixels for each of at least a subset of pixels within the line images of the same field of view, and generate an up-sampled line image of the position comprising the produced plurality of subpixels; and combine the up-sampled line images of each of the plurality of positions on the sample into an image.

In an alternative not covered by the claims, a slide scanning device is disclosed that comprises: a stage which supports a microscope slide having a sample; a line scan camera comprising a two-dimensional array of pixel sensors, wherein the two-dimensional array comprises columns parallel to a first axis and rows parallel to a second axis that is orthogonal to the first axis; an objective lens that provides a same field of view of the sample to each of a plurality of rows in the two-dimensional array, successively, such that, for each of a plurality of positions on the sample, each of the plurality of rows senses the same field of view of the position and generates a line image of the same field of view of the position; and at least one hardware processor that, for each of the plurality of positions on the sample, controls one or more of the stage, the objective lens, and the line scan camera, such that the field of view of the sample, provided to the plurality of rows in the two-dimensional array, moves across the plurality of rows in the two-dimensional array at a non-zero angle with respect to the first axis and the second axis, such that each row in the plurality of rows senses the same field of view at an offset equal to a different fraction of a length of each pixel sensor, combines the line images of the same field of view, generated by the plurality of rows in the two-dimensional array at their respective offsets, to produce a plurality of subpixels for each of at least a subset of pixels within the line images of the same field of view, and generates an up-sampled line image of the position comprising the produced plurality of subpixels, and combines the up-sampled line images of each of the plurality of positions on the sample into an image. The stage may be a motorized stage, and controlling one or more of the stage, the objective lens, and the line scan camera may comprise controlling the motorized stage to move relative to the objective lens. The plurality of rows may consist of N rows, wherein the angle with respect to the first axis is equal to an arctangent of <NUM>/N. For each of the plurality of positions on the sample, the line images of the same field of view may comprise a line image for N offsets from zero to (N-<NUM>)/N.

In an alternative not covered by the claims, a method is disclosed that comprises using at least one hardware processor to: receive image data from a line scan camera comprising a two-dimensional array of pixel sensors, wherein the two-dimensional array comprises columns parallel to a first axis and rows parallel to a second axis that is orthogonal to the first axis; control an objective lens that provides a same field of view of a sample to each of a plurality of rows in the two-dimensional array, successively, such that, for each of a plurality of positions on the sample, each of the plurality of rows senses the same field of view of the position and generates a line image of the same field of view of the position; for each of the plurality of positions on the sample, control one or more of a stage, the objective lens, and the line scan camera, such that the field of view of the sample, provided to the plurality of rows in the two-dimensional array, moves across the plurality of rows in the two-dimensional array at a non-zero angle with respect to the first axis and the second axis, such that each row in the plurality of rows senses the same field of view at an offset equal to a different fraction of a length of each pixel sensor, combines the line images of the same field of view, generated by the plurality of rows in the two-dimensional array at their respective offsets, to produce a plurality of subpixels for each of at least a subset of pixels within the line images of the same field of view, and generates an up-sampled line image of the position comprising the produced plurality of subpixels; and combines the up-sampled line images of each of the plurality of positions on the sample into an image.

In an alternative not covered by the claims, a non-transitory computer-readable is disclosed having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to: receive image data from a line scan camera comprising a two-dimensional array of pixel sensors, wherein the two-dimensional array comprises columns parallel to a first axis and rows parallel to a second axis that is orthogonal to the first axis; control an objective lens that provides a same field of view of a sample to each of a plurality of rows in the two-dimensional array, successively, such that, for each of a plurality of positions on the sample, each of the plurality of rows senses the same field of view of the position and generates a line image of the same field of view of the position; for each of the plurality of positions on the sample, control one or more of a stage, the objective lens, and the line scan camera, such that the field of view of the sample, provided to the plurality of rows in the two-dimensional array, moves across the plurality of rows in the two-dimensional array at a non-zero angle with respect to the first axis and the second axis, such that each row in the plurality of rows senses the same field of view at an offset equal to a different fraction of a length of each pixel sensor, combines the line images of the same field of view, generated by the plurality of rows in the two-dimensional array at their respective offsets, to produce a plurality of subpixels for each of at least a subset of pixels within the line images of the same field of view, and generates an up-sampled line image of the position comprising the produced plurality of subpixels; and combines the up-sampled line images of each of the plurality of positions on the sample into an image.

Other features and advantages of the present invention will become readily apparent to those of ordinary skill in the art after reviewing the following detailed description and accompanying drawings.

The structure and operation of the present invention will be understood from a review of the following detailed description and the accompanying drawings in which like reference numerals refer to like parts and in which:.

In general, the resolution of a digital imaging device is limited by the pixel size of the sensors and the resolution of the imaging optics. However, certain embodiments disclosed herein provide for a digital imaging device (e.g., digital slide scanner) that utilizes pixel-shifting to efficiently capture images at a higher resolution than could otherwise be captured by the size of the pixel-generating elements in its image sensor(s), while utilizing standard imaging optics at reasonable cost. In addition, the disclosed schemes can provide solutions for overcoming the field-of-view limit that is usually tied to the required pixel resolution of digital slide scanners, thereby resulting in faster scanning, especially in a line-scan mechanism.

After reading this description it will become apparent to one skilled in the art how to implement the invention in various alternative embodiments and alternative applications within the scope as set forth in the appended claims.

In a first embodiment, the image sensor comprises a plurality of line sensors which are logically offset from each other, in their longitudinal directions, by a distance equal to a fraction of a pixel length. Thus, as each successive line sensor receives light from the same portion of a sample, that line sensor generates a line of image data (also referred to herein as a "line image") that is slightly offset (i.e., by the fraction of a pixel length) from the line of image data generated by the preceding line sensor. While this offset can be a physical offset of a plurality of line sensors in space, in an alternative embodiment, the offset may instead be induced by shifting a single line sensor in space at a defined time interval.

<FIG> illustrates a sensor <NUM>, comprising three offset line sensors <NUM>. In a direction orthogonal to the direction of the scan motion, each of the line sensors <NUM> is offset from its adjacent line sensor(s) by a fraction of a pixel length. The direction of the scan motion indicates the direction that the field of view, being imaged, moves, relative to sensor <NUM>. The scan motion may be implemented my moving a sample (e.g., by moving a motorized stage on which the sample is supported) with respect to the optical path provided to the image sensor and/or by moving the optical path with respect to the sample (e.g., by moving the image sensor and/or objective lens of the scanning device).

Although <FIG> shows one line for each line sensor <NUM>, each line sensor <NUM> could comprise a set of multiple line sensors. For example, each line sensor <NUM> may comprise a set of three color sensors (e.g., three lines representing a trilinear sensor). In addition, although sensor <NUM> is illustrated with three offset line sensors (i.e., line sensors 110A, 110B, and 110C), sensor <NUM> may have any number of multiple line sensors <NUM>. Generally, if N offset line sensors <NUM> are used, each of the N line sensors <NUM> should be offset from its adjacent line sensor(s) <NUM> by <NUM>/Nth of the length of a pixel sensor <NUM>. It should be understood that each line sensor <NUM> captures the same line image as each of the other line sensors <NUM>, but offset by <NUM>/Nth of a pixel length with respect to the same line image captured by each adjacent line sensor <NUM>. This produces, for each line image in the whole slide image, N lines of the same image data, but, again, with the image data in each line image offset by <NUM>/Nth of a pixel length from the image data in the line image captured by an adjacent line sensor.

In the illustrated example with three offset line sensors <NUM> (i.e., 110A, 110B, and 110C), line sensor 110B is offset from line sensor 110A by one-third of a pixel length, and line sensor 110C is offset from line sensor 110B by one-third of a pixel length and offset from line sensor 110A by two-thirds of a pixel length. In the case with two offset line sensors <NUM>, each line sensor would be offset from the other line sensor by one half of a pixel length. Similarly, in the case with four offset line sensors <NUM>, each line sensor would be offset from its adjacent line sensor(s) by one-fourth of a pixel length.

Although sensor <NUM> is illustrated with offset line sensors offset towards a particular side (i.e., towards the right side of <FIG>), the offset line sensors may instead be offset towards the other side (i.e., towards the left side of <FIG>) without any change to the techniques described herein.

<FIG> illustrates how offset pixels are captured by pixel sensors <NUM>. Compared to a single line sensor with the same optics for low-resolution (i.e., native or non-sub-pixel) imaging, the speed of the scanning motion for a sensor <NUM> that utilizes N offset line sensors for high-resolution (i.e., sub-pixel) imaging should be N times slower. This is because each line image needs to be captured N times at N different offsets. Thus, in the illustrated embodiment with three line sensors <NUM>, in order to achieve the same line rate as a sensor with only a single line sensor, the speed of the scanning motion should be three times slower than the speed of the scanning motion for the sensor with only a single line sensor.

<FIG> illustrates the logical positions of pixel sensors 112A, 112B, and 112C, with respect to a lens <NUM> (e.g., an objective lens described elsewhere herein) and a sample <NUM> on a slide <NUM>, at times <NUM>, t2, and t3. As discussed above, the scanning speed of sensor <NUM> for high-resolution imaging may be set to three times slower than the scanning speed of a sensor consisting of only a single line sensor for low-resolution imaging, such that each of times <NUM>, t2, and t3 takes the time of the single line sensor to capture a single line of image data.

As illustrated in <FIG>, at time t1, position x1 of sample <NUM> is within the field of view provided by lens <NUM>, while lens <NUM> is at a focus height z1. The view of position x1 on sample <NUM> is provided by lens <NUM>, via an optical path, to line sensor 110A, including pixel sensor 112A of line sensor 110A. Thus, pixel sensor 112A generates a first pixel of image data representing a pixel-sized portion of sample <NUM> at position x1.

At time t2, position x1 on sample <NUM> remains within the field of view provided by lens <NUM>, while lens <NUM> remains at focus height z1. However, the view of position x1 on sample <NUM> has moved in the direction of the scan motion, such that the view of position x1 on sample <NUM> is provided by lens <NUM>, via an optical path, to line sensor 110B, including pixel sensor 112B of line sensor 110B. Thus, pixel sensor 112B generates a second pixel of image data representing a pixel-sized portion of sample <NUM> at position x1. This second pixel is offset from the first pixel by one-third of a pixel length.

At time t3, position x1 on sample <NUM> remains within the field of view provided by lens <NUM>, while lens <NUM> remains at focus height z1. However, the view of position x1 on sample <NUM> has moved in the direction of the scan motion, such that the view of position x1 on sample <NUM> is provided by lens <NUM>, via an optical path, to line sensor 110C, including pixel sensor 112C of line sensor 110C. Thus, pixel sensor 112C generates a third pixel of image data representing a pixel-sized portion of sample <NUM> at position x1. This third pixel is offset from the first pixel by two-thirds of a pixel length, and offset from the second pixel by one-third of a pixel length.

Accordingly, in a sensor <NUM> with three offset line sensors <NUM>, each pixel that is captured (i.e., representing a pixel-sized portion of sample <NUM>) is captured three times at three different offsets (i.e., zero offset, offset by one-third of a pixel length, and offset by two-thirds of a pixel length). Similarly, in a sensor with two offset line sensors <NUM>, each pixel would be captured twice at two different offsets (i.e., zero offset and offset by one half of a pixel length). In a sensor with four offset line sensors <NUM>, each pixel would be captured four times at four different offsets (i.e., zero offset, offset by one-fourth of a pixel length, offset by two-fourths of a pixel length, and offset by three-fourths of a pixel length).

In each case, each offset pixel may be combined, in a manner described elsewhere herein, to generate N subpixels. Thus, in the case of three offset line sensors <NUM>, three subpixels would be generated for each pixel. In the case of two offset line sensors <NUM>, two subpixels would be generated for each pixel. In the case of four offset line sensors <NUM>, four subpixels would be generated for each pixel.

While examples with two, three, and four line sensors <NUM> have been described, it should be understood that the techniques described herein can be extrapolated to any N line sensors <NUM> (e.g., five, ten, etc.) to achieve N subpixels for each pixel. During each time (e.g., t1, t2, t3) at which the same position (e.g., x1) on sample <NUM> is being imaged by one of the N offset line sensor <NUM>, lens <NUM> may remain at the same focus height (e.g., z1), such that each pixel is generated with the same focus. The focus height may be adjusted, if appropriate to maintain optimal focus (e.g., based on a focus map or other auto-focus technique), once the same position has been imaged by each of the N offset line sensors <NUM> in sensor <NUM>.

In an alternative not covered by the claims, a single line sensor <NUM> could be used to achieve the same effect as sensor <NUM> with N line sensors <NUM> by scanning each linear portion of the sample N times, while shifting the position, being imaged on the sample, N-<NUM> times (i.e., between each individual scan of the same linear portion). In such an alternative, each time the logical position of the sample is shifted relative to the line sensor <NUM>, the logical position is shifted by a length that is <NUM>/Nth of the length of a pixel sensor <NUM>, such that N line images are captured from an offset of zero to an offset of (N-<NUM>)/Nth of a pixel length. Once N line images have been generated for the same linear portion, the logical position of the sensor <NUM>, relative to the sample, is moved so that the next linear portion may be imaged N times, beginning from a zero offset and ending with an offset of (N-<NUM>)/Nth of a pixel length.

In an alternative not covered by the claims, the scanning motion follows a trajectory that is angled with respect to the image sensor. In other words, instead of an array of a plurality of offset line sensors, the image sensor may comprise one or more sensors that are flush (i.e., not offset), but which scan at an angled trajectory. Thus, standard image sensors can be adapted to achieve the same offset line images as the offset line sensors in the first embodiment.

<FIG> illustrates a sensor <NUM>, which scans at an angled trajectory, according to this alternative. As illustrated, sensor <NUM> comprises a line scan camera having a plurality of pixel sensors <NUM> arranged in a two-dimensional array with rows <NUM> and columns. Alternatively, sensor <NUM> may comprise an area scan camera.

In cases in which a line scan camera is used, sensor <NUM> may comprise a time delay integration (TDI) line scan camera. The Piranha XL™, from Teledyne DALSA Inc. in Waterloo, Ontario, Canada, is an example of a TDI line scan camera that may be used as the line scan camera in such an embodiment. TDI line scan cameras comprise multiple stages of line sensors (e.g., <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, etc.), which each capture the same line image. The intensity data from each line image are then summed to provide an output line image with a substantially better signal-to-noise (SNR) ratio. For example, line images captured by row 210A of pixel sensors <NUM>, row 210B of pixel sensors <NUM>, and so on, through row 210Z of pixel sensors <NUM>, would be integrated in this manner to produce a single output line image.

However, in cases which utilize a TDI line scan camera, the TDI line scan camera can be switched from a TDI mode to an area mode. In the area mode, the line images generated by rows <NUM> are not summed into a single output line image. Rather, in the area mode, each line image represents a different line of image data at a different position on the sample.

The motion of a position on the sample being imaged by sensor <NUM>, relative to sensor <NUM>, follows a direction at an angle θ with respect to an X axis that is orthogonal to the longitudinal direction of each row <NUM> and parallel to the longitudinal direction of the columns of sensor <NUM>. Angle θ should be equal to the inverse tangent of N-<NUM> (i.e., θ = arctan(<NUM>/N)), where N is the number of subpixels desired for each pixel. In order to maintain the same line rate as a sensor <NUM> which moves parallel to the X axis and does not implement the subpixel technique described herein, the speed of the scan motion should be <MAT> wherein v<NUM> represents the speed of the sensor moving parallel to the X axis and not capturing each pixel at N offsets.

As sensor <NUM> images a position on the sample along the angled trajectory, the relative movement between sensor <NUM> and the sample is set, such that each set of N rows is used to capture the same line of image data. For example, in the illustrated example where N=<NUM>, rows 210A, 210B, and 210C are each used to capture a line image at the same position x<NUM> along the X axis (e.g., at times <NUM>, t2, and t3, respectively). Similarly, rows 210X, 210Y, and 210Z are each used to capture a line image at the same position x<NUM> along the X axis, where x<NUM> > x<NUM>. The line image produced by each subset of N rows <NUM> represents the same linear portion of the sample, but offset at N different positions. In other words, the output is the same as in the first example illustrated in <FIG>. Specifically, for each position on the sample that is imaged by the image sensor, N linear images are produced that are each offset by <NUM>/Nth of a pixel length from the linear image captured by each adjacent line sensor <NUM> or row <NUM>. Where N=<NUM>, three line images would be produced for each position on the same, including a first line image having zero offset, a second line image having an offset of one-third of a pixel length, and a third line image having an offset of two-thirds of a pixel length.

Notably, sensor <NUM> captures fewer line images than would otherwise be captured if each row <NUM> was used to capture a different linear portion of the sample. Specifically, if sensor <NUM> comprises M rows <NUM> (e.g., an M-stage TDI line scan camera), sensor <NUM> outputs M/N line images at a time, rather than the M line images that would be output if each row <NUM> was used to capture a different linear portion of the sample.

<FIG> and <FIG> illustrate how three offset pixels from offset line images may be used to generate a subpixel that is one-third of a pixel length, according to the invention and to an alternative not covered by the claims. While <FIG> and <FIG> illustrate the use of three offset pixels to generate a subpixel that is one-third the length of a pixel length, it should be apparent how this technique may be used with any N offset pixels to divide a pixel into N subpixels that are each <NUM>/Nth of a pixel length.

<FIG> illustrates how rectangular pixels can be used to generate a combined image pixel <NUM>. Each line sensor <NUM> produces a line image <NUM>. Thus, for example, line sensor 110A produces line image 140A, line sensor 110B produces line image 140B, and line sensor 110C produces line image 140C. It should be understood that <FIG> only illustrates a partial segment of each line image <NUM>. In the illustrated embodiment, each of line images <NUM> comprise rectangular image pixels <NUM>. The length of image pixels <NUM> in the direction of motion may be controlled by adjusting the scan speed. As illustrated, when rectangular pixels <NUM> from each of line images <NUM> are combined, the combination results in a square image subpixel <NUM>. The combination of the rectangular pixels <NUM> may comprise summing the intensity values of portions of each of the pixels <NUM>, with the sum of those intensity values being used as subpixel <NUM>.

While the embodiment illustrated in <FIG> uses rectangular pixels to produce square subpixels <NUM>, the alternative illustrated in <FIG> uses square pixels to produce rectangular subpixels <NUM>. As illustrated in <FIG>, at time t1, pixel sensor 112A or 212A generates a first square pixel with zero offset. Next, at time t2, pixel sensor 112B or 212B generates a second square pixel with an offset of one-third the length of a pixel sensor <NUM> or <NUM>. Finally, at time t3, pixel sensor 112C or 212C generates a third square pixel with an offset of two-thirds the length of a pixel sensor <NUM> or <NUM>. The overlapping portions of the first, second, and third pixels are combined to generate rectangular subpixel <NUM>. For example, the combination of the pixels may comprise summing the intensity values of each of the first, second, and third pixels, with the sum of those intensity values being used as subpixel <NUM>. To obtain a square subpixel <NUM>, the same subpixel operation may be repeated in the orthogonal direction.

This summation of pixels into subpixel <NUM> may be repeated for each set of offset pixels (i.e., collectively representing a single pixel of image data) captured by each pixel sensor <NUM> or <NUM> to generate N subpixels for each single pixel of image data, to thereby up-sample the resolution of a whole slide image by N times. In the example, illustrated herein, in which N=<NUM>, three subpixels are generated for each pixel of image data, producing an up-sampled image with three times the resolution.

<FIG> illustrates a process <NUM> of generating subpixel resolution for a digital image (e.g., whole slide image), using the pixel-shifting technique of the first or second alternatives described herein. Process <NUM> may be referred to as "de-mosaicing," and may be implemented as software and/or hardware in a digital imaging device (e.g., as software stored in memory <NUM> and executed by processor(s) <NUM> of device <NUM>). Process <NUM> may be performed after or in parallel with the scanning of a sample (i.e., the generation of line images using sensor <NUM> or <NUM>).

In step <NUM>, it is determined whether or not all line images - each representing a position within a region of interest on a sample (e.g., a portion of the sample or the entire sample or slide) - have been acquired. If line images, representing additional positions, remain to be acquired (i.e., "NO" in step <NUM>), process <NUM> proceeds to step <NUM>. Otherwise, if all line images have been acquired (i.e., "YES" in step <NUM>), process <NUM> proceeds to step <NUM>.

In step <NUM>, a current position is changed to a next position, for which line images are to be acquired. It should be understood that, at the beginning of process <NUM>, the next position in step <NUM> will represent the starting position from which line images are to be acquired.

In step <NUM>, N offset line images of the current position are acquired. As described elsewhere herein, each line image represents an image of the same position, but offset by increments of <NUM>/Nth of a pixel length. Collectively, the N offset line images represent the current position at offsets from zero to (N-<NUM>)/Nth of a pixel length.

In step <NUM>, each of the N offset line images, acquired in step <NUM>, are stacked to generate N subpixels for each pixel in the resulting stacked line image. Each pixel in the offset line images may be stacked as described elsewhere herein (e.g., with respect to <FIG> and <FIG>). After step <NUM>, process <NUM> returns to step <NUM>.

Once it has been determined that all line images, representing an entire region of interest to be imaged on the sample, have been acquired in step <NUM>, process <NUM> proceeds to step <NUM>. In step <NUM>, the acquired line images are combined into image stripes or tiles. The resulting image stripes or tiles are N times the resolution of image stripes or tiles that have not been up-sampled using the subpixels generated in step <NUM>.

In step <NUM>, the up-sampled image stripes or tiles are aligned into a contiguous digital image (e.g., a whole slide image).

In step <NUM>, one or more filters are applied to the contiguous digital image to, for example, reduce noise (e.g., edge effects resulting from "stitching" the image stripes or tiles together, etc.), and process <NUM> ends.

<FIG> is a block diagram illustrating an example processor-enabled device <NUM> that may be used in connection with various alternatives described herein. The device <NUM> is presented as a digital imaging device (also referred to herein as a digital slide scanner, scanner system, or scanning system) that comprises one or more processors <NUM>, one or more memories <NUM>, one or more motion controllers <NUM>, one or more interface systems <NUM>, one or more movable stages <NUM> that each support one or more glass slides <NUM> with one or more samples <NUM>, one or more illumination systems <NUM> that illuminate the sample, one or more objective lenses <NUM> that each define an optical path <NUM> that travels along an optical axis, one or more objective lens positioners <NUM>, one or more optional epi-illumination systems <NUM> (e.g., included in a fluorescence scanner system), one or more focusing optics <NUM>, one or more line scan cameras <NUM> and/or one or more area scan cameras <NUM>, each of which define a separate field of view <NUM> on the sample <NUM> and/or glass slide <NUM>. The various elements of digital imaging device <NUM> are communicatively coupled via one or more communication busses <NUM>. Although there may be one or more of each of the various elements of digital imaging device <NUM>, for simplicity in the description that follows, these elements will be described in the singular except when needed to be described in the plural to convey the appropriate information. Alternative forms of digital imaging device <NUM> may also be used as will be understood by the skilled artisan.

One or more processors <NUM> may include, for example, a central processing unit ("CPU") and a separate graphics processing unit ("GPU") capable of processing instructions in parallel, or processor <NUM> may include a multicore processor capable of processing instructions in parallel. Additional separate processors may also be provided to control particular components or perform particular functions such as image processing. For example, additional processors may include an auxiliary processor to manage data input, an auxiliary processor to perform floating point mathematical operations, a special-purpose processor having an architecture suitable for fast execution of signal-processing algorithms (e.g., digital signal processor), a slave processor subordinate to the main processor (e.g., back-end processor), an additional processor for controlling line scan camera <NUM>, stage <NUM>, objective lens <NUM>, and/or a display (not shown). Such additional processors may be separate discrete processors or may be integrated with processor <NUM>.

Memory <NUM> provides storage of data and instructions for programs that can be executed by processor <NUM>. Memory <NUM> may include one or more volatile and/or non-volatile computer-readable storage mediums that store the data and instructions, such as a random access memory (RAM), a read only memory (ROM), a hard disk drive, a removable storage drive, and/or the like. Processor <NUM> may be configured to execute instructions that are stored in memory <NUM> and communicate via communication bus <NUM> with the various elements of the digital imaging device <NUM> to carry out the overall function of digital imaging device <NUM>.

One or more communication busses <NUM> may include a communication bus <NUM> that is configured to convey analog electrical signals and/or a communication bus <NUM> that is configured to convey digital data. Accordingly, communications from processor <NUM>, motion controller <NUM>, and/or interface system <NUM> via the one or more communication busses <NUM> may include electrical signals and/or digital data. Processor <NUM>, motion controller <NUM>, and/or interface system <NUM> may also be configured to communicate with one or more of the various elements of digital imaging device <NUM> via a wireless communication link.

Motion control system <NUM> is configured to precisely control and coordinate XYZ movement of stage <NUM> and/or objective lens <NUM> (e.g., via objective lens positioner <NUM>). Motion control system <NUM> may also be configured to control movement of any other moving part in digital imaging device <NUM>. For example, in a fluorescence scanner embodiment, motion control system <NUM> may be configured to coordinate movement of optical filters and the like in the epi-illumination system <NUM>.

Interface system <NUM> allows digital imaging device <NUM> to interface with other systems and human operators. For example, interface system <NUM> may include a user interface (e.g., graphical user interface) to provide information directly to an operator and/or to allow direct input from an operator. Interface system <NUM> may also be configured to facilitate communication and data transfer between digital imaging device <NUM> and one or more local external devices that are directly connected (e.g., a printer, removable storage medium, etc.) and/or one or more remote external devices that are connected to digital imaging device <NUM> via a network (e.g., an image server, operator station, user station, administrative server, etc.).

Illumination system <NUM> may be configured to illuminate a portion of sample <NUM>. For example, illumination system <NUM> may include a light source and illumination optics. The light source could be a variable intensity halogen light source with a concave reflective mirror to maximize light output and a KG-<NUM> filter to suppress heat. The light source could also be any type of arc-lamp, laser, or other source of light. In an embodiment, illumination system <NUM> illuminates sample <NUM> in transmission mode, such that line scan camera <NUM> and/or area scan camera <NUM> sense optical energy that is transmitted through sample <NUM>. Alternatively, or in combination, illumination system <NUM> may be configured to illuminate sample <NUM> in reflection mode, such that line scan camera <NUM> and/or area scan camera <NUM> sense optical energy that is reflected from sample <NUM>. Overall, illumination system <NUM> is configured to be suitable for interrogation of microscopic sample <NUM> in any known mode of optical microscopy.

In an embodiment, digital imaging device <NUM> optionally includes an epi-illumination system <NUM> to optimize digital imaging device <NUM> for fluorescence scanning. Fluorescence scanning is the scanning of samples <NUM> that include fluorescence molecules, which are photon sensitive molecules that can absorb light at a specific wavelength (excitation). These photon sensitive molecules also emit light at a higher wavelength (emission). Because the efficiency of this photoluminescence phenomenon is very low, the amount of emitted light is often very low. This low amount of emitted light typically frustrates conventional techniques for scanning and digitizing sample <NUM> (e.g., transmission mode microscopy). Advantageously, in a fluorescence scanner system embodiment of digital imaging device <NUM>, use of a line scan camera <NUM> that includes multiple linear sensor arrays (e.g., a time delay integration ("TDI") line scan camera) increases the sensitivity to light of line scan camera <NUM> by exposing the same area of sample <NUM> to each of the multiple linear sensor arrays of line scan camera <NUM>. This is particularly useful when scanning faint fluorescence samples with low emitted light.

Accordingly, in a fluorescence scanner system embodiment, line scan camera <NUM> is preferably a monochrome TDI line scan camera. Monochrome images are ideal in fluorescence microscopy, because they provide a more accurate representation of the actual signals from the various channels present on sample <NUM>. As will be understood by those skilled in the art, a fluorescence sample <NUM> can be labeled with multiple florescence dyes that emit light at different wavelengths, which are also referred to as "channels.

Furthermore, because the low-end and high-end signal levels of various fluorescence samples <NUM> present a wide spectrum of wavelengths for line scan camera <NUM> to sense, it is desirable for the low-end and high-end signal levels that line scan camera <NUM> can sense to be similarly wide. Accordingly, in a fluorescence scanner embodiment, line scan camera <NUM> used in digital imaging device <NUM> is a monochrome <NUM>-bit <NUM>-linear-array TDI line scan camera. It should be noted that a variety of bit depths for line scan camera <NUM> can be employed for use with a fluorescence scanner embodiment of digital imaging device <NUM>.

In an embodiment, movable stage <NUM> is configured for precise XY movement under control of processor <NUM> or motion controller <NUM>. Movable stage <NUM> may also be configured for Z movement under control of processor <NUM> or motion controller <NUM>. Moveable stage <NUM> may be configured to position sample <NUM> in a desired location during image capture by line scan camera <NUM> and/or area scan camera <NUM>. Moveable stage <NUM> may also be configured to accelerate sample <NUM> in a scanning direction to a substantially constant velocity and then maintain the substantially constant velocity during image capture by line scan camera <NUM>. In an embodiment, digital imaging device <NUM> may employ a high precision and tightly coordinated XY grid to aid in the location of sample <NUM> on movable stage <NUM>. In an embodiment, movable stage <NUM> is a linear motor based XY stage with high precision encoders employed on both the X and the Y axis. For example, very precise nanometer encoders can be used on the axis in the scanning direction and on the axis that is in the direction perpendicular to the scanning direction and on the same plane as the scanning direction. Stage <NUM> may also be configured to support glass slide <NUM> upon which sample <NUM> is disposed.

Sample <NUM> can be anything that may be interrogated by optical microscopy. For example, a glass microscope slide <NUM> is frequently used as a viewing substrate for specimens that include tissues and cells, chromosomes, DNA, protein, blood, bone marrow, urine, bacteria, beads, biopsy materials, or any other type of biological material or substance that is either dead or alive, stained or unstained, labeled or unlabeled. Sample <NUM> may also be an array of any type of DNA or DNA-related material such as cDNA, RNA, or protein that is deposited on any type of slide or other substrate, including any and all samples commonly known as a microarrays. Sample <NUM> may be a microtiter plate, for example a <NUM>-well plate. Other examples of sample <NUM> include integrated circuit boards, electrophoresis records, petri dishes, film, semiconductor materials, forensic materials, or machined parts.

In an embodiment, objective lens <NUM> is mounted on objective positioner <NUM>, which may employ a very precise linear motor to move objective lens <NUM> along the optical axis defined by objective lens <NUM>. For example, the linear motor of objective lens positioner <NUM> may include a <NUM>-nanometer encoder. The relative positions of stage <NUM> and objective lens <NUM> in XYZ axes are coordinated and controlled in a closed loop manner using motion controller <NUM> under the control of processor <NUM> that employs memory <NUM> for storing information and instructions, including the computer-executable programmed steps for overall operation of digital imaging device <NUM>.

In an embodiment, objective lens <NUM> is a plan apochromatic ("APO") infinity corrected objective with a numerical aperture corresponding to the highest spatial resolution desirable, where objective lens <NUM> is suitable for transmission mode illumination microscopy, reflection mode illumination microscopy, and/or epi-illumination mode fluorescence microscopy (e.g., an Olympus 40X, <NUM>. 75NA or 20X, <NUM> NA). Advantageously, objective lens <NUM> may be capable of correcting for chromatic and spherical aberrations. When objective lens <NUM> is infinity corrected, focusing optics <NUM> can be placed in optical path <NUM> above objective lens <NUM> where the light beam passing through the objective lens becomes a collimated light beam. Focusing optics <NUM> focus the optical signal captured by objective lens <NUM> onto the light-responsive elements of line scan camera <NUM> and/or area scan camera <NUM>, and may include optical components such as filters, magnification changer lenses, and/or the like. Objective lens <NUM>, combined with focusing optics <NUM>, provides the total magnification for digital imaging device <NUM>. In an embodiment, focusing optics <NUM> may contain a tube lens and an optional 2X magnification changer. Advantageously, the 2X magnification changer allows a native 20X objective lens <NUM> to scan sample <NUM> at 40X magnification.

In an embodiment, line scan camera <NUM> comprises at least one linear array of picture elements ("pixels"). The line scan camera may be monochrome or color. Color line scan cameras typically have at least three linear arrays, while monochrome line scan cameras may have a single linear array or plural linear arrays. Any type of singular or plural linear array, whether packaged as part of a camera or custom-integrated into an imaging electronic module, can also be used. For example, a <NUM>-linear-array ("red-green-blue" or "RGB") color line scan camera or a <NUM>-linear-array monochrome TDI may be used. TDI line scan cameras typically provide a substantially better signal-to-noise ratio ("SNR") in the output signal by summing intensity data from previously imaged regions of a specimen, yielding an increase in the SNR that is in proportion to the square-root of the number of integration stages. TDI line scan cameras comprise multiple linear arrays. For example, TDI line scan cameras are available with <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or even more linear arrays. Digital imaging device <NUM> also supports linear arrays that are manufactured in a variety of formats, including some with <NUM> pixels, some with <NUM> pixels, and others having <NUM> or more pixels. Similarly, linear arrays with a variety of pixel sizes can also be used in digital imaging device <NUM>. The salient requirement for the selection of any type of line scan camera <NUM> is that the motion of stage <NUM> can be synchronized with the line rate of line scan camera <NUM>, so that stage <NUM> can be in motion with respect to line scan camera <NUM> during the image capture of sample <NUM>.

The image data generated by line scan camera <NUM> may be stored in memory <NUM> and processed by processor <NUM> to generate a contiguous digital image of at least a portion of sample <NUM>. The contiguous digital image can be further processed by processor <NUM>, and the revised contiguous digital image can also be stored in memory <NUM>.

In an embodiment with two or more line scan cameras <NUM>, at least one of line scan cameras <NUM> can be configured to function as a focusing sensor that operates in combination with at least one of line scan cameras <NUM> that is configured to function as an imaging sensor. The focusing sensor can be logically positioned on the same optical path as the imaging sensor, or the focusing sensor may be logically positioned before or after the imaging sensor with respect to the scanning direction of digital imaging device <NUM>. In such an embodiment with at least one line scan camera <NUM> functioning as a focusing sensor, the image data generated by the focusing sensor may be stored in memory <NUM> and processed by processor <NUM> to generate focus information to enable digital imaging device <NUM> to adjust the relative distance between sample <NUM> and objective lens <NUM> to maintain focus on sample <NUM> during scanning.

In operation, the various components of digital imaging device <NUM> and the programmed modules stored in memory <NUM> enable automatic scanning and digitizing of sample <NUM>, which is disposed on glass slide <NUM>. Glass slide <NUM> may be securely placed on movable stage <NUM> of digital imaging device <NUM> for scanning sample <NUM>. Under control of processor <NUM>, movable stage <NUM> accelerates sample <NUM> to a substantially constant velocity for sensing by line scan camera <NUM>, where the speed of stage <NUM> is synchronized with the line rate of line scan camera <NUM>. After scanning a stripe of image data, movable stage <NUM> decelerates and brings sample <NUM> to a substantially complete stop. Movable stage <NUM> then moves orthogonal to the scanning direction to position sample <NUM> for scanning of a subsequent stripe of image data (e.g., an adjacent stripe). Additional stripes are scanned until an entire portion of sample <NUM> or the entire sample <NUM> is scanned.

For example, during digital scanning of sample <NUM>, a contiguous digital image of sample <NUM> may be acquired as a plurality of contiguous fields of view that are combined together to form an image stripe. A plurality of adjacent image stripes are similarly combined together to form a contiguous digital image of a portion or the entire sample <NUM>. The scanning of sample <NUM> may include acquiring vertical image stripes or horizontal image stripes. The scanning of sample <NUM> may be either top-to-bottom, bottom-to-top, or both (bi-directional), and may start at any point on sample <NUM>. Alternatively, the scanning of sample <NUM> may be either left-to-right, right-to-left, or both (bi-directional), and may start at any point on sample <NUM>. Additionally, it is not necessary that image stripe s be acquired in an adjacent or contiguous manner. Furthermore, the resulting image of sample <NUM> may be an image of the entire sample <NUM> or only a portion of sample <NUM>.

In an embodiment, computer-executable instructions (e.g., programmed modules or software) are stored in memory <NUM> and, when executed, enable digital imaging device <NUM> to perform the various functions described herein. In this description, the term "computer-readable storage medium" is used to refer to any media used to store and provide computer executable instructions to digital imaging device <NUM> for execution by processor <NUM>. Examples of these media include memory <NUM> and any removable or external storage medium (not shown) communicatively coupled with digital imaging device <NUM>, either directly or indirectly, for example via a network (not shown).

<FIG> illustrates a line scan camera <NUM> having a single linear array <NUM>, which may be implemented as a charge coupled device ("CCD") array. The single linear array <NUM> comprises a plurality of individual pixels <NUM>. In the illustrated example, the single linear array <NUM> has <NUM> pixels <NUM>. In alternative example, linear array <NUM> may have more or fewer pixels <NUM>. For example, common formats of linear arrays include <NUM>, <NUM>, and <NUM> pixels <NUM>. Pixels <NUM> are arranged in a linear fashion to define a field of view <NUM> for linear array <NUM>. The size of field of view <NUM> varies in accordance with the magnification of digital imaging device <NUM>.

<FIG> illustrates a line scan camera <NUM> having three linear arrays, each of which may be implemented as a CCD array. The three linear arrays combine to form a color array <NUM>. Each individual linear array in color array <NUM> detects a different color intensity (e.g., red, green, or blue). The color image data from each individual linear array in color array <NUM> is combined to form a single field of view <NUM> of color image data.

<FIG> illustrates a line scan camera having a plurality of linear arrays, each of which may be implemented as a CCD array. The plurality of linear arrays collectively form a TDI array <NUM>. Advantageously, a TDI line scan camera may provide a substantially better SNR in its output signal by summing intensity data from previously imaged regions of a specimen, yielding an increase in the SNR that is in proportion to the square-root of the number of linear arrays (also referred to as integration stages). A TDI line scan camera may comprise a larger variety of numbers of linear arrays. For example, common formats of TDI line scan cameras include <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and even more linear arrays.

Claim 1:
A slide scanning device comprising:
a plurality of line sensors, each configured to generate line images of a sample, wherein each of the plurality of line sensors comprises a plurality of rectangular pixel sensors elongated in the longitudinal direction of the line sensor, wherein each of the plurality of line sensors, in a longitudinal direction of the line sensor, is offset from an adjacent one of the plurality of line sensors by a fraction of a length of each pixel sensor;
an objective lens configured to successively provide a same field of view of the sample to each of the plurality of line sensors; and
at least one hardware processor configured to, for each of a plurality of positions on the sample:
combine the line images of the same field of view to produce a plurality of subpixels for each of at least a subset of pixels within the line images of the same field of view, wherein rectangular pixels from each of the line images are combined, the combination resulting in a square image subpixel,
generate an up-sampled line image of the position comprising the produced plurality of subpixels, and
combine the up-sampled line images of each of the plurality of positions on the sample into an image.