Patent Description:
It is known to view samples under microscopic imaging devices. Historically, biological samples have been prepared on standard format glass slides to be placed under bright light in the field of view of a conventional optical microscope, to be viewed manually by an operator. A single optical microscope device might include interchangeable lenses to view the sample at different magnifications. The lenses are focusable to ensure a clear image. Typically, under such conventional microscopy, the operator views only a small portion of the whole sample at a time, and aligns the slide under the point of focus in order to view a different portion of the sample on the slide. There are many classes of optical microscopes, which can generally be differentiated according to their observation methods, including: bright field microscopes; dark field microscopes; phase difference microscopes; polarized light microscopes; interference microscopes; confocal microscopes; hyperspectral microscopes; and fluorescent microscopes. Each such microscope can use either a transmission or reflection approach: in a transmission microscope, the incident light passes through a transparent sample; whereas in a reflection microscope, the light source illuminates non-transparent samples from above, with the reflected light being collected by the lens.

Digital microscopes (or imaging systems) combine a traditional optical microscope with digital image capture means - such as a digital camera, a CCD or a CMOS device - and digital processing technology, all typically under a centralised computer control.

'Whole Slide Imaging' (WSI) comprises digitally capturing an image of a whole sample on a slide at once using high-speed, high-resolution digital equipment, in a format that allows the image to be viewed on a computer monitor. An operator viewing the resulting digital image can zoom in on the image and pan the image to focus on particular areas of interest.

Some imaging devices can be operated remotely, under computer control. For example, some microscopic imaging devices available on the market can be controlled either manually or via a computer connected by a wired network to the various devices making up the device. These approaches limit the use of the device to those physically present in the location where the device is housed and force potential users to acquire equipment and/or travel to a location with the equipment.

Whether they are designed to operate automatically or require the intervention of an operator, commercial microscopes today are designed and developed to perform a single, specific task. Accordingly, such microscopes can operate using only one, or at most a few imaging modalities. This makes it complex, sometimes impossible, to exploit the potential of a multimodal analysis using several different microscopy techniques. One primary obstacle to combining images from microscopes with different specifications is that it is necessary to associate and recalibrate their respective data sets. As such, the final consistency of the resulting combined data set is only partial because it is significantly affected by the biases induced by the multitude of sources.

In pathology, it can be useful to analyse a single sample using different imaging modalities. By way of just one example, analysis of hepatic tissue specimens is important for diagnosis of the presence and severity of liver diseases, both for clinical practice and research purposes, and by imaging a hepatic tissue specimen under different imaging modalities, different aspects of the histology may be highlighted, with some imaging modalities being better for showing e.g. cancerous tissue than others, whilst yet other imaging modalities may be better for showing fatty tissue. By way of another example, diabetes characterized by the presence of collagen in the liver can be diagnosed using two imaging techniques on the same organ.

<CIT> discloses a multimodal imaging platform for imaging a sample with multiple imaging modalities using the same imaging system layout. The system includes a sample stage that is translatable on the platform so as to position the sample at either a first region (for a first imaging modality) or a second region (for a different imaging modality).

<CIT> discloses an automated system of processing biological specimens, which includes a sample storage module and automated sample handling via robot.

<CIT> teaches a slide handler for automated movement of slides between various stations of an imager, to generate a whole image of a specimen affixed to a slide.

Thus it is known in the field of microscopy to combine a multitude of imaging modalities from different imaging techniques. However, this makes it difficult to aggregate and combine measurements from different sources without loss and degradation of information. For example, in order to benefit from the advantages of multimodal imaging, it is useful to superimpose images captured using those various different imaging modalities. To date, such superimposition of images has been carried out manually, e.g. by physical manual overlay of one image over another or using digital image manipulation to superimpose one image on another, in each case with the operative aiming to align the images by sight so that they are in registration. Such techniques have the disadvantage of not being scalable or repeatable because the registration remains very approximate. Moreover, it is a complex process because of the heterogeneity of the various imaging devices, such as microscopes of various differing optics, and scanners. This complexity particularly affects the efficiency of multimodal analyses and induces limitations in the development of predictive algorithms. For an improved diagnostic process, which may incorporate aspects of. Al and associated algorithms for automatically identifying aspects of interest within datasets collated from the captured images, it is envisaged that hundreds of images may be analysed.

It is an object of the invention to address at least some of these identified difficulties.

In accordance with the present invention there is provided an imaging system for capturing different images of a sample, as defined in independent claim <NUM>.

Advantageously, this technique facilitates the automated capture of images from different imaging devices on a common platform to provide efficient production of multimodal digital images from samples, thereby improving accuracy and throughput, hence improved diagnostic outcomes. The system avoids difficulties of the prior approaches by this provision of different, interoperable imaging devices on a common platform, thereby generating coherent, calibrated and exploitable data sets based on the captured digital images without the need to carry out prior processing.

The first imaging device may comprise one of: UV, visible, IR and multispectral brightfield microscopy / UV, visible, IR and multispectral darkfield microscopy / UV, visible, IR and multispectral phase contrast microscopy / UV, visible, IR and multispectral differential interference contrast microscopy / UV, visible, IR and multispectral epi-fluorescence widefield microscopy / UV, visible, IR absorption microspectrophotometry / UV, visible, IR absorption microspectrophotometry / UV, visible, IR scattering microspectrophotometry / UV, visible, IR hyperspectral interferometric microscopy / confocal fluorescence microscopy, confocal hyperspectral fluorescence microscopy, confocal Raman microspectroscopy, confocal reflectance microspectroscopy, fluorescence lifetime confocal microscopy / fluorescence lifetime widefield microscopy / two and three photons excitation fluorescence microscopy / two and three photons excitation fluorescence lifetime microscopy / Coherent anti-Stokes Raman scattering microscopy / two (and three) photons second (and third) harmonic generation microscopy / stimulated Raman scattering microscopy, Scanning Electron Microscopy / X ray fluorescence microscopy.

The second imaging device may comprise one of: UV, visible, IR and multispectral brightfield microscopy / UV, visible, IR and multispectral darkfield microscopy / UV, visible, IR and multispectral phase contrast microscopy / UV, visible, IR and multispectral differential interference contrast microscopy / UV, visible, IR and multispectral epi-fluorescence widefield microscopy / UV, visible, IR absorption microspectrophotometry / UV, visible, IR absorption microspectrophotometry / UV, visible, IR scattering microspectrophotometry / UV, visible, IR hyperspectral interferometric microscopy / confocal fluorescence microscopy, confocal hyperspectral fluorescence microscopy, confocal Raman microspectroscopy, confocal reflectance microspectroscopy, fluorescence lifetime confocal microscopy / fluorescence lifetime widefield microscopy / two and three photons excitation fluorescence microscopy / two and three photons excitation fluorescence lifetime microscopy / Coherent anti-Stokes Raman scattering microscopy / two (and three) photons second (and third) harmonic generation microscopy / stimulated Raman scattering microscopy, Scanning Electron Microscopy / X ray fluorescence microscopy, different to the first.

The system may further comprise a sample retention mechanism. The sample retention mechanism may comprise a sample slide holder.

The imaging devices may each be located at discrete positions on a single planar surface. Thus, the imaging devices may be fixed on a common surface thereby having a defined spatial relationship and facilitating the manoeuvre of samples in the system.

The system may further comprise a sample storage area. The sample storage are may be fixed to the common surface too, ensuring ease of transport of the samples therefrom to the different image capture devices.

The system comprises a pick and place robot for selectively gripping and moving a selected sample to a desired location, under the control of the processor.

According to one embodiment, the pick and place robot is for selectively moving the selected sample from the sample storage area to the mechanism for translating the sample relative to each of the first and second imaging devices. This provides a convenient mechanism for automatically retrieving a selected sample from the sample storage area for onward distribution to the image capture devices on the separate mechanism for translating the sample.

According to an alternative embodiment, the pick and place robot itself comprises the mechanism for translating the-sample relative to each of the first and second imaging devices and is thus for selectively moving the sample from the sample storage area directly to the first and second imaging devices.

In some embodiments, the system further comprises a sample support associated with each imaging device for receiving the selected sample and for positioning the sample at a specified X-Y position in the field of view of the imaging device. The sample support may comprise a double stage sample support for lateral X-Y positioning of the sample at a first resolution on the first stage and at a second resolution, finer than the first, on the second stage.

The sample includes a calibration symbol and the processor is further configured to combine the first and second digital images of the sample by:.

Advantageously, this technique facilitates automated registration of images from different imaging devices to provide efficient production of composite multimodal images, thereby improving accuracy and throughput, hence improved diagnostic outcomes.

The sample may comprise a biological sample. Image capture as provided by the invention is particularly beneficial in the field of biological samples, whereby different imaging modalities can each highlight different aspects of interest in the sample so as to provide a holistic diagnosis taking into account those different aspects.

The sample may be on a sample slide. A sample slide is a convenient and common platform in the art for handling a specimen for imaging.

<FIG> illustrates, in schematic form, a system <NUM> for combining images of a sample taken using different imaging devices. The system comprises a support <NUM> on which are arranged a plurality of imaging devices <NUM>, each of which may be configured to capture images according to a different imaging modality. The support <NUM> may comprise a planar surface, such as a table top. The surface may be approximately <NUM> x <NUM>. Samples to be imaged are stored in a sample storage area <NUM>, which is also fixed on the surface. A sample handling mechanism <NUM> comprises a sample holder <NUM> or carriage onto which individual samples <NUM> (see <FIG>) can be placed. The sample holder <NUM> is movable on a gantry <NUM> between the sample storage area <NUM> and any of the different imaging devices <NUM> in turn for the capture there of an image of the sample according to the image modality of the selected imaging device <NUM>, as described in more detail below. A processor <NUM> is in operative communication with the imaging devices <NUM> and the sample handling mechanism <NUM> for the automated control of the system.

The sample holder <NUM> is loaded by using a pick and place robot having an end effector (e.g. a gripper, not shown) under control of the processor, to pick a selected sample from the storage area <NUM> and to place the selected sample <NUM> on the sample holder. The sample holder <NUM> may, in some embodiments, be capable of holding more than one sample - for example it may hold up to four samples <NUM>. In some embodiments, the sample handling mechanism itself comprises a pick and place robot that is capable of picking samples from the sample storage area <NUM> and delivering them directly to a selected image capture device <NUM>, all under the control of the processor <NUM>.

Examples of the different imaging modalities that the respective different image capture devices <NUM> may use include: a digital camera; a microscope (e.g. digital optical microscopes of differing magnification, a scanning electron microscope, confocal microscope,. ); a scanner; any of which may be configured to capture light in the visible, infrared (IR) or ultraviolet (UV) spectrum; an X-ray device; thus, but not exclusively: UV, visible, IR and multispectral brightfield microscopy / UV, visible, IR and multispectral darkfield microscopy / UV, visible, IR and multispectral phase contrast microscopy / UV, visible, IR and multispectral differential interference contrast microscopy / UV, visible, IR and multispectral epi-fluorescence widefield microscopy / UV, visible, IR absorption microspectrophotometry / UV, visible, IR absorption microspectrophotometry / UV, visible, IR scattering microspectrophotometry / UV, visible, IR hyperspectral interferometric microscopy / confocal fluorescence microscopy, confocal hyperspectral fluorescence microscopy, confocal Raman microspectroscopy, confocal reflectance microspectroscopy, fluorescence lifetime confocal microscopy / fluorescence lifetime widefield microscopy / two and three photons excitation fluorescence microscopy / two and three photons excitation fluorescence lifetime microscopy / Coherent anti-Stokes Raman scattering microscopy / two (and three) photons second (and third) harmonic generation microscopy / stimulated Raman scattering microscopy, Scanning Electron Microscopy / X ray fluorescence microscopy.

As best shown in <FIG>, a sample <NUM> may typically take the form of a microscopy slide <NUM>, on which is placed a biological specimen <NUM>, such as a tissue biopsy. It can be envisaged that other forms of sample retention other than a slide <NUM>, such as a petri dish, may be employed instead.

In use, a sample <NUM> is translated to a selected image capture device <NUM> by the sample handling mechanism <NUM> under the control of the processor <NUM>. At the selected image capture device <NUM>, the sample is positioned in a field of view of the image capture device <NUM> by suitable manipulation of the sample holder <NUM>. In some embodiments, the sample holder <NUM> is positioned on a sample stage (not shown) of the image capture device. The sample stage may be translatable within the image capture device <NUM> to accurately position the sample holder <NUM>, ergo the sample <NUM>, for a focused image of a desired portion of the sample. By way of example, the sample stage may comprise a double stage translation mechanism for X-Y (lateral, horizontal) positioning of the sample relative to the field of view; a first stage may be translated at a first resolution, such as a <NUM> micron resolution, in order to get the sample to an approximate X-Y position quickly; and a second stage, attached to the first stage, may be translated with a finer resolution, such as a <NUM> resolution for fine-tuned accurate final positioning. An auto-focus mechanism, as well known in the art, may be employed for adjustment of the Z (vertical) position of the sample <NUM> relative to the optical stage of the image capture device.

Once positioned at a first selected imaging device <NUM>, a first digital image <NUM> of the sample <NUM> - or, more accurately, at least a portion of the specimen <NUM> (because the image may be taken of the entire slide, just a portion of the slide, or just a portion of the sample on the slide, depending on the situation and requirements) - can be captured and stored in a memory, such as a database, under the control of the processor <NUM>. By way of example, the first digital image <NUM> of the sample may be a conventional visible light microscopic image at a given magnification of, say 20x. It will be understood that any suitable magnification may be used or indeed that other imaging modalities may be employed instead.

Then, the sample <NUM> is manoeuvred, by the sample handling mechanism <NUM>, to a second selected imaging device <NUM>, which is of a different imaging modality to the first imaging device. Thus, once accurately positioned at the second selected imaging device <NUM>, a second digital image <NUM> of the sample <NUM> - or, more accurately, at least a portion of the specimen <NUM> - can be captured and stored under the control of the processor <NUM>. By way of example, the second digital image <NUM> of the sample may be a fluorescent light microscopic image. It will be understood that any suitable magnification may be used or indeed that other imaging modalities may be employed instead. As seen in <FIG>, the first and second images <NUM>,<NUM> of the sample <NUM> may capture different aspects <NUM>', <NUM>" of the specimen, due to having been taken using different imaging modalities. Further images of the sample <NUM> may be captured at third and subsequent image capture devices <NUM> under further different imaging modalities.

To date, as explained in the introduction, capturing images of a sample using different imaging modalities has been difficult because it has required manual manipulation of the samples to move them from one imaging device to another, often in remote locations. That is thus not a scalable process due to the time and labour required. Furthermore, the images that are captured will necessarily be inconsistent because they are taken using separate image capture devices and the placement of the samples at each location involves so many variables. Thus, the present invention addresses this by introducing an automated process for the movement and positioning of the samples at the different image capture devices all on a common unitary platform. The system consists of a modular optical bench arranged on a single optical table associated with a robotic unit in charge of managing the movement of the samples. The uniqueness of the system facilitates a total control of the displacements of the samples and on their calibrations. This approach overcomes the limitations induced by the use of independent devices to perform measurements on the modalities of interest.

Images taken from the different imaging devices <NUM> may be combined to form a composite image. In a raw form, as shown in <FIG>, the composite image <NUM> may be misaligned, due to possible differences between the position of the optical imaging stages of the respective image capture devices relative to the positioning of the sample <NUM> thereunder during image capture. In order to align the images captured using the various different imaging devices <NUM> so that the pixels of each digital image <NUM>,<NUM> are in registration with one another in the final composite image, a calibration step is needed to determine an offset <NUM> of the respective components of the composite image <NUM>. Where the second image <NUM> is captured with a different magnification to the first image <NUM>, a scaling step is also needed in order to combine the images.

To date, as explained in the introduction, combining images has been a manual and labour-intensive process, using either a physical manual overlay of one image over another or using digital image manipulation to superimpose one image on another, in each case with the operative aiming to align the images by sight so that they are in registration.

That is also not a scalable process due to the time and skill required to achieve accurate results. Thus, embodiments of the present invention address this by introducing an automated process for the alignment and registration of the respective images. Accordingly, each slide <NUM> may have printed, etched, adhered or otherwise deposited thereon a calibration symbol <NUM>. The calibration symbol <NUM> may be in the form of a pair of target symbols 31a,b on opposed left- and right-hand sides of the slide <NUM>. In other embodiments, only a single target <NUM> may be employed, or more than two targets <NUM> may be used.

Thus, at each image capture device <NUM>, once the sample <NUM> has been positioned in the field of view by the sample handling mechanism <NUM> and, optionally, by translation of the sample stage, a digital image of the whole slide <NUM> including the calibration symbol <NUM> is captured. Examples are illustrated in <FIG>, which respectively show a first image <NUM> of the calibration symbol <NUM>' as captured by the first image capture device, and a second image <NUM> of the calibration symbol <NUM>" as captured by the second image capture device. These may be separate captures to those of the specimen <NUM> on the slide <NUM>. For example, the images <NUM>,<NUM> of the whole slide <NUM> including the calibration symbol <NUM> may be captured using a conventional digital camera incorporated into each image capture device <NUM>. Alternatively, the images of the calibration symbol <NUM>',<NUM>" may be captured at the respective first and second imaging devices <NUM> using the native imaging modality that is used to capture the images of the specimen.

A separate image <NUM>,<NUM> of the specimen <NUM> may then be taken (or indeed may be taken first) using the specialised imaging modality of that particular image capture device. Because the digital camera is in a fixed position relative to the field of view of the specialised imaging modality, and because the calibration symbol <NUM> is fixed in position on the slide <NUM> relative to the specimen <NUM>, if the image of the calibration symbol <NUM> as taken by a first imaging device <NUM> is aligned with the image of the calibration symbol <NUM> as taken by a second imaging device <NUM>, then it can be assured that the respective images of the specimen <NUM> as captured by those first and second image capture devices will be in registration. This holds true even if the sample <NUM> is moved within the selected imaging device, e.g. by the sample stage, between capturing the image of the calibration symbol <NUM> and the image of the specimen <NUM>, provided that the movement can be tracked accurately, e.g. by counting, in the processor <NUM>, the number of nm steps that the stage has been translated. By having multiple targets, the calibration symbol <NUM> can more readily be used for accurate alignment and scaling of the images.

To this end, the first and second images <NUM>,<NUM> of the calibration symbol <NUM> (e.g. the targets 31a,b) are analysed by the image processor <NUM> and the relative positions of the symbols <NUM>',<NUM>" in the images are determined. Thus, an offset <NUM> (see <FIG>) can be determined. Typically, the offset <NUM> will be determined in terms of X and Y translations in a cartesian coordinate system of reference. However, it will be understood that the offset could be determined in other ways, such as by reference to a polar coordinate system. The offset <NUM> may be a translational offset or a rotational offset, or a combination thereof; thus, the offset can be considered as a rigid transformation.

Once the offset <NUM> has been determined, the second image <NUM> of the sample containing the specimen <NUM> may be translated by that offset so that the resulting composite image <NUM> (see <FIG>) is in perfect registration - with the respective calibration symbols <NUM>',<NUM>" from each of the first and second captures in alignment, and with the respective aspects <NUM>',<NUM>'' of the specimen images likewise in alignment.

In summary, a multi-modal image capture process according to embodiments of the invention may thus comprise the steps of:.

Alternatively, the imaging of the calibration symbol at each of the first and second image capture devices may take place first, before the imaging of the sample. Hence, in that alternative, a multi-modal image capture process according to the invention may thus comprise the steps of:.

In some embodiments, where the calibration symbol <NUM> is visible under the imaging modality of the selected image capture device <NUM>, an image of the whole slide <NUM>, including the calibration symbol <NUM> and the specimen <NUM> may be taken in a single image capture. Thus, steps c and h on the one hand, and e and j on the other hand can be combined, and the associated movement steps d and g can be eliminated. A calibration symbol that is visible under all the imaging modalities of interest can be envisaged for this purpose.

Whereas the system and process has been described in terms of translating the sample <NUM> to the different image capture devices <NUM>, it will be understood that the sample <NUM> may be retained in a static position and that the image capture devices <NUM> may be translated to an imaging position at the sample from respective non-imaging positions away from the sample location.

Where the term 'image capture' and analogues has been used, this encompasses not only the capture of a single digital image, but also the possibility that a plurality of images may be captured and processed in order to produce a single, final image. For example, more than a single image of the whole slide may be combined or interpolated, or multiple images of different or overlapping portions of the slide may be captured and a "stitching" algorithm be used which reconstructs a global scan of the whole sample.

Furthermore, whereas the system and method have been described in the context of biological samples, it will be understood that the samples that are imaged may be from other fields, such as (but not limited to): pharmacological, geological, semi-conductor or material samples.

Claim 1:
An imaging system (<NUM>) for capturing different images of a sample (<NUM>) including a calibration symbol (<NUM>), comprising:
a first imaging device (<NUM>) for acquiring a first digital image (<NUM>) of the sample (<NUM>) with a first imaging modality;
a second imaging device (<NUM>) for acquiring a second digital image (<NUM>) of the sample (<NUM>) with a second imaging modality, different to the first;
a mechanism (<NUM>) for translating the sample relative to each of the first and second imaging devices (<NUM>) to position the sample, in turn, at an imaging position at each of the first and second imaging devices (<NUM>); and
a processor (<NUM>) configured to:
control the mechanism (<NUM>) for translating the sample (<NUM>);
receive the first digital image (<NUM>) of the sample (<NUM>) from the first imaging device (<NUM>); and
receive the second digital image (<NUM>) of the sample (<NUM>) from the second imaging device (<NUM>), said imaging system (<NUM>) comprising a pick and place robot for selectively gripping and moving a selected sample to a desired location, under the control of the processor (<NUM>), the sample including a calibration symbol (<NUM>) and the processor (<NUM>) being further configured to combine the first and second digital images (<NUM>,<NUM>) of the sample by:
receiving a first digital image (<NUM>) of the calibration symbol (<NUM>) from the first imaging device (<NUM>); receiving a second digital image (<NUM>) of the calibration symbol (<NUM>) from the second imaging device (<NUM>); comparing the position of the first digital image (<NUM>) of the calibration symbol with the position of the second digital image (<NUM>) of the same calibration symbol, thereby determining an image offset (<NUM>) between the different imaging devices; translating the second digital image (<NUM>) of the sample by the determined offset (<NUM>); and superimposing the first digital image (<NUM>) of the sample and the translated second digital image (<NUM>) of the sample, whereby the first image is in registration with the second image.