Patent Description:
A high-content imaging system (HCIS) may be used to obtain a microscopy image of one or more biological sample(s) such as DNA, proteins, cells, and the like. The biological sample(s) may be disposed in a well of a microplate having a two-dimensional pattern of such wells. Such microplates typically have <NUM> or <NUM> wells but may have more or fewer wells. To acquire images of the biological sample(s) disposed in such microplates, an autofocus system of the HCIS focuses on each well of the microplate or a portion thereof and acquires an image. To develop a high-resolution image of the biological sample(s) in a particular well, the imaging sensor may be positioned relative to different portions of the well, focused on such portion, and acquire an image. The images of such different portions may be combined (e.g., concatenated) to produce an image of the entire well. Further, the HCIS may develop a plurality of images of the well, wherein each one of the plurality of images is captured using different imaging configurations. Such imaging configurations specify the illumination source used when the image is captured, one or more filters disposed in the light path between the well (or portion thereof) and the imaging sensor, and the like.

It should be apparent that the repeated focusing and positioning operations described above combined with scanning a plate to produce high resolution images can require a significant amount of time. Further, such time increases substantially if multiple microplates have to be imaged or if a user of the HCIS has to verify and/or adjust positioning of the sample and/or focus of the HCIS during the imaging process. In addition, image processing operations such as deconvolution, noise reduction, and the like may be applied, typically using a computer that receives an image generated by the HCIS, further adding to the amount of time necessary to produce images of the biological sample(s) from the HCIS that are suitable for further analysis.

<CIT> discloses an image evaluation device including a design data image generation unit that images design data; a machine learning unit that creates a model for generating a design data image from an inspection target image, using the design data image as a teacher and using the inspection target image corresponding to the design data image, and a design data prediction image generation unit that predicts the design data image from the inspection target image.

<CIT> discloses a method for generating a trained neural network model for scanning correction corresponding to one or more imaging parameters. The trained neural network model may be trained using training data.

In accordance with one aspect, a high-content imaging system is disclosed in claim <NUM>.

In accordance with another aspect, a method of operating a high-content imaging system is disclosed in claim <NUM>.

Other aspects and advantages will become apparent upon consideration of the following detailed description and the attached drawings wherein like numerals designate like structures throughout the specification.

As described in detail below, a high-content imaging system (HCIS) in accordance with the present disclosure includes a stage on which a sample or a microplate having a sample may be disposed, one or more illumination sources, one or more objective lens(es), one or more filter(s), a focusing apparatus, an imaging sensor, a machine learning system, a controller and one or more machine learning system training model(s) that may be used with the machine learning system. Each training model is associated with an input imaging configuration and an output imaging configuration and includes data necessary to configure and train an untrained machine learning system (e.g., a neural network or another deep learning system). For example, if the untrained machine learning system is a neural network, the training model includes parameters regarding the interconnections of one or more convolution layers, one or more neuron layers, and one or more pooling layers therebetween. The training model also includes scaling factors, kernel weights, and the like associated with such layers.

The untrained machine learning system is configured with a particular training model to develop a trained machine learning system. Thereafter, when the trained machine learning system is presented with an input (or source) image captured when the HCIS is configured with the input imaging configuration associated with the particular training model, the machine learning system produces an output (or target) image that represents the image that would have been generated if the HCIS had been configured with the output imaging configuration associated with the particular training model.

During operation, a user uses a computer in communication with the HCIS to transmit a request to the controller of the HCIS to produce an output image of a biological sample disposed in a well of a microplate disposed therein. Such request may include a specification of an output imaging configuration to use when producing the image and identifies, for example, an objective lens, resolution, illumination source, one or more optical filters, focusing accuracy, and/or types of imaging aberrations or artifacts that should be corrected for in the output image. Such aberrations/artifact correction may include correcting for field curvature of the objective lens, deconvolution of the image to improve contrast and/or resolution, and the like.

The controller analyzes the output image specification and selects a training model associated with such output image specification. The controller then determines the input image specification associated with the selected training model, configures the untrained machine learning system with the training model to produce the trained machine learning system, configures the HCIS according to the input imaging configuration, and captures an image. The controller provides the captured im age to the trained machine learning system as an input, the trained machine learning system generates an output image that represents an image that would have been captured if the HCIS had been configured using the output imaging configuration, and the controller transmits the output image generated by the HCIS to the user's computer.

In some embodiments, the training model may be associated with an input configuration that requires a first set of physical components (e.g., objective lens, illumination source, etc.) installed in the HCIS and an output configuration that requires a second set of physical components, some of which may not present in the HCIS. If an untrained machine learning system of the HCIS is configured with such training model to develop a trained machine learning system, the HCIS (having the trained machine learning system) will produce from an input image acquired using the first set of physical components, an output image that represents the image that would have been acquired with the second set of physical components. Thus, for example, in such embodiments, the HCIS is able to produce output images as if the HCIS included additional components without the costs associated with such additional components.

In some embodiments, the HCIS can be operated using an input imaging configuration that captures images quickly (e.g., capture low resolution images using coarse focusing). A training model associated with such input configuration and an output configuration that requires more time to capture an image (e.g., capture high resolution images with very precise focusing) may be used to train the machine learning system. The HCIS having such trained machine learning system produces output images as if they were captured using a more time-consuming configuration from input images captured relatively faster, which improves the throughput of the HCIS. Also, because the additional image processing/correctionoperationsmay be incorporated into the training model used to train the machine learning system, undertaking these operations does not require any additional time beyond what the trained machine learning system takes to generate the output image from the input image, thus yielding additional throughput improvements.

Referring to <FIG>, an embodiment of an HCIS <NUM> includes a controller <NUM>, a stage <NUM> on which a sample (or a microplate carrying such sample) may be disposed, one or more objective lens(es) <NUM>, one or more illumination sources <NUM>, an image sensor <NUM>, and a focusing apparatus <NUM>. As described below, the HCIS <NUM> may be used to generate a transmitted light image and/or a fluorescent light image of a sample <NUM> disposed in a well <NUM> of a microplate <NUM> disposed on the stage <NUM>. The stage <NUM> may be an X-Y stage that is moveable along a plane that is parallel to the imaging plane of the image sensor or an X-Y-Z that is movable along the plane parallel to an imaging plane of the image sensor <NUM> and also a plane perpendicular to the imaging plane.

To generate the transmitted light image of the sample <NUM>, the controller <NUM> operates an illumination source 108a to transmit light through the sample <NUM> (and the microplate <NUM> in which the sample <NUM> is disposed). The light is then directed through a selected objective lens <NUM>, optionally redirected by mirrors <NUM> to form an image on a surface of the image sensor <NUM>. The image sensor <NUM> generates signals in accordance with the light sensed thereby and such signals are converted into an image by an image generator <NUM>.

To generate a fluorescent image of a sample <NUM>, the controller <NUM> operates an illumination source 108b to generate light that is redirected by the one or more mirrors <NUM>, through the objective lens <NUM>, through the microplate <NUM>, and to the sample <NUM>. The light that reaches the sample <NUM> may excite the sample <NUM> and cause the sample <NUM> to fluoresce. The light emitted by the sample <NUM> passes through the objective lens <NUM> and is directed by the one or more mirrors <NUM> to form an image on the surface of the image sensor <NUM>, which generates signals in response to such image that are converted by the image generator <NUM> into an image.

One or more optical filters may also be optionally disposed in the light path between the illumination source <NUM> and the sample <NUM> and/or between the sample <NUM> and the image sensor <NUM>.

As would be understood by one who has ordinary skill in the art, the image generator <NUM> receives signals and converts such signals into pixel values of an image. Further, the image generator <NUM> may receive signals associated with sub-images of the biological sample <NUM> as the image sensor <NUM> scans such sample <NUM> and combines such sub-images into a complete image.

With continued reference to <FIG>, during operation, the controller <NUM> receives from a computer <NUM> operated by a user a request to generate one or more image(s) of the sample <NUM>. As noted above, such request includes a specification of the output imaging configuration noted above. The controller <NUM> queries a database <NUM> to select a training model associated with the output imaging configuration received with the request. In some embodiments, for some training models in the database <NUM>, the HCIS <NUM> will produce an image more efficiently when configured with an input imaging configuration associated with the training model than when configured with the output imaging configuration associated with the request. Further, the input imaging configuration associated with the training model in the database <NUM> may specify equipment (e.g., illumination source, objective lens, etc.) that is installed in the HCIS <NUM>, whereas the output imaging configuration received with the request may require equipment not installed in the HCIS <NUM>.

After selecting a particular training model from the database <NUM>, the controller <NUM> configures an untrained machine learning system <NUM> in accordance with the training model to develop a trained machine learning system <NUM>. In some embodiments, the controller <NUM> may reset the machine learning system <NUM> to an untrained state before configuring such machine learning system <NUM> with the selected training model.

In some embodiments, the training model is stored on the user computer <NUM> or on another computer remote from the HCIS <NUM>. In such embodiments, the controller <NUM> requests the training model from the computer on which the training model is stored. In some embodiments, the user computer <NUM> may provide the training model to the controller <NUM> as part of the request sent thereby to generate one or more image(s) of the sample <NUM>. In such embodiments, the user computer <NUM> may load the training model from storage associated therewith or from a computer remote therefrom. Further, in some cases, the user computer <NUM> may ask the user to locate, for example, one or more data file(s) in which the training model is stored and provide such data file(s) to the controller <NUM>.

After the trained machine learning system <NUM> has been developed, the controller <NUM> configures the HCIS <NUM> in accordance with the input imaging configuration associated with the selected training model. In particular, the controller <NUM> loads the objective lens <NUM>, configures the image sensor <NUM>, operates the illumination source <NUM> and the focusing apparatus <NUM> in accordance with the input imaging configuration and directs the image sensor <NUM> and the image generator <NUM> to acquire an image.

After the image has been acquired, the acquired image is presented as an input to the trained machine learning system <NUM> that generates an output image therefrom. The controller <NUM> receives the output image and transmits such output image to the user computer <NUM>. In some embodiments, the image generator <NUM>, the trained machine learning system <NUM>, and the controller <NUM> all have access to shared memory that allows images generated by one such component to be accessed by another In other embodiments, these components communicate on a local area network (or another type of network) and transmit images therebetween.

In some embodiments, the HCIS <NUM> may include an image quality analyzer (not shown) that receives the output image generated by the trained machine learning system <NUM>. The image quality analyzer applies image analysis algorithms that detect, for example, noise in the image or other artifacts in the image and develops a score that represents the quality of the output image. Thereafter, the image quality analyzer provides the output image and the score to the controller <NUM> to transmit to the user computer <NUM>.

In some embodiments, the HCIS <NUM> may include a first image processor (not shown) that automatically receives the acquired image from the image generator <NUM>. The first image processor applies one or more image processing function(s) to the acquired image, for example, to scale the image, reduce noise in the image, and the like, and the processed image is automatically provided as an input to the machine learning system <NUM>. In response, the machine learning system <NUM> generates the output image that is then provided to the controller <NUM>, as described above.

In some embodiments, the first image processor is integral with the image generator <NUM> and the generated image is an output of such image processor.

In some embodiments, a second image processor (not shown) may processes the output image developed by the machine learning system <NUM>, for example, to scale the image, reduce noise in the image, and the like, and such processedimage is provided to the controller <NUM> to provide to the user computer <NUM>, as described above.

Whether the first and/or the second image processor(s) is/are used may be determined by the request received from user computer <NUM> or may be determined in accordance with the input and/or the output imaging configuration(s).

<FIG> shows a flowchart <NUM> of the steps undertaken by the controller <NUM> to generate an image in accordance with the request received from the computer <NUM>. Referring to <FIG> and <FIG>, at step <NUM> the controller <NUM> receives the requestto capture an image. As noted above, the request specifies the output imaging configuration for generating the output image.

In some embodiments, the user directs the computer <NUM> to send a request to the controller <NUM> that includes an indication whether the HCIS <NUM> should be operated in a non-enhanced or an enhanced mode. In the non-enhanced mode, the machine learning system <NUM> is not used. Rather, the controller <NUM> directs the image generator <NUM> to capture images and any such captured images are transmitted by the controller <NUM> to the user computer <NUM>. In the enhanced mode, the trained machine learning system <NUM> is invoked as described herein to enhance or generate output images from input images generated by the image generator <NUM>. For example, the request received by the controller <NUM> may specify that enhanced mode is to be used, input images should be acquired using 10x magnification, and the machine learning system <NUM> should generate output images that represent images that would have been captured using 20x magnification (i.e., enhance resolution). Similarly, the request may specify that enhanced mode is to be used, input images should be acquired using a short exposure time, that the machine learning system <NUM> should generate output images that represent images that would have been captured if a long exposure time had been used (i.e., enhanced exposure).

At step <NUM>, the controller <NUM> queries the database <NUM> to determine if a training model is stored therein associated with the output imaging configuration specified in the request. If such a training model is identified, the controller <NUM> proceeds to step <NUM>, otherwise, the controller <NUM> proceeds to step <NUM>.

At step <NUM>, the controller <NUM> loads the training model identified in step <NUM> and in step <NUM> configures the untrained machine learning system <NUM> with the loaded training model to develop the trained machine learning system <NUM>.

At step <NUM>, the controller <NUM> configures the HCIS <NUM> in accordance with the input imaging configuration associated with the training model identified in step <NUM>. Thereafter, the controller <NUM> directs the image sensor <NUM> and/or image generator <NUM> to capture and generate an image In some embodiments, when the controller <NUM> configures the HCIS <NUM> at step <NUM>, the controller <NUM> also configures the image generator <NUM> to automatically provide the image generated thereby to the machine learning system <NUM> as an input. For example, in some cases, the image generator <NUM> may store the image generated thereby in a data store (e.g., a memory, a disk, etc.) accessible by the machine learning system <NUM> or transmit the image using an interprocess communications channel between the image generator <NUM> and the machine learning system <NUM>. The machine learning system <NUM> may be configured to poll such data store or channel for the presence of image data and, when available, load the image data as an input. In other embodiments, the image generator <NUM> may invoke the machine learning system <NUM> automatically, for example, as a function call and pass the image to the machine learning system <NUM> as part of such invocation.

At step <NUM>, the controller <NUM> receives the output image generated by the machine learning system <NUM> either via a shared data store or communications channel as described in the foregoing.

At step <NUM>, the controller <NUM> transmits the output image received from the machine learning system <NUM> to the computer <NUM>. In some embodiments, the controller <NUM> also receives from the image generator <NUM> the image that was provided to the machine learning system <NUM>, for example, at step <NUM>, and at step <NUM> transmits to the computer <NUM> both the image received from the image generator <NUM> and the image produced by the machine learning system <NUM>.

In some embodiments, the request received from the controller <NUM>, at step <NUM>, may specify that enhanced mode should be used to generate a plurality of output images from one input image captured at step <NUM>. For example, the request may specify that the input image should be captured using an input imaging configuration (e.g., using a 10x objective lens <NUM>) and that first and second output images should be generated in accordance with a first output imaging configuration (e.g., using a 20x objective lens <NUM>) and a second output imaging configuration (e.g., using a 30x objective lens <NUM>). In response to such request, the controller <NUM> loads a first training model associated with the input imaging configuration and the first output imaging configuration and a second training model associated with the input imaging configuration and the second output imaging configuration. Thereafter, the controller <NUM> trains first and second instances of the machine learning system <NUM> with the first and second training models. In such embodiments, the controller <NUM> directs the image generator <NUM> to provide the input image generated thereby as an input to each of the first and second instances of the trained machine learning system <NUM>, each instance of the trained machine learning system <NUM> generates first and second output images, respectively, in response to such input image. The controller <NUM> receives the first and second output images and transmits such output images (and optionally the input image) to user computer <NUM>.

If at step <NUM>, the controller <NUM> determines that the database <NUM> does not include a training model associated with the output imaging configuration received with request at step <NUM>, the controller <NUM> configures the HCIS <NUM> in accordance with the output imaging configuration specified in the request at step <NUM>. At step <NUM>, the controller <NUM> directs the image sensor <NUM> and/or the image generator <NUM> to capture an image. At step <NUM>, the controller <NUM> receives the captured image from the image generator <NUM> and, at step <NUM>, the controller <NUM> transmits the captured image to the computer <NUM>.

In some embodiments, the controller <NUM> undertakes steps <NUM>-<NUM> automatically if a suitable training model is not identified at step <NUM>. In other embodiments, the controller <NUM>, at step <NUM>, instructs the computer <NUM> to notify the user thereof that a suitable training model is not available in the database <NUM>. In some embodiments, the controller <NUM> further instructs the computer <NUM> (also at step <NUM>) to prompt the user whether the HCIS <NUM> should be configured in accordance with the output imaging configuration received at step <NUM>. If the computer <NUM> transmits to the controller <NUM> that the user responded affirmatively to such prompt, the controller <NUM> undertakes the steps <NUM>-<NUM>.

Referring to <FIG>, some embodiments of the HCIS <NUM> include a machine learning system trainer <NUM> to develop additional training models for use with the HCIS <NUM> to provide additional enhancement functionalities.

<FIG> is a flowchart <NUM> of the steps undertaken by the machine learning system trainer <NUM> to develop a new training model for use with the machine learning system <NUM>. Referring to <FIG> and <FIG>, at step <NUM>, the machine learning system trainer <NUM> receives a request to develop the new training model. The request includes the input imaging configuration and the output imaging configuration that are associated with the new training model.

At step <NUM>, the machine learning system trainer <NUM> receives an indication that a microplate has been loaded in the HCIS <NUM>.

At step <NUM>, the machine learning system trainer <NUM> directs the controller <NUM> to configure the HCIS <NUM> in accordance with the input imaging configuration specified in the request received at step <NUM>. At step <NUM>, the machine learning system trainer <NUM> directs the controller <NUM> to capture a plurality of training images and store such images in a datastore (not shown) that is accessible by the machine learning system trainer <NUM> and the machine learning system <NUM>.

At step <NUM>, the machine learning system trainer <NUM> directs the controller <NUM> to configure the HCIS <NUM> in accordance with the output imaging configuration specified in the request received at step <NUM>. At step <NUM>, the machine learning system trainer <NUM> directs the controller <NUM> to acquire a plurality of ground truth images. Each ground truth image is associated with a particular training image captured at step <NUM>. Such ground truth image and the particular training image are both images taken using the input imaging configuration and the output imaging configuration, respectively, of an identical portion of the biological sample <NUM> disposed in the microplate <NUM>.

In some embodiments, the machine learning system trainer <NUM> may apply additional image processing functions to each ground truth image in accordance with the output imaging configuration. Such additional image processing functions may include lens field curvature correction, deconvolution, contrast enhancement, shading correction, image flattening, image stretching, denoising, and the like.

At step <NUM>, the machine learning system trainer <NUM> trains the machine learning system <NUM> using a first subset of the plurality training images and the ground truth images corresponding to such training images in the first subset. In particular, the machine learning system trainer <NUM> operates the machine learning system <NUM> with a selected one of the first subset of training images and receives a predicted image generated by the machine learning system <NUM>. For each pixel of the predicted image, the machine learning system trainer <NUM> calculates an error value between such predicted pixel and a corresponding pixel of the ground truth image associated with the selected training image. Such error value may be calculated using, for example, a loss function such as a weighted categorical cross entropy function. The error values calculated for all of the pixels of the predicted image are used to adjust the parameters of the machine learning system <NUM>, for example, using backpropagation, as would be understood by one who has ordinary skill in the art. The machine learning system trainer <NUM> undertakes developing such error values and adjusting of the parameters with all of the images that comprise the first subset of the training images.

At step <NUM>, the machine learning system trainer <NUM> evaluates the performance of the trained machine learning system developed at step <NUM>. In particular, the machine learning system trainer <NUM> selects a second subset of training images as evaluation images and presents each of the evaluation images as an input to the machine learning system <NUM>. The machine learning system <NUM> generates a predicted image in response to each evaluation image presented as an input. For each evaluation image, the machine learning system trainer <NUM> compares the pixels of the predicted image generated therefrom by the machine learning system <NUM> to corresponding pixels of the ground truth image associated with such evaluation image and develops an aggregate error value. All of the aggregate error values developed in this manner are combined to form an aggregate error metric (e.g., percent of pixels of the predicted images are within a predetermined threshold of corresponding pixels of the ground truth images).

At step <NUM>, the machine learning system trainer <NUM> compares the aggegate error metric to a predetermined acceptable error, and if the aggregate error metric is greater than the predetermined error, machine learning system trainer <NUM> proceeds to step <NUM> to further train the machine learning system <NUM> with a further subset of training images and corresponding ground truth images. In some embodiments, the machine learning system trainer <NUM> may instruct the computer <NUM> to display the aggegate error metric and query the user of such computer <NUM> whether to undertake further training. In other embodiments, the machine learning system trainer <NUM>, also at step <NUM>, determines whether to undertake additional training in accordance with a quantity training images that have been used for training, a quantity of iterations of training (at step <NUM>) that have been undertaken, a rate of improvement in the aggregate error metric between successive training iterations, an amount of time undertaken for training, and other such conditions apparent to one who has ordinary skill in the art. If additional training is warranted, the machine learning system trainer <NUM> proceeds to step <NUM>.

In some embodiments, a metric that does not depend on a pixel-by-pixel comparison of the predicted image generated from the evaluation image and the gound truth image associated with the evaluation image may be developed. For example, the machine learning system trainer <NUM> may run an object classification operation on the predicted image and the ground truth image and compare the number and types of objects identified in each image. The percent of objects of each type that match may be used to evaluate performance of the trained machine learning system <NUM>, to develop the aggregate metric, and/or presented to the user of the computer <NUM>.

In some cases, the machine learning system trainer <NUM>, at step <NUM>, may determine that the aggregate error metric is greater the predetermined acceptable error but that additional training is not warranted (e.g., if the aggregate error metric is not improving). In such cases, the machine learning system trainer <NUM> may instruct the computerto display the aggregate error metric with a message that such aggregate error metric is greater than the predetermined acceptable error and not undertake additional training.

If at step <NUM>, the machine learning system trainer <NUM> determines that additional training is not to be undertaken, then, at step <NUM>, the machine learning system trainer <NUM> retrieves the parameters of the trained machine learning system <NUM> and saves such parameters in the database <NUM> as a training model associated with the input and output imaging configurations received at step <NUM>. Thereafter, the machine learning system trainer <NUM> exits.

In some embodiments, at step <NUM>, the machine learning system trainer <NUM> transmits the training model developed thereby to the user computer <NUM> or to another computer remote from the HICS <NUM> for storage.

The HCIS <NUM> may include a robotic microplate loader (not shown) apparent to one who has ordinary skill in the art and the HCIS <NUM> may be operated to automatically generate images of the wells <NUM> of a plurality of microplates <NUM>. <FIG> is a flowchart <NUM> of the steps undertaken by the controller <NUM> to operate the HCIS <NUM> in this manner.

The user loads the microplates <NUM> to be imaged into a holding area of the microplate loader and directs the computer <NUM> to send a request to the HCIS <NUM> to scan the microplates <NUM> that have been loaded.

Referring to <FIG> and <FIG>, at step <NUM> the controller <NUM> receives the request for imaging from the computer <NUM> that includes an output imaging configuration. At step <NUM>, the controller <NUM> directs the robotic microplate loader to load a first microplate.

At step <NUM> the controller <NUM> selects and loads from the database <NUM> a training model associated with the output imaging configuration. In particular, the controller <NUM> selects a training model that is associated with the output imaging configuration.

At step <NUM><NUM>, controller <NUM> configuresthe HCIS <NUM> in accordance with the input imaging configuration associated with the training model selected at step <NUM>, generates one or more test images of the samples <NUM> disposed in the microplate <NUM>, and transmits such test images to the computer <NUM>, undertaking steps identical to steps <NUM>-<NUM> shown in <FIG> for each such test image. In addition, the controller <NUM> instructs the computer <NUM> to display such test images and to prompt the user to verify that the images are acceptable.

At step <NUM>, the controller <NUM> checks whether the user indicated that the test images are acceptable and proceeds to step <NUM>. Otherwise, at step <NUM> the controller <NUM> receives additional adjustments to the HCIS <NUM> (e.g., a change in illumination, a change in focus, a change in the input and/or output imaging configuration(s), change of sample, change in architecture of the machine learning system <NUM>, a change in the hyperparameters of the machine learning system <NUM>, etc.) or the output imaging configuration. The controller <NUM> adjusts the components of the HCIS <NUM> in accordance with the adjustments. Thereafter, the controller <NUM> returns to step <NUM>. At step <NUM>, if the output imaging configuration has changed, the controller <NUM> loads a training model that is associated with the modified output imaging configuration.

In some embodiments, the user may adjust the configuration at step <NUM> to build a more robust training module. For example, the user may direct the controller <NUM> to develop a training model using a plurality of first, second, and third sets of training images captured using <NUM>, <NUM>, and <NUM> millisecond exposures, respectively, and ground truth images captured using <NUM> millisecond exposure. Alternatively, the user may direct the controller <NUM> to develop a model using training fluorescent images captured using different wavelengths of light to generate an output image associated with a particular output imaging configuration. A machine learning system configured using a training model developed using multiple input imaging configurations and associated with one output imaging configuration may be able to generate an output image associated with the output imaging configuration from input images captured using a plurality of different input imaging configurations. As would be appreciated by one who has ordinary skill in the art, the plurality of different imaging configurations should be related to those used to develop the training model.

At step <NUM>, the controller <NUM> automatically generates and transmits images of all of the wells <NUM> in the microplate <NUM> loaded at step <NUM>. As should be apparent to one who has ordinary skill in the art, for each image generated, the controller <NUM> adjusts the stage <NUM> to position an unimaged well <NUM> (or portion thereof) of the microplate <NUM> relative to the lens <NUM>, illumination source <NUM>, and image sensor <NUM>. The controller <NUM> then undertakes the steps identical to steps <NUM>-<NUM> shown in <FIG> to obtain an image of such well <NUM> and transmit the acquired image to an image storage device (not shown) accessible by the computer <NUM>.

At step <NUM>, the controller <NUM> determines if any additional microplates remain to be imaged. If so, the controller <NUM> proceeds to step <NUM> to direct the robotic microplate loader (not shown) to load an unimaged microplate onto the stage <NUM>. Otherwise, the controller <NUM> exits.

Referring to <FIG>, it should be apparent to one who has ordinary skill in the art that the machine learning system <NUM> may be implemented using a graphical processing unit (GPU). Referring also to <FIG> and <FIG>, the machine learning system <NUM> includes machine learning system controller <NUM> and one or more GPU(s) <NUM>. At step <NUM> (<FIG>) the controller <NUM> directs the machine learning system controller <NUM> to configure each GPU <NUM> with the training model selected at step <NUM> (<FIG>). Thereafter, the image generator <NUM> provides each image generated thereby to the machine learning system controller <NUM>. The machine learning system controller <NUM> selects a GPU <NUM> that is not busy and transfers the generated image thereto for processing. As each GPU <NUM> completes generating an output image, the machine learning system controller <NUM> receives the output image developed by such GPU and transfers such output image to the controller <NUM>, which in turn transmits the output image to the computer <NUM>. In this manner, the automated scanning process illustrated in <FIG> may be implemented in a high throughput manner. Further, the performance of the HCIS <NUM> may be scaled up or down in accordance with the number GPU(s) <NUM> included therein. Further, it should be apparent that the machine learning system controller <NUM> may use GPU(s) that are installed within the HCIS <NUM> and/or instances of GPU(s) that are available through cloud services providers such as Amazon A WS, Google Cloud, and the like.

Referring once again to <FIG>, in some embodiments the machine learning system <NUM> is a convolutional neural network. In some embodiments, the machine learning system or neural network <NUM> is configured using AutoML and NASNet technologies developed by Google Inc. of Mountain View, California. It should be apparent that other neural network technologies known to those who have skill in the art may be used including, for example, a fully convolutional DenseNet, neural networks optimized for machine vision applications, and the like. It should be apparent that the machine learning system <NUM> may be another type of machine learning system including a random forest tree and the like.

Although the embodiments described in the foregoing are directed to the use of a machine learning system <NUM> in an HCIS <NUM> to generate output images associated with output imaging configurations from input images captured using a different input imaging configurations of the HCIS <NUM>, one of ordinary skill in the art would appreciate that such embodiments may be adapted for use with other types of microscopy and/or imaging systems.

Although, the HCIS <NUM> described in the foregoing is described as being used to generate individual <NUM>-dimensional images, such HCIS <NUM> may be adapted to generate a series of two-dimensional transmitted light images of a sample <NUM> disposed on the microplate <NUM> taken at different focal points that represent a three-dimensional representation of such sample <NUM>, wherein the images that comprise such series are associated with a substantially identical location of the microplate <NUM>, and the corresponding images of the series are associated with a different focal point (i.e., different Z location). In some embodiments, all of the images that comprise a series may be simultaneously provided to the machine learning system <NUM> and the machine learning system <NUM> generates a series of corresponding output images.

In some embodiments, the machine learning system <NUM> may be trained to perform super-resolution in three dimensions. For example, such machine learning system <NUM> may generate an output series of images from an input series of captured images. For example, the input series of image may comprise a plurality of captured images wherein the focal distance between successive captured images is varied by a first predetermined amount. The output series of images generated comprises a series of images that represents a series of images that would be captured if the HCIS <NUM> were operated to vary the focal distance between successive images by a second predetermined amount. Typically, the second predetermined amount would be smaller than the first predetermined amount. For example, the focal distance between successive images of the input series may be varied by <NUM> microns, and the machine learning system <NUM> may generate from such input series an output series that represents successive images that would have captured if the focal distance were varied by <NUM> microns. It should be apparent to one who has skill in the art that a training model may be developed to training the machine learning system <NUM> to perform such super-resolution using the steps described above in connection with <FIG>.

It should be apparent to one who has ordinary skill in the art that the foregoing disclosure may be applied to other types of imaging systems such as, for example, a confocal microscopy system, a microscopy system that uses structured illumination, and the like. Further, it should be apparent that a training model associated with an imaging configuration associated with a first type of HCIS (e.g., widefield microscopy) and an outputimaging configuration with a second type of HCIS (e.g., confocal microscopy) may be used to configure the untrained machine learning system to develop the trained machine learning system. The first type of HCIS having the trained machine learning system may then be used to generate output images representative of an image captured using the second type of HCIS.

It should be apparent to those who have skill in the art that any combination of hardware and/or software may be used to implement the HCIS <NUM> described herein. It will be understood and appreciated that one or more of the processes, sub-processes, and process steps described in connection with <FIG> may be performed by hardware, software, or a combination of hardware and software on one or more electronic or digitally-controlled devices. The software may reside in a software memory (not shown) in a suitable electronic processing component or system such as, for example, one or more of the functional systems, controllers, devices, components, modules, or sub-modules schematically depicted in <FIG>. The software memory may include an ordered listing of executable instructions for implementing logical functions (that is, "logic" that may be implemented in digital form such as digital circuitry or source code, or in analog form such as analog source such as an analog electrical, sound, orvideo signal). The instructions may be executed within a processing module or controller (e.g., the controller <NUM>, the image generator <NUM>, the machine learning system <NUM>, and the machine learning system trainer <NUM> of <FIG> and the machine learning system controller <NUM> and GPU(s) <NUM> of <FIG>), which includes, for example, one or more microprocessors, general purpose processors, combinations of processors, digital signal processors (DSPs), field programmable gate arrays (FPGAs), or application-specific integrated circuits (ASICs). Further, the schematic diagrams describe a logical division of functions having physical (hardware and/or software) implementations that are not limited by architecture or the physical layout of the functions. The example systems described in this application may be implemented in a variety of configurations and operate as hardware/software components in a single hardware/software unit, or in separate hardware/software units.

The executable instructions may be implemented as a computer program product having instructions stored therein which, when executed by a processing module of an electronic system, direct the electronic system to carry out the instructions. The computer program product may be selectively embodied in any non-transitory computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as an electronic computer-based system, processor-containing system, or other system that may selectively fetch the instructions from the instruction execution sy stem, apparatus, or device and execute the instructions. In the context of this document, computer-readable storage medium is any non-transitory means that may store the program for use by or in connection with the instruction execution system, apparatus, or device. The non-transitory computer-readable storage medium may selectively be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. A non-exhaustive list of more specific examples of non-transitory computer readable media include: an electrical connection having one or more wires (electronic); a portable computer diskette (magnetic); a random access, i.e., volatile, memory (electronic); a read-only memory (electronic); an erasable programmable read only memory such as, for example, Flash memory (electronic); a compact disc memory such as, for example, CD-ROM, CD-R, CD-RW (optical); and digital versatile disc memory, i.e., DVD (optical).

It will also be understood that receiving and transmitting of signals or data as used in this document means that two or more systems, devices, components, modules, or sub-modules are capable of communicating with each other via signals that travel over some type of signal path. The signals may be communication, power, data, or energy signals, which may communicate information, power, or energy from a first system, device, component, module, or sub-module to a second system, device, component, module, or sub-module along a signal path between the first and second system, device, component, module, or sub-module. The signal paths may include physical, electrical, magnetic, electromagnetic, electrochemical, optical, wired, or wireless connections. The signal paths may also include additional systems, devices, components, modules, or sub-modules between the first and second system, device, component, module, or sub-module.

The use of the terms "a" and "an" and "the" and similar references in the context of describing the invention (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradictedby context. The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illuminate the disclosure and does not pose a limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

Claim 1:
A high-content imaging system (<NUM>), comprising:
a stage (<NUM>) adapted to have a sample (<NUM>) disposed thereon;
a controller (<NUM>) adapted to receive a request that includes a specification of an output imaging configuration and in response the controller is adapted to:
(<NUM>) select a training model (<NUM>) associated with the output imaging configuration,
(<NUM>) determine an input imaging configuration associated with the training model (<NUM>), and
(<NUM>) configure the high-content imaging system in accordance with the
input imaging configuration;
an image generator (<NUM>) adapted to generate an image of the sample (<NUM>) disposed on the stage (<NUM>) in accordance with the input imaging configuration; and
a machine learning system (<NUM>) adapted to automatically receive the image of the sample (<NUM>) generated by the image generator (<NUM>), and in response, automatically generates an output image, wherein the machine learning system (<NUM>) is configured using the training model (<NUM>) so that when the machine learning system (<NUM>) is presented with an input image acquired in accordance with the input imaging configuration, the machine learning system (<NUM>) generates an output image in accordance with the output imaging configuration,
- wherein the output image represents the image that is generated if the high-content imaging system had been configured in accordance with the output imaging configuration.