Patent Description:
Image recognition algorithms enable machines to distinguish between different images to and the identity objects represented in the images and, in various instances, to classify the images accordingly. In order for an image recognition algorithm to perform these tasks, however, the algorithm must first be trained with a training dataset. Training datasets include images useable to train the algorithm as well as "truth labels" that indicate what is represented in the images. In various instances, these truth labels are assigned by a human who has reviewed the images in the training dataset.

The present disclosure concerns the use of machine learning techniques to assist a user in labeling images with classification labels to prepare a training dataset for a target machine learning algorithm. Because the accuracy of the target machine learning algorithm depends in part on the accuracy of the training dataset, preparing a training dataset typically requires that a user review each individual image and apply a classification label. For a training dataset comprised of hundreds or thousands of images, this user review process can be time consuming and expensive, especially when a user with specialized training is needed to review the images. The subject matter of the independent claims reduces the required time and cost of the user review process. The subject matter of the dependent claims provide advantageous embodiments.

In various embodiments, a computer system uses a first machine learning algorithm to derive features from a plurality of images. Then, the computer system uses a dimensionality reduction algorithm to reduce the dimensionality of a database of these image-derived features. After that, the computer system uses clustering algorithm identifies clusters of datapoints in the dimensionally-reduced dataset. The computer system then generates a visual representation of the datapoints of the dimensionally-reduced dataset and causes the display of one or more of the images for the user. The computer system receives user input applying user classification labels to the images.

<CIT> relates to deep learning techniques to generate embeddings for high dimensional data objects that can both simulate prior art embedding algorithms. These deep learning techniques comprise generating an initial formal embedding of selected high-dimensional data objects using any of the traditional formal embedding techniques. Next a deep embedding architecture is designed. The deep embedding architecture includes choosing the types and numbers of inputs and outputs, types and number of layers, types of units/nonlinearities, and types of pooling, for example, among other design choices, typically in a convolutional neural network. Further a training strategy is designed and the parameters of a deep embedding architecture are tuned to reproduce, as reliably as possible, the generated embedding for each training sample. The trained deep embedding architecture is subsequently used to convert new high dimensional data objects into approximately the same embedded space as found in the first step.

The present disclosure also concerns a user interface operable to present information to a user and to receive input from the user to apply classification labels for images to be included in a training dataset for a target machine learning algorithm. In various embodiments, the user interface includes a two or three-dimensional representation of dimensionally-reduced dataset of image data that was derived from a plurality of images and one or more of the plurality of images. The user interface is useable to receive user input to apply user classification labels to the images to prepare a labeled training dataset for training a target machine learning algorithm to classify image.

This disclosure includes references to "one embodiment" or "an embodiment. " The appearances of the phrases "in one embodiment" or "in an embodiment" do not necessarily refer to the same embodiment. Particular features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

Within this disclosure, different entities (which may variously be referred to as "units," "circuits," other components, etc.) may be described or claimed as "configured" to perform one or more tasks or operations. This formulation-[entity] configured to [perform one or more tasks]-is used herein to refer to structure (i.e., something physical, such as an electronic circuit). More specifically, this formulation is used to indicate that this structure is arranged to perform the one or more tasks during operation. A structure can be said to be "configured to" perform some task even if the structure is not currently being operated. A "computer system configured to generate a dataset" is intended to cover, for example, a computer system has circuitry that performs this function during operation, even if the computer system in question is not currently being used (e.g., a power supply is not connected to it). Thus, an entity described or recited as "configured to" perform some task refers to something physical, such as a device, circuit, memory storing program instructions executable to implement the task, etc. This phrase is not used herein to refer to something intangible. Thus, the "configured to" construct is not used herein to refer to a software entity such as an application programming interface (API).

The term "configured to" is not intended to mean "configurable to. " An unprogrammed FPGA, for example, would not be considered to be "configured to" perform some specific function, although it may be "configurable to" perform that function and may be "configured to" perform the function after programming.

As used herein, the terms "first," "second," etc. are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless specifically stated. For example, references to "first" and "second" machine learning algorithms would not imply an ordering between the two unless otherwise stated.

As used herein, the term "based on" is used to describe one or more factors that affect a determination. This term does not foreclose the possibility that additional factors may affect a determination. That is, a determination may be solely based on specified factors or based on the specified factors as well as other, unspecified factors. Consider the phrase "determine A based on B. " This phrase specifies that B is a factor is used to determine A or that affects the determination of A. This phrase does not foreclose that the determination of A may also be based on some other factor, such as C. This phrase is also intended to cover an embodiment in which A is determined based solely on B. As used herein, the phrase "based on" is thus synonymous with the phrase "based at least in part on.

As used herein, the word "module" refers to structure that stores or executes a set of operations. A module refers to hardware that implements the set of operations, or a memory storing the set of instructions such that, when executed by one or more processors of a computer system, cause the computer system to perform the set of operations. A module may thus include an application-specific integrated circuit implementing the instructions, a memory storing the instructions and one or more processors executing said instructions, or a combination of both.

Referring now to <FIG>, a block diagram of an exemplary embodiment of a computer system <NUM> is depicted. In various embodiments, computer system <NUM> receives a plurality of images <NUM> and prepares a training dataset <NUM> using the plurality of images <NUM> and user input. In various embodiments, computer system <NUM> employs a first machine learning algorithm <NUM>, a dimensionality reduction algorithm <NUM>, a clustering algorithm <NUM>, a second machine learning algorithm <NUM>, and a user interface <NUM> to prepare training dataset <NUM> using images <NUM>.

In various embodiments, computer system <NUM> is any of a number of computing systems configured to receive images <NUM>, receive user input, and prepare training dataset <NUM>. In various embodiments, computer system <NUM> is implemented with a single computing system (e.g., a single server, desktop computer, laptop computer, tablet computer, smart phone) but in other embodiments is implemented with a plurality of computers working together (e.g., a cloud of servers). In various embodiments, a first portion of computer system <NUM> (e.g., a server or cloud of servers) is configured to perform the various algorithms and a second portion of computer system <NUM> (e.g., a laptop computer, a tablet computer) is configured to implement user interface <NUM> to present information to the user and to receive information from the user.

In various embodiments, the plurality of images <NUM> can be any of a group of images that, with metadata such as a user classification label, are useable to be included in a training dataset <NUM>. Images <NUM> include images of cells. In various embodiments, these images of cells include a plurality of multispectral images of cells, a plurality of multimodal images of the cells, or both. Such images, for example, may have been created using fluorescence imagery in which a specimen is dyed with fluorescent dye and excited with a light source. The disclosed techniques, however, are not merely limited to images of cells and can be used on any type of images that can be included in a training dataset <NUM> (e.g., picture of plants, picture of animals, pictures of taken from a vehicle traveling on a street of the surroundings, image of human faces, etc.) In various embodiments, the number of images <NUM> can vary depending on criteria for the target machine learning algorithm, the amount of acceptable training time for target machine learning algorithm, and the desired amount of precision in target machine learning algorithm. For example, a larger set of images <NUM> can be turned into a larger training dataset <NUM>. The amount of time needed to train the target machine learning algorithm increases as the size of the training dataset <NUM> increase, but in various embodiments the precision of the target machine learning algorithm may also increase. In various embodiments, the plurality of images <NUM> includes between <NUM> and <NUM> images <NUM>. In various embodiments, images <NUM> are randomly selected from a larger pool of images.

In various embodiments, training dataset <NUM> is useable for training a target machine learning algorithm to classify other images (i.e., images other than images <NUM>). In such embodiments, training dataset <NUM> includes some or all of images <NUM> and the user classification labels applied to these images <NUM> as discussed herein. As used herein, "target machine learning algorithm" includes any image recognition algorithm that, when trained with a training dataset <NUM>, is useable to classify images. In the embodiments discussed herein in connection to <FIG>, training dataset <NUM> include images of cells with a user classification label identifying the number of nuclei in each image (e.g., one, two, or three or more). After being trained with training dataset <NUM>, the target machine learning algorithm is operable to determine whether other images include one, two, or three or more nuclei.

In various embodiments, first machine learning algorithm <NUM> is any of a number of algorithms executable to analyze images (e.g., by analyzing pixels of the image) and derive features from these images <NUM> to generate a dataset of image-derived features. In various embodiments, first machine learning algorithm <NUM> is a convolutional neural network (CNN). In some of such embodiments, first machine learning algorithm <NUM> is the Inception V3 Convolutional Neural Network, which has been trained on a large database of images from ImageNet. In embodiments where first leaning algorithm <NUM> is a CNN, the image-derived features for an image are "bottleneck features" that can be used to describe the contents of the image and to differentiate between different images <NUM>. In various instances, there may be thousands of bottleneck features per image. The output of the first machine learning algorithm <NUM> includes a multi-dimensional dataset (e.g., one dimension per feature of the image <NUM>) in various embodiments. For example, after analyzing images <NUM>, first machine learning algorithm <NUM> generates a dataset of <NUM> features per channel of images <NUM>. In various instances, the plurality of images <NUM> includes between one to twelve channels of images.

Dimensionality reduction algorithm <NUM> is any of a number of algorithms executable to reduce the dimensionality of the multi-dimensional dataset output by first machine learning algorithm <NUM> to a dimensionally-reduced dataset by reducing the number of random variables under consideration by obtaining a set of principal variables. In various embodiments, dimensionality reduction algorithm <NUM> reduces the dimensionality by several orders of magnitude. For example, in some embodiments first machine learning algorithm <NUM> outputs <NUM> features for each image <NUM>, and dimensionality reduction algorithm <NUM> reduces the dimensionality of this dataset to three or fewer dimensions. According to the invention, dimensionality reduction algorithm <NUM> is the principal component analysis (PCA). Other non-claimed alternatives include uniform manifold approximation and projection (UMAP), or t-distributed stochastic neighbor embedding (t-SNE).

Dimensionality reduction algorithm <NUM> is also executable to take input from second machine learning algorithm <NUM>. As discussed herein, second machine learning algorithm <NUM> is executable to output predicted classification labels for unlabeled images <NUM> based on user classification labels received via user interface <NUM>. Dimensionality reduction algorithm <NUM> is executable, for each unlabeled image <NUM>, to take these predicted classification labels into account along with the multi-dimensional dataset output by first machine learning algorithm <NUM> to generate another reduced-dimension dataset having, for example, three or fewer dimensions.

In various embodiments, dimensionality reduction algorithm <NUM> is executable to output this reduced-dimension dataset to clustering algorithm <NUM>. In such embodiments, clustering algorithm <NUM> is executable to determine clusters of datapoints within the reduced-dimension dataset. Clustering algorithm <NUM> may be any of a number of suitable clustering algorithms including but not limited to k-means clustering or spectral clustering algorithms. In various embodiments, the number of clusters is set by the user, and the various datapoints in the reduced-dimension dataset is grouped into the nearest cluster. In various embodiments, the plurality of clusters is equal to X times Y clusters, where X is the number of groups into which a user wants to classify the images (e.g., the number of potential user classification labels) and Y is greater than or equal to <NUM>. In various embodiments, Y is equal to five, for example, although other numbers can be used. In various embodiments during the second or later iteration (i.e., the user has input user classification labels and second machine learning algorithm <NUM> and dimensionality reduction algorithm <NUM> have output a dimensionally-reduced dataset with predicted classification labels) clustering algorithm <NUM> clusters datapoints corresponding to unlabeled images to the nearest classification label. This clustering is presented to the user as predicted classification labels via user interface <NUM>.

In various embodiments, user interface <NUM> is executable to present information to the user and receive input from the user such that the user can prepare training dataset <NUM> using images <NUM>. In various embodiments, user interface <NUM> is a graphical user interface (GUI) that is executable to present a visual representation (e.g., visual representation <NUM> discussed herein in reference to <FIG>) of the datapoints in the reduced-dimension dataset as icons grouped by cluster in which each icon represents one or more particular datapoints. In various embodiments, various portions of user interface <NUM> are selectable to cause the display of the one or more images <NUM> associated with the one or more particular datapoints such as the icons themselves, a list of the clusters in the dataset, a list of user classification labels, a list of predicted classification labels, or a combination. User interface <NUM> is also executable to receive user input of a user classification label for various ones of the images <NUM>. User interface <NUM> is discussed in further detail herein in reference to <FIG> and <FIG>.

In various embodiments, second machine learning algorithm <NUM> is executable to predict classification labels for unlabeled images <NUM> based on user classification labels input by the user for other images <NUM>. In various embodiments, second machine learning algorithm <NUM> is an iterative optimization algorithm. In various embodiments, second machine learning algorithm <NUM> can be any suitable supervised learning algorithm including but not limited to a stochastic gradient descent (SGD) model with logarithmic loss or Random Forest model. As discussed herein, in various embodiments second learning algorithm <NUM> is executable to output to dimensionality reduction algorithm <NUM>. In turn, clustering algorithm <NUM> clusters the datapoints for the unlabeled images <NUM> into the nearest classification label. In such embodiments, the results of this clustering are presented to the user using user interface <NUM> as predicted classification labels. In various instances, the user responds to the predicted classification labels by accepting them as user classification labels or rejecting them and either selecting a different user classification label, marking the classification label for the image <NUM> as unknown (e.g., leaving it to a second user to review), or excluding the image <NUM> from training dataset <NUM> altogether. In various embodiments, the loop illustrated in <FIG> from dimensionality reduction algorithm <NUM> to clustering algorithm <NUM> to user interface <NUM> to second machine learning algorithm <NUM> to dimensionality reduction algorithm <NUM> iterates until all of images <NUM> have been labeled or excluded. The processes of labeling images <NUM> is discussed in further detail in reference to <FIG>, <FIG>, and <FIG> herein.

In various embodiments, the techniques disclosed herein enable a user to more quickly and accurately prepare a training dataset <NUM> from a plurality of unlabeled images <NUM>. Rather than the user having to look at each image <NUM> in isolation and assign a user classification label to that image <NUM> for inclusion in the training dataset <NUM>, instead the various algorithms employed by computer system <NUM> provide the user with various aids in decision making to make the labeling process more efficient. This is especially important in instances where the decision of which label to apply to a particular image <NUM> is reviewed by an individual with particular training (for example, a microbiologist or radiologist) and whose labor time is expensive.

In various embodiments discussed herein, first machine learning algorithm <NUM>, dimensionality reduction algorithm <NUM>, and clustering algorithm <NUM> use machine-learning techniques to pre-sort images <NUM> into various clusters that are predicted to share visual characteristics and, in many cases, will be given the same user classification labels. A visual representation of the clustering (e.g., in a visual representation <NUM> discussed in connection to <FIG>) as well as the images being labeled are displayed using user interface <NUM> in various embodiments. As discussed herein, the user is able to review the various clusters and assign user classification labels to multiple images <NUM> at the same time (e.g., by highlighting multiple images <NUM> and applying a label to each highlighted image). As discussed herein, this process of clustering uses "unsupervised" (i.e., user input was not used in the initial clustering) training techniques that are then reviewed by a user to prepare labeled material for a training dataset <NUM>.

As discussed herein, after a number of user classification labels have been input, using second learning algorithm <NUM>, computer system <NUM> is able to take the user's input into account to further streamline the labeling process in various embodiments. As discussed herein, by using second learning algorithm <NUM> computer system <NUM> is operable to predict which classification labels might be correct for some (or all) of the images <NUM> that remain unlabeled. In various embodiments, dimensionality reduction algorithm <NUM> factors in the output of second learning algorithm <NUM> into generating a second dimensionally-reduced dataset that is then clustered using clustering algorithm <NUM>. As discussed herein, user interface <NUM> is updated to show the user the clusters of predicted user classification labels (e.g., in a visual representation <NUM> discussed in connection to <FIG> and <FIG>). As discussed herein, the user is able to review the various clusters and assign user classification labels to multiple images <NUM> at the same time (e.g., by highlighting multiple images <NUM> and applying a label to each highlighted image). As discussed herein, this process of clustering uses "semi-supervised" (i.e., the previous user input was used in the revised clustering, but the user has not yet reviewed all of the images <NUM>) training techniques that are then reviewed by a user to prepare labeled material for a training dataset <NUM>. Accordingly, in various embodiments, the techniques disclosed herein provide a user who is labeling images <NUM> for training dataset <NUM> with a guided path from unsupervised clustering to semi-supervised predictions while providing visualizations and an intuitive user interface to aid in decision making.

Referring now to <FIG>, a sampling of images <NUM> is depicted. In various embodiments, each image <NUM> includes a visual portion and metadata <NUM> (e.g., the name of the particular image <NUM>, when it was created, etc.). As discussed herein in connection to <FIG>, when a particular image <NUM> is being reviewed and labeled (e.g., using user interface <NUM>) the image is represented using an object <NUM> in various embodiments. In various embodiments, the object <NUM> includes metadata <NUM> about the particular image (such as the name of the image shown in <FIG>) and is selectable. As discussed herein, selecting object <NUM> allows the user to apply a user classification label (and/or respond to a predicted classification label) in various embodiments.

<FIG> also includes a small number of examples of image-derived features <NUM>. In various embodiments, first machine learning algorithm <NUM> derives various features from images <NUM>. In various instances, these features are represented by mathematical description of the pixel data of the image <NUM>. Represented visually, however, these image-derived features <NUM> are portions of the image <NUM> that collectively describe the image <NUM> such that it can be differentiated from the other images <NUM>. Accordingly, three image-derived features 204a, 204b, and 204c are shown in <FIG>, although the number of image-derived features may be much greater than three as discussed above (e.g., thousands of features per image <NUM> in various embodiments).

Referring now to <FIG>, a flowchart illustrating an embodiment of a training dataset creation method <NUM> is shown. In various embodiments, the various actions associated with method <NUM> are performed with computer system <NUM>.

At block <NUM>, a user inputs an unlabeled (or insufficiently labeled) training dataset (e.g., a plurality of images <NUM>) for labeling to prepare training dataset <NUM>. As discussed herein, the user may randomly select the images <NUM> to label from a larger collection of images. The user may input the images <NUM> by any suitable method including but not limited to inserting storage media (e.g., a disk or hard drive) or downloading the images <NUM> to computer system <NUM>. In various embodiments in which some of the techniques discussed herein are performed by computer systems <NUM> implemented on remote clouds of computers, the user may upload the images <NUM> to the cloud for processing.

At block <NUM>, computer system <NUM> derives (e.g., with first machine learning algorithm <NUM>) features from the pixel data from the training dataset (e.g., features <NUM> shown in <FIG>). At block <NUM>, computer system <NUM> (e.g., with dimensionality reduction algorithm <NUM>) reduces the dimensionality of the features <NUM>. In various instances, dimensionality reduction is performed on the dataset of derived features <NUM> prepared by first machine learning algorithm <NUM> using the plurality of images. In other instances (e.g., when method <NUM> proceeds to block <NUM> from block <NUM>), dimensionality reduction is performed on the dataset of derived features <NUM> while taking into account user classification labels that have been applied to some of the plurality of images <NUM>. In either instance, dimensionality reduction algorithm <NUM> receives a relatively large dimensional dataset each of the plurality of images <NUM> (e.g., a dataset of <NUM> features of an image <NUM>) and reduces it down to a substantially smaller number of dimensions such as two dimensions in some embodiments or three dimensions in other embodiments.

At block <NUM>, computer system <NUM> prepares a visual representation <NUM> (also referred to herein as an "object map") of the datapoints of the dimensionally-reduced dataset. As discussed in further detail in reference to <FIG>, this visual representation <NUM> is a two-dimensional plot with icons representing one or more datapoints in the dimensionally-reduced database in various embodiments. In other embodiments, the visual representation <NUM> is a three-dimensional plot with icons representing one or more datapoints in the dimensionally-reduced database.

At block <NUM>, a determination is made whether to predict classification labels for the plurality of images <NUM>. In various embodiments, computer system <NUM> is configured to make this determination based on the number of user classification labels that have been input. For example, if the percentage of images <NUM> that are label below a threshold (e.g., <NUM>%, <NUM>%, or any other threshold) or when no user classifications have been received, the determination is made automatically and method <NUM> proceeds to block <NUM>. If the percentage is above the threshold, method <NUM> proceeds to block <NUM>. In various embodiments, the determination is made by a user who determines whether method <NUM> should proceed to block <NUM> or block <NUM>, and computer system <NUM> proceeds according to commands from the user.

At block <NUM>, computer system <NUM> clusters the dimensionally-reduced dataset (e.g., with clustering algorithm <NUM>) into a predetermined number of clusters in various embodiments. In iterations of method <NUM> in which no predicted classification labels have been generated, clustering algorithm clusters the datapoints into X times Y clusters, wherein X is the number of groups (e.g., the number of user classification labels) into which a user wants to classify the images; and Y is greater than or equal to <NUM> (e.g., <NUM>, <NUM>, <NUM>). In various embodiments, these clusters are incorporated in visual representations <NUM> discussed herein in connection to <FIG> and <FIG>.

At block <NUM>, having determined to predict classification labels, computer system <NUM> (e.g., with second learning algorithm <NUM>) predicts classification labels for the unlabeled images in the plurality of images <NUM>. In iterations of method <NUM> in which classification labels have been predicted, the various datapoints are clustered into clusters for each user classification label. In such embodiments, datapoints representing images <NUM> that have user classification labels are clustered into the cluster associated with their respective labels and unlabeled datapoints are clustered into the nearest cluster as a predicted classification label. Computer system <NUM> generates a visual representation <NUM> incorporating the predicted classification labels. In various embodiments, this updated visual representation <NUM> appears on a user interface as discussed in further detail in reference to <FIG>, <FIG>, and <FIG>.

At blocks <NUM> and <NUM>, computer system <NUM> receives user input to apply user classification labels. At block <NUM>, computer system <NUM> receives user input to apply user classification labels to one or more unlabeled images <NUM>. In various embodiments, this input is received via a menu appearing on a user interface as discussed in further detail in reference to <FIG>. Similarly, at block <NUM> computer system <NUM> receives user input to apply user classification labels to one or more unlabeled images <NUM> that have been given predicted classification labels in various embodiments. In various embodiments, this input is received via a menu appearing on user interface <NUM> as discussed in further detail in reference to <FIG>. In various embodiments, such user classification labels include labels for the various images <NUM> that describe what is contained in the image for use in training the target machine learning algorithm (e.g., as discussed in <FIG>, the image contains one nucleus, two nuclei, or three or more nuclei). In various embodiments, the user classification label can also be a label excluding the image from the training dataset <NUM>. In various embodiments, the user classification label can be that the label is unknown (e.g., the user is unable to identify what label to apply). In various embodiments, images <NUM> labeled unknown and exclude are not included in training dataset <NUM>. After blocks <NUM> and <NUM>, if some of the plurality of images <NUM> remain unlabeled, method <NUM> loops back to block <NUM> in various embodiments.

Referring now to <FIG>, an example visual representation <NUM> of a dimensionally-reduced dataset for a plurality of images <NUM> is depicted. In the embodiment shown in <FIG>, visual representation <NUM> is a two-dimensional rendering that includes a plurality of icons <NUM> that represent datapoints in the dimensionally-reduced dataset output by dimensionality reduction algorithm <NUM> grouped into clusters <NUM>. In the embodiment shown in <FIG>, the datapoints in the dimensionally-reduced dataset have been grouped into <NUM> clusters, 404a-404o. In various embodiments, this clustering is represented in visual representation <NUM> using one or more techniques including but limited to (a) rendering the visual representation <NUM> such that icons <NUM> that correspond to datapoints in the same cluster <NUM> are positioned closed together, (b) rendering the visual representation <NUM> such that icons <NUM> that correspond to datapoints in the same cluster <NUM> are shaded with a same color (e.g., red for cluster 402a, blue for cluster 402b, green for cluster 402c), (c) rendering the visual representation <NUM> such that icons <NUM> that correspond to datapoints in the same cluster <NUM> are encircled by a polygon, or a combination. In various embodiments, the position of the various icons <NUM> on the two-dimensional embodiment of visual representation <NUM> shown in <FIG> is based on the two dimensions of the dimensionally-reduced dataset (e.g., the X axis coordinate is based on a first dimension and the Y axis coordinate is based on a second dimension). Similarly, when the dimensionally-reduced dataset has three dimensions, visual representation <NUM> is a three-dimensional figure with the position of the various icons <NUM> based on the three dimensions of the dimensionally-reduced dataset (e.g., the X axis coordinate is based on a first dimension and the Y axis coordinate is based on a second dimension, the Z axis coordinate is based on a third dimension). As discussed herein, the number of clusters may vary according to the number of user classification labels and whether predicted classification labels have been generated.

As discussed herein in reference to <FIG> and <FIG>, an updated visual representation <NUM> is generated in various instances displaying the dimensionally-reduced dataset output by dimensionality reduction algorithm <NUM> (this time taking into account the output of second machine learning algorithm <NUM>) grouped into one cluster for each of the user classification labels. In various embodiments, this clustering is represented in visual representation <NUM> using one or more techniques including but limited to (a) rendering the visual representation <NUM> such that icons <NUM> that correspond to datapoints in the same cluster <NUM> are positioned closer together, (b) rendering the visual representation <NUM> such that icons that correspond to datapoints in the same cluster <NUM> are shaded with a same color, (c) rendering the visual representation <NUM> such that icons <NUM> that correspond to datapoints in the same cluster <NUM> are encircled by a polygon, or a combination.

Referring now to <FIG>, various display screens of an exemplary embodiment of a graphical user interface (GUI) <NUM> operated by user interfaced <NUM> in accordance with the disclosed embodiments are illustrated. In various embodiments, GUI <NUM> is displayed on a display screen (e.g., a monitor, a laptop computer display, a tablet computer display) coupled to computer system <NUM> directly (e.g., via an HDMI cable) or indirectly (e.g., streamed to the display screen over a WAN and/or LAN). As discussed herein, GUI <NUM> is useable to present information to a user and receive input from the user (e.g., input classifying images <NUM>) to prepare labeled training dataset <NUM> for training a target machine learning algorithm.

In each screen of GUI <NUM>, a plurality of regions is used to display various information discussed herein. In various embodiments, each screen includes a first region <NUM> including a two-dimensional visual representation <NUM> (or an updated visual representation <NUM>). As discussed herein, visual representation <NUM> represents a dimensionally-reduced dataset of image data that was derived from a plurality of images. In various embodiments, two-dimensional visual representation <NUM> includes a plurality of icons <NUM> and indications of clusters <NUM> within the dataset. In various embodiments, each screen also includes a second region including one or more of the plurality of images <NUM>. In various embodiments, the various screens include a third region <NUM> to display a list of the identified clusters (and in embodiments the number of images <NUM> grouped into each). In various other embodiments, the various screens include a updated third region <NUM> to display a list of the predicted classification labels (and in embodiments the number of images <NUM> grouped into each). In various embodiments, the various screens include a fourth region <NUM> to display a list of the user classification labels (and in embodiments the number of images <NUM> labeled with each). In <FIG>, first region <NUM> is disposed on the right side of GUI <NUM>, second region <NUM> is disponed in the middle of GUI <NUM>, and third region <NUM> (and updated third region <NUM>) and fourth region <NUM> are disposed on the left side of GUI <NUM>, but these various regions can be arranged in any order. The various regions in <FIG> are depicted as being part of the same window, but in other embodiments some or all of the regions may be presented as separate windows. Referring again to <FIG>, the actions of blocks <NUM>-<NUM> are performed prior to the display the screen depicted in <FIG>, the actions of block <NUM> are performed during the display of the screen depicted in <FIG>, the decision at block <NUM> is made during the display of the screen depicted in <FIG>, the actions of block <NUM> are performed prior to the display of the screen depicted in <FIG>, and the actions of block <NUM> are performed during the display of the screen depicted in <FIG>.

In various embodiments, first region <NUM> is useable to display visual representation <NUM> discussed herein in reference to <FIG> or updated visual representation <NUM> discussed herein in reference to <FIG> and <FIG>. In various embodiments, each icon <NUM> of visual representation <NUM> (or updated visual representation <NUM>) represents one of more datapoint in the dimensionally-reduced dataset. Further, in such embodiments each icon <NUM> represents one or more of the plurality of images <NUM> and is selectable to cause the represented images <NUM> to be displayed in second region <NUM>.

In various embodiments, second region <NUM> is useable to display one or more images <NUM>. In various embodiments, the images <NUM> displayed in second region <NUM> are displayed in response to use selection of portions of first region <NUM> (e.g., one or more icons <NUM> causing images <NUM> represented by the icons <NUM> to be displayed), portions of third region <NUM> or updated third region <NUM> (e.g., a portion of a list corresponding to a particular cluster causing images <NUM> associated with that cluster to be displayed), and/or portions of fourth region <NUM> (e.g., a portion of a list corresponding to the user classification labels causing images <NUM> labeled with a particular user classification label to be displayed). Each image <NUM> displayed in second region <NUM> is displayed as an object <NUM> in various embodiments. As discussed herein in reference to <FIG>, each object is associated with metadata for the image <NUM> and is selectable. Selecting the image, for example, allows the user to apply a user classification label or to respond to a predicted classification label for the selected image <NUM> as discussed herein.

In various embodiments, third region <NUM> is useable to display a list of the clusters within the dataset. Similarly, updated third regions <NUM> (also referred to herein as a "fifth region") is useable to display a list of the predicted classification labels by which the remaining unlabeled images <NUM> are clustered. In either case, each entry of the list is selectable to cause the images <NUM> associated with that cluster to be displayed in second region <NUM> in various embodiments. In various embodiments, the lists displayed in third region <NUM> and updated third region <NUM> include respective indication of the number of images <NUM> associated with each cluster.

In various embodiments, fourth region <NUM> is useable to display a list of the user classification labels applied to images <NUM>. In some of such embodiments, each entry of the list is selectable to cause the images <NUM> labeled with the user classification label to be displayed in second region <NUM>. In various embodiments, the list displayed in fourth region <NUM> includes respective indication of the number of images <NUM> labeled with each user classification label.

Referring now to <FIG>, a first screen of GUI <NUM> is shown. Prior to the display of this first screen, images <NUM> have been received by computer system, first machine learning algorithm <NUM> derived features from the images <NUM>, the dimensionality of the dataset of the derived features have been reduced by dimensionality reduction algorithm <NUM>, and clusters have been determined by clustering algorithm <NUM>. A visual representation <NUM> of the dimensionally-reduced dataset is displayed in first region <NUM>. A number of images <NUM> are displayed in second region <NUM>, however because no user selection have been received, the images <NUM> displayed are not associated with a particular cluster (e.g., they may be display randomly, they may be displayed in chronological order of when they were captured, they may be displayed in alphabetical order by name). A list of the clusters is displayed in third region <NUM> with indications of the number of images <NUM> associated with each cluster. Finally, a list of the three user classification labels used in this instance is displayed in fourth region <NUM>. In the examples shown in <FIG>, the user classification labels are determined based on the number of cell nuclei present in each image <NUM>: 1N for images <NUM> including one nucleus, 2N for images <NUM> including two nuclei, and 3_4N for image <NUM> including three or more nuclei. As discussed herein, more than three user classification labels may be used, and the criteria for determining which label should be applied to a particular image <NUM> also varies in various instances.

Referring now to <FIG>, a user selection of one or more icons <NUM> in cluster 402b has been received. In response, images <NUM> associated with cluster 402b are displayed in second region <NUM> and the portion of the list in third region <NUM> associated with cluster 402b is highlighted.

Referring now to <FIG>, in various embodiments, user classification labels are received from the user via a menu <NUM> displayed in the GUI <NUM>. In various embodiments, menu <NUM> includes indications of each user classification label, which includes the various labels for training dataset <NUM> as well as additional labels such as a label to exclude one or more images <NUM> from training dataset <NUM>. In the example shown in <FIG>, menu <NUM> includes indications of the three user classification labels 1N, 2N, and 3_4N as well as commands to "Move to Unknown" and "Move to Exclude" to mark the selected images <NUM> accordingly. As shown in <FIG>, a number of images <NUM> are highlighted in second region <NUM>, and the user input to menu <NUM> will apply the user classification label or command to the highlighted images <NUM>.

Referring now to <FIG>, user input applying user classification labels to various images <NUM> has been received. As shown in fourth region <NUM>, <NUM> images have been labeled 1N, <NUM> images have been labeled 2N, and <NUM> images <NUM> have been labeled 3_4N. In the embodiment shown in <FIG>, the user can enter a command to predict classification labels be clicking on button <NUM>. In response to this command, predicted classification labels are assigned as discussed in block <NUM> of method <NUM>. Alternatively, a prediction of classification labels is made automatically after a threshold number of images <NUM> have been labeled.

Referring now to <FIG>, GUI <NUM> now includes updated visual representation <NUM> and updated third region <NUM>. As discussed herein, the remaining unlabeled images <NUM> have been assigned predicted classification labels and are clustered by predicted classification labels. Accordingly, visual representation <NUM> includes four clusters: one associated with each user classification label and one for images for which a classification label has not (or for whatever reason cannot) been determined. Updated third region <NUM> includes a list of the predicted classification labels Pred 1N, Pred 2N, Pred 3_4N and Unknown as well as indications of the number of images <NUM> in with each predicted label. In various embodiments, updated visual representation <NUM> also includes datapoint associated with the image <NUM> that have user classification labels. The icons <NUM> representing these labeled images <NUM> may be visually different from icons <NUM> representing unlabeled images including but not limited to being a different color (e.g., icons <NUM> representing labeled images <NUM> are a darker color than icons <NUM> representing unlabeled images <NUM> but are in the same color family such as dark green and light green or dark blue and light blue) or by being different shapes (e.g., circular icons <NUM> for unlabeled images and star-shaped icons for labeled images <NUM>). As with visual representation <NUM> discussed herein, the icons <NUM> of updated visual representation <NUM> a selected able to cause the display of the image <NUM> represented by the selected icon(s) <NUM>. In the screen shown in <FIG>, the Pred IN cluster is selected. In response to this selection, images <NUM> in the Pred 1N cluster are displayed in second region <NUM>.

Referring now to <FIG>, in various embodiments, the user responds to the predicted classification labels by commands that are received from the user via a menu <NUM> displayed in the GUI <NUM>. In various embodiments, the menu <NUM> allows a user to accept the predicted classification label as a user classification label by selecting the indication of the user classification label corresponding to the predicted classification label in menu <NUM> or to reject the predicted classification label by selecting a different indication in menu <NUM>. In various embodiments, menu <NUM> includes indications of each user classification label, which includes the various labels for training dataset <NUM> as well as additional labels such as a label to exclude one or more images <NUM> from training dataset <NUM>. In the example shown in <FIG>, menu <NUM> includes indications of the three user classification labels 1N, 2N, and 3_4N as well as commands to "Move to Unknown" and "Move to Exclude" to mark the selected images <NUM> accordingly. As shown in <FIG>, a number of images <NUM> are highlighted in second region <NUM>, and the user input to menu <NUM> will apply the user classification label or command to the highlighted images <NUM>.

Referring now to <FIG>, a flowchart illustrating an embodiment of a training dataset creation method <NUM> is shown. In various embodiments, the various actions associated with method <NUM> are performed with computer system <NUM>. At block <NUM>, computer system <NUM> receives a dataset of image-derived features for each of a plurality of images <NUM>, wherein the image-derived features are determined by using a first machine learning algorithm <NUM> to analyze the plurality of images <NUM>. At block <NUM>, computer system <NUM> generates a dimensionally-reduced dataset from the dataset of image-derived features using a dimensionality reduction algorithm <NUM>. At block <NUM>, computer system identifies a plurality of clusters of datapoints within the dimensionally-reduced dataset using a clustering algorithm <NUM>. At block <NUM>, computer system <NUM> generates a visual representation <NUM> of the datapoints as icons <NUM> grouped by cluster <NUM>. Each icon <NUM> represents one or more particular datapoints and is selectable to cause the display of the one or more images <NUM> associated with the one or more particular datapoints. At block <NUM>, computer system <NUM> receives a selection of one or more of the icons <NUM>. At block <NUM>, computer system <NUM> causes the display of the images <NUM> associated with the one or more particular datapoints represented by the one or more selected icons <NUM>. At block <NUM>, computer system <NUM> receives a user classification label for at least one of the displayed images. At block <NUM>, computer system <NUM> predicts classification label(s) for unlabeled images <NUM> and receives a user response to the predicted classification label(s).

Referring now to <FIG>, a flowchart illustrating an embodiment of a training dataset creation method <NUM> is shown. In various embodiments, the various actions associated with method <NUM> are performed with computer system <NUM>. At block <NUM>, computer system <NUM> causes a user interface (e.g., GUI <NUM>) for preparing a labeled training dataset <NUM> for training a target machine learning algorithm to classify images to be displayed on a user device. The user interface includes a first region <NUM> including a two-dimensional visual representation <NUM> of a dimensionally-reduced dataset of image data that was derived from a plurality of images <NUM>. The two-dimensional visual representation <NUM> includes a plurality of icons <NUM> and indications of clusters <NUM> within the dataset. The user interface also includes a second region <NUM> that includes include one or more of the plurality of images <NUM>. At block <NUM>, computer system <NUM> receives user input applying user classification labels to one or more of the images <NUM> displayed in the second region <NUM>.

Turning now to <FIG>, a block diagram of an exemplary computer system <NUM>, which may implement the various components of computer system <NUM> is depicted. Computer system <NUM> includes a processor subsystem <NUM> that is coupled to a system memory <NUM> and I/O interfaces(s) <NUM> via an interconnect <NUM> (e.g., a system bus). I/O interface(s) <NUM> is coupled to one or more I/O devices <NUM>. Computer system <NUM> may be any of various types of devices, including, but not limited to, a server system, personal computer system, desktop computer, laptop or notebook computer, mainframe computer system, tablet computer, handheld computer, workstation, network computer, a consumer device such as a mobile phone, music player, or personal data assistant (PDA). Although a single computer system <NUM> is shown in <FIG> for convenience, system <NUM> may also be implemented as two or more computer systems operating together.

Processor subsystem <NUM> may include one or more processors or processing units. In various embodiments of computer system <NUM>, multiple instances of processor subsystem <NUM> may be coupled to interconnect <NUM>. In various embodiments, processor subsystem <NUM> (or each processor unit within <NUM>) may contain a cache or other form of on-board memory.

System memory <NUM> is usable to store program instructions executable by processor subsystem <NUM> to cause system <NUM> perform various operations described herein. System memory <NUM> may be implemented using different physical memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM-SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read only memory (PROM, EEPROM, etc.), and so on. Memory in computer system <NUM> is not limited to primary storage such as memory <NUM>. Rather, computer system <NUM> may also include other forms of storage such as cache memory in processor subsystem <NUM> and secondary storage on I/O Devices <NUM> (e.g., a hard drive, storage array, etc.). In some embodiments, these other forms of storage may also store program instructions executable by processor subsystem <NUM>.

I/O interfaces <NUM> may be any of various types of interfaces configured to couple to and communicate with other devices, according to various embodiments. In one embodiment, I/O interface <NUM> is a bridge chip (e.g., Southbridge) from a front-side to one or more back-side buses. I/O interfaces <NUM> may be coupled to one or more I/O devices <NUM> via one or more corresponding buses or other interfaces. Examples of I/O devices <NUM> include storage devices (hard drive, optical drive, removable flash drive, storage array, SAN, or their associated controller), network interface devices (e.g., to a local or wide-area network), or other devices (e.g., graphics, user interface devices, etc.). In one embodiment, computer system <NUM> is coupled to a network via a network interface device <NUM> (e.g., configured to communicate over WiFi, Bluetooth, Ethernet, etc.).

Claim 1:
A method for determining a user classification label for images, comprising:
(a) receiving, with a computer system (<NUM>), a dataset of image-derived features for each of a plurality of images (<NUM>), the plurality of images including images of cells, wherein the image-derived features are features that have been derived through analysis of the plurality of images (<NUM>) using a first machine learning algorithm (<NUM>);
(b)reducing, through execution of a dimensionality reduction algorithm (<NUM>) by the computer system (<NUM>) , the dimensionality of the multi-dimensional dataset output of the first machine learning algorithm (<NUM>) to a dimensionally-reduced dataset by reducing the number of random variables under consideration by obtaining a set of principal variables;
(c) identifying, with the computer system (<NUM>) using a clustering algorithm (<NUM>), a plurality of clusters (<NUM>) of datapoints within the dimensionally-reduced dataset;
(d) generating, with the computer system (<NUM>), a visual representation of the datapoints as icons grouped by cluster, wherein each icon represents one or more particular datapoints and is selectable to cause the display of the one or more images associated with the one or more particular datapoints;
(e) receiving, at the computer system (<NUM>), a selection of one or more of the icons;
(f) in response to the selection, causing, with the computer system (<NUM>), a display of the images associated with the one or more particular datapoints represented by the one or more selected icons; and
(g) receiving, at the computer system (<NUM>), a user classification label for at least one of the displayed images, characterised in that
wherein the dimensionality reduction algorithm (<NUM>) takes input from a second machine learning algorithm (<NUM>),
wherein the second machine learning algorithm (<NUM>) outputs predicted classification labels for unlabeled images based on the user classification labels.