Patent Description:
Network, software, and user interface architectures can be used to display images. However, significant inadequacies exist in current architectures, especially with regard to large image files. Raw data may be obtained and stored in a database. The raw data may be obtained from a scientific experiment, an industrial process, or from some other type of sensors. The data may be disorganized or unclear to a user and/or a computer (e.g., in a machine learning environment). The data may be associated with various data objects, and the data objects may include various properties associated with the object. However, this data may be unconnected to other relevant data objects.

<CIT> discloses a method for implementing aspects of a digital mapping system. The method includes sending a location request from a client-side computing device to a map tile server, receiving a set of map tiles in response to the location request, assembling said received map tiles into a tile grid, aligning the tile grid relative to a clipping shape, and displaying the result as a map image. The mapping system may further include direction control or zoom control objects as interactive overlays on the displayed map image, and may also include route or location overlays on the map image.

<CIT> discloses an apparatus for implementation of a system for visualization and analysis of a complex system such as an aircraft composed of a plurality of elements. The apparatus may be caused to receive and process data for physical instance(s) of the complex system to identify a topical element. The apparatus may be caused to receive a digital 3D model of the complex system, generate a visual presentation of at least a portion of the digital 3D model, with the visual presentation depicting the topical element and one or more other elements of the plurality of elements. And the apparatus may be caused to apply a visual effect to the topical element depicted by the visual presentation to distinguish the topical element from the other element(s) depicted by the visual presentation.

<CIT> discloses a system and method for providing remotely accessible medical image data. The system and method allows for increased accuracy and semiquantitative or fully quantitative data from images by enabling the remote user to select regions of interest on a compressed image, and then conducting quantitative analysis on original images at a central location.

Reference will now be made in detail to example embodiments, the examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

Image data can be obtained by a sensor and stored in one or more data objects (e.g., image slices of a larger scan or mapping) that each contain a great deal of information that cannot be feasibly manipulated and/or extracted in real time due to the significant processing power that would be required. The data may come from sensors that receive medical imaging data (e.g., from an MRI, CT, or CAT scan), agricultural information (e.g., farm plot usage), and/or any other data that can be applied to a map or other spatial representation, such as a spatial map, for additional processing. Moreover, a need exists for improved user interfaces that allow one or more users (e.g., biologists, doctors, etc.) to provide further processing of the images (e.g., image slices). Such user interfaces would allow each user to supply additional information to the one or more images and for that information to be associated with the one or more images for use by other users. Thus, there is a need for user interfaces that allow a user to identify and/or isolate, for example, a concentration of features (e.g., cells, proteins, crops, etc.) within one or more of the images and to communicate that data automatically and/or remotely to a second user. Further, there is a need for a system that can adequately present a visual representation of an information-dense map that can quickly and accurately present just the right amount of information a user would find useful but also not be computationally overwhelming. Additionally, there is a need for a system that can allow such a visual representation to be manipulated and characterized by a plurality of users. Such problems are not restricted to scientific data or experimental results but may be found in any context where mapped information and images related thereto may be made.

As discussed herein, a novel user interface can allow for large image files to be condensed into chunks that allow portions of the image files to be visually represented in a real-time user interface, such as in a browser, while also presenting pertinent information a user will find helpful to identify and manipulate. A computer system can receive a plurality of images, such as medical images. Machine learning techniques can automatically identify and categorize certain features in the images. For example, machine learning may determine quantitative measures for each of a plurality of characteristics of interest in the images. In the medical image context, machine learning may be used to identify different cells of interest, such as based on protein content within the cells. In some implementations, the presence of different proteins may be part of the input image data because, for example, each image data may reflect a different protein.

To provide a framework for the following discussion of specific systems and methods described herein, an example graphical user interface <NUM> will now be described with reference to <FIG>. This description is provided for the purpose of providing an example and is not intended to limit the techniques to the example system architecture.

<FIG> shows an example graphical user interface <NUM> that can be displayed in a browser. While a "browser" is discussed herein with reference to many example implementations, other applications could be used in place of a browser, such as a proprietary image viewing application or other standalone application. The graphical user interface <NUM> can display an image in an image viewer interface <NUM>. The image viewer interface <NUM> can include an image map <NUM> and/or a current view indicator <NUM> that identifies a portion of the image that is currently depicted in the image viewer interface <NUM>. The image may be a high-quality and/or ultra-large data file or a portion thereof.

The example image shown in <FIG> represents data obtained from an immunofluorescence (IF) stain of a slice of tissue. Images obtained from other sensors are possible (e.g. chromatography, CT scan, CAT scan, X-ray, MRI, ultrasound, endoscopy, etc.). As is discussed in more detail below, the displayed image may be a portion of one or more tiles of an image. This portion may include a downsampling of image features corresponding to a currently selected zoom level for the image portion. Thus, the features that are loaded and displayed in the image viewer interface <NUM> are at a downsampling level that is optimized for transmission and display of relevant image data at the currently selected zoom level.

In some embodiments, the graphical user interface <NUM> can allow a user to navigate to the displayed image via a URL <NUM>. Additionally or alternatively, the graphical user interface <NUM> can include one or more browser tabs <NUM> for allowing a user to access a plurality of browser elements.

The image viewer interface <NUM> can include one or more image interface selectors <NUM>. As shown, the image interface selectors <NUM> include zoom selectors (zoom in, zoom out), pan selectors (left, right, up, down), rotation selectors (clockwise, counterclockwise), a pen annotation selector, a shape annotation selector, a box selector, a share selector, and a trash selector. The image interface selectors <NUM> can allow a user to annotate the image. The annotations can be saved with the image file. Thus, once a user makes annotations and saves those annotations, the annotations may be viewable by the user later and/or by a second user who views the same image file.

<FIG> shows an example graphical user interface <NUM> where an example image portion annotation <NUM> has been made. As noted above, the image portion annotation <NUM> can be created by a user using the one or more image interface selectors <NUM> discussed above. The image portion annotation <NUM>, as shown, represents a false coloring of a region of the image. The image portion annotation <NUM> may be a user annotation representing a tumor region of the displayed tissue portion. This image portion annotation <NUM> can be saved for viewing later or by another user.

The graphical user interface <NUM> of <FIG> includes an image layer interface <NUM>. The image layer interface <NUM> may be viewable by a user through a selection using the graphical user interface <NUM>. For example, the user may scroll up or down in the browser. The image layer interface <NUM> can be selected by a selection of the "Image Layers" tab near the top of the graphical user interface <NUM>. The image layer interface <NUM> displays various layer indicators <NUM> that indicate details of what is being shown in the image viewer interface <NUM>. As shown, the layer indicators <NUM> signify that a tumor region is displayed in red (represented as larger circles in the image viewer interface <NUM>) as an annotation. Other layers are already embedded in the image (e.g., from raw image data with false coloring applied, e.g., from machine learning algorithms) as different colors-blue, green, and yellow (represented using variously sized shapes in the image viewer interface <NUM>). Thus, in some embodiments, raw images that are black-and-white (or grayscale) may be modified to include such false coloring. These other layers represent features of interest that can be indicated by the different colors corresponding to each respective layer, which are represented using variously sized shapes. As shown in <FIG>, false coloring from multiple images can be blended or mixed into a single image to form a composite image. In this particular example, the features of interest correspond to proteins identified manually based on visual inspection of raw images performed on results from immunofluorescence. In some embodiments, the features of interest may be made manually from a visual inspection of output from a machine learning algorithm, manually in response to quantitative analysis, automatically through one or more machine learning algorithms, or in some other way. Thus, the various colors present an easily digestible image to a user that allow the user to identify patterns or other worthwhile aspects of the image that demand further attention or investigation. The Layers <NUM>-<NUM> have suggested to an expert user that the image includes a tumor region, which the user has approximately annotated as indicated by the image portion annotation <NUM>. A user can add or remove various layers (including the annotated tumor region) using one or more corresponding image layer selectors <NUM>.

The image layer interface <NUM> shown in <FIG> also includes a layer editing interface <NUM>. A user can select a layer and adjust various attributes of the layer. For example, as shown the user can use a layer data indicator <NUM> to view a histogram of a pixel count within the image. The layer data indicator <NUM> displays a pixel count of pixels having a particular intensity. This information can provide a user with a graphical representation of the distribution of the layer. A user may be able to select the layer data indicator <NUM> to increase or decrease the pixel count proportionally across the image. The user can use layer data selectors <NUM> to adjust an opacity, contrast/brightness, and/or other display parameters of the layer that impact how the features of interest in layers of the composite image are blended together. Other layer data selectors <NUM> are possible in various embodiments.

As noted above, machine learning and/or segmentation algorithms may be used to analyze image data obtained from medical imaging (e.g., CT scans, etc.) to identify cells of interest, and to generate visual data representations of the identified cells of interest that may be overlaid onto the original image. The images may be viewed in relation to visual data representations (e.g., charts, graphs, etc.), such as those illustrated in <FIG> (discussed below). A user can define cohorts of cells based on attributes of the cells that the machine learning segmentation has determined. This cohort information may be obtained from a manual and/or automatic inspection of the image data. For example, a user may color cells by cohort, view only cells within a cohort, and/or make further edits described herein.

A user may identify and annotate (e.g., with a marker and/or written comment) a clustering of certain cell types at a certain location within the image, while the image is displayed at a first zoom level. The user may then adjust the zoom level and/or other visual setting, and may then identify, for example, a relationship between the clustering of the cell types and a location of a portion of relevant anatomy that wasn't readily apparent in the first zoom level. In some embodiments, the segmentation outputs parameters (e.g., minimum density, minimum distance of certain features from other features, maximum concentration of a certain feature, etc.) that may be used to help identify various characteristic of cells (e.g., that the features represent cancer cells or are a particular phenotype). In some embodiments, the user interface allows the user to set and/or adjust one or more threshold associated with these parameters. For example, a threshold parameter may include a range of values, such as a range that may be selected by drawing a box around a graph or chart of displayed feature values.

The adjustments added and/or set by the user can be connected to the image for review and/or further adjustment by a different user. Additionally, combinations of characteristics may be developed by the user to generate further characteristics of images.

<FIG> shows another aspect of a graphical user interface <NUM> that includes a feature analysis interface <NUM>. The feature analysis interface <NUM> allows a user to set one or more threshold parameters (sometimes referred to simply as "thresholds") for filtering cell parameters. The threshold parameters may include one or more of an upper threshold value, a lower threshold value, a feature type, and/or any other threshold parameter and/or filter. The threshold values may determine which features of interest are displayed based on a distribution of output values from the one or more segmentation algorithms associated with features of interest, such as an amount of protein in a cell, a distance between a cell and a nearest cell. The feature analysis interface <NUM> can allow a user to view the distribution of the output values associated with a selection of one or more cells (e.g., all of the cells in an image, a selected subgroup of cells, all of the cells in a selected region, etc.). The feature analysis interface <NUM> allows for additional analysis.

In one particular implementation, for example, a segmentation algorithm outputs quantitative properties (e.g., columns) of numbers that describe each cell. For example, a numeric representation of the amount of protein in that cell, or things like how close the cell is to its neighbors. Then, for each quantitative property, the user can view the distribution of values within the selected cells (e.g., all the cells in the image, or cells within a selected subregion in the image). Based on the distribution, the user can define one or more thresholds, such as an upper threshold value and/or a lower threshold value.

The feature analysis interface <NUM> includes data source selectors <NUM> that indicate which sources provide the image data for the images. As shown, the data source selectors <NUM> indicate that the image comes from selected Region <NUM> of Experiment <NUM> and Region <NUM> of Experiment <NUM>. The respective file paths are provided by corresponding data source path indicators <NUM>.

The feature analysis interface <NUM> also includes a feature type analysis indicator <NUM> and a plurality of feature threshold indicators <NUM>. The feature analysis interface <NUM> allows a user to input a feature type using the feature type selector <NUM>. The feature type selected in <FIG> is a Protein Type <NUM>. The feature type may be a result of a machine learning algorithm or some other algorithm that takes in image data as an input and outputs quantitative values. The feature type analysis indicator <NUM> of <FIG> shows a graphical representation of the number of cells having a fluorescence intensity corresponding to the Protein Type. Other feature types can be used, depending on which features of interest are included in the image. For example, in some embodiments, the features of interest may correspond to medical features (e.g., bone densities, Alzheimer protein formations, muscle or tendon tears), agricultural features (e.g., crops in farm plots, domesticated animals in a series of fields), or other features found in an image.

The user can use the feature threshold selector <NUM> to select a subgroup of the features indicated in the feature type analysis indicator <NUM>. The feature threshold selector <NUM> allows selection of a portion of the features by dragging across the histogram to selection a portion of the cells (e.g., as indicated with a rectangular outline in <FIG>). In other embodiments, the selector may be a digital dial, a keyboard field, or a clickable selector (e.g., checkboxes, bars, radio buttons, etc.). The feature threshold selector <NUM> allows a user to fine-tune what features of interest the graphical user interface <NUM> displays in the image. A plurality of feature threshold selector <NUM> can be used one on or more feature thresholds using the feature type selector <NUM>. As shown, a user can apply the threshold parameter by clicking a "Set Threshold" button. Other configurations are possible. The feature threshold indicators <NUM> show which thresholds have been applied to which proteins based on the user's selections using the feature threshold selector <NUM>. In some embodiments, the system can automatically supply one or more threshold parameters using a machine learning algorithm. For example, a group of tumor tissue may be identified by supplying the image to a trained model configured to identify tumor tissue. The system may then supply one or more of the threshold parameters and/or display them among the feature threshold indicators <NUM>.

<FIG> shows another view of the feature analysis interface <NUM> where the feature type selector <NUM> indicates that Protein Type <NUM> has been selected as the feature type. Once again, a user can select a subgroup of the feature type using the feature threshold selector <NUM>. The feature analysis interface <NUM> further allows a user to select other threshold parameters or filters using one or more feature group identifier selectors <NUM>. For example, a user can define one or more phenotypes based on a selection of threshold parameters to be included or excluded. The user can select a color to be used for representing the phenotypes. The user may name a subgroup of features as a new phenotype using one of the feature group identifier selectors <NUM>. As shown, the user has previously identified phenotypes called "Cell Type <NUM>" and "Tumor. " The user may be able to identify a characteristic of the features of interest using the feature analysis interface <NUM>. A characteristic may generally include specific commonalities among a group (e.g., subgroup) of features and may include, for example, a protein type, a feature name, a minimum distance from a target cell or protein, etc. These features may be determined by a machine learning algorithm. In some contexts, a "subgroup" or "phenotype" may describe a group of features having a common characteristic. The characteristic can be identified by a visual indicator that identifies each feature of the subgroup. The visual indicator of the subgroup of features can include a coloring, a highlighting, a shading, and/or an outlining associated with the subgroup of features. Other configurations are possible. Thus, while threshold features are numerical, a phenotype combines the numbers in such a way that a scientific interpretation and/or explanation may be attached.

A user can select a portion or region of the image tile at the given zoom level. The selected portion can visually indicate which features meet the one or more threshold parameters. For example, if a user selects a rectangular area of the image tile, the system may display red dots for areas with features (e.g., cells) having a first characteristic (e.g., cells of a first phenotype) and may display blue dots for areas with cells having a second qualifying characteristic (e.g., cells of a second phenotype). When the user selects another region, that other region is updated with the same types of thresholding areas using the corresponding colors. As described in further detail below, various alternative implementations of the present disclosure may include additional or fewer characteristics from those described above.

<FIG> and <FIG> show a user selection of a portion of an image within the image viewer interface <NUM>. The image viewer interface <NUM> can allow a user to overlay threshold areas on an image. Within the image viewer interface <NUM>, a false coloring is shown by image portion annotation <NUM>. A user had previously selected and annotated the image portion annotation <NUM>. As shown in <FIG>, a user can also select a subregion of the image as indicated by the image subregion selector <NUM>. The user can apply the subgroup thresholds and other display aspects (e.g., coloring, sizing, opacity, etc.). <FIG> shows the same image viewer interface <NUM> after the selected subregion has had the selections applied to the subregion. The resulting subregion features display <NUM> shows a plurality of features having the display characteristics indicated above with reference to <FIG>. For example, Phenotypes <NUM>, <NUM>, and <NUM> may be displayed as different colors (represented as different hatchings). The subregion features display <NUM> is described in more detail with respect to <FIG> below.

<FIG> shows a zoomed in view of the subregion features display <NUM> of <FIG>. The graphical user interface <NUM> shown also includes a feature display overlay editor interface <NUM>, which corresponds to a "Segmentation Overlay" tab in <FIG>. The feature display overlay editor interface <NUM> allows a user to further modify the display of the displayed features. Overlay selectors <NUM> allow a user to adjust the fill opacity, the outline opacity, and the radius of the outline. In some embodiments, additional or alternative overlay selectors <NUM> may be used, such as an outline type, a highlight, and/or an animation in response to a user selection (e.g., a mouse over). Other options are possible. As shown, the image viewer interface <NUM> shows a plurality of Tumor cell indicators <NUM> and Type <NUM> cell indicators <NUM>, each of which correspond to a coloring indicated by a feature display overlay legend interface <NUM>. The feature display overlay legend interface <NUM> shows which features (e.g., proteins, cells, phenotypes, etc.) correspond to which color aspects. A user can select a particular phenotype using the feature display overlay legend interface <NUM> and then can modify the display of the corresponding phenotype using the overlay selectors <NUM>.

Users may customize the look and feel of images displayed to the user, such as by modifying (e.g., tinting) an image and/or customizing artificial colors added to an image to show levels of a particular characteristic that are within a desired level. These user experience (e.g., look and feel) settings may be saved with the image and/or a larger investigation that includes the image, and used later by the user and/or other users that access the image.

In some embodiments, artificial colors are applied to an image by artificially tinting the image with a color, thereby transforming the image (rather than adding to it). In some embodiments, false coloring provides more than just a nice look and feel in the various user interfaces. The false coloring may allow analysis of images that is not possible otherwise. For example, false colors are very important when viewing more than one image layer, and allow the user to make sense of the relationship between image layers. In some embodiments, false colors are blended together to form composite colors in composite images. For example, a red tinted image layer composited with a blue tinted image layer will result in purple coloring of areas where both images have intensity.

Areas of images that are tagged and/or otherwise annotated by a user may also be save and shared with other users, such as in a cloud environment that allows real-time updates to other users that may be viewing the same image or investigation. Additionally, criteria for identifying qualifying characteristics of an image may be shared and used by other users. The tagged/annotated images can be saved and viewed by others via a browser.

<FIG> shows how the image annotations, thresholds, and other settings can be saved and viewed separately later by the same user and/or by a separate user. The information that a user supplies using one or more of the interfaces described above can be linked to the underlying image file for further modification at a separate time or user interface. As noted above, one of the many benefits of the technology described herein is that in some embodiments these annotations can be stored and viewed within a browser. Thus, one or more users can view the annotations from different consoles (e.g., user interfaces) and/or at different times. In some embodiments, a plurality of users may view and/or modify the annotations simultaneously.

As shown, the graphical user interface <NUM> is a browser that includes a workflow analysis interface <NUM> for viewing and manipulating aspects of workflow <NUM>. The workflow <NUM> shown includes a plurality of dataset blocks <NUM> representing respective datasets from which the image tiles are derived. Image tiling is described in more detail below. The dataset blocks <NUM> are attached to the image tiles block <NUM>, which indicates that the data flows between the Datasets <NUM> and <NUM> and the Image Tiles. Moreover, a separate machine learning (ML) analysis is performed on the image tiles. In some embodiments, the ML analysis is performed directly on one or more underlying images before the images are tiled. In this example, the machine learning analysis block <NUM> is connected to the image tiles block <NUM>, which indicates that the ML Analysis is performed on the Image Tiles. The ML Analysis can include machine learning features, such as those discussed above. For example, the Image Tiles may be analyzed by a trained model that is configured to identify features of interest within the Image Tiles. For example, the model may be trained to identify proteins, cells, cell types, phenotypes, interfaces, and/or other features. The model may be used to identify features in images related to agriculture, medicine, and/or other fields.

The workflow <NUM> can further include one or more feature analysis blocks <NUM> that represent corresponding one or more analyses that a user has provided, such as those discussed above. For example, the feature analysis can include a user's setting of a combination of one or more threshold parameters and/or filters that are associated with all of the image tiles of the image tiles block <NUM>. These settings of threshold parameters and/or other filters can be viewable by one or more users. For example, a user could select one of the feature analysis blocks <NUM> to view the respective feature analysis and/or modify the feature analysis (e.g., by modifying the threshold parameters, adding/subtracting threshold parameters).

The workflow <NUM> includes a tile block <NUM> that represents a tile of the image tiles associated with the image tiles block <NUM>. A user can select the tile block <NUM> to view the associated tile (Tile <NUM> as shown). The tile can include the associated annotations, including possible associated filters and/or threshold parameters associated specifically with that tile. The user can modify, add, and/or subtract annotations and/or threshold parameters associated with the tile. These changes can be saved and made viewable by a later different viewer (or the same viewer). In some embodiments, a selection of the tile block <NUM> will allow the user to view only the image itself but not the annotations.

The workflow <NUM> includes a display overlay block <NUM> that is also selectable by a user. The display overlay block <NUM> in this example is a heat map summarizing the analysis that has been performed. The display overlay block <NUM> represents different samples across the x-axis and different proteins across the y-axis. The display overlay block <NUM> shows a concentration of certain proteins in a particular sample. The display overlay block <NUM> can additionally or alternatively include user-selected overlays for viewing later. An example of such a user-selected display overlay is described above with respect to <FIG>. A user of can select the display overlay block <NUM> and view at a glance the concentration of proteins in various samples and/or view the analysis previously done. The user may then modify, add, or subtract aspects in the display overlay. In this way, in some embodiments, users can collaboratively analyze an image and refine that analysis through a distributed network.

The workflow analysis interface <NUM> can further include one or more workflow adjustment selectors <NUM>. The workflow adjustment selectors allow a user to select the one or more datasets to view (that may be represented, for example, by the dataset blocks <NUM>). The user can use the workflow adjustment selectors <NUM> to identify particular features and/or sources of experimental data for analysis in the workflow <NUM>.

Each image can be preprocessed for efficient data transmission at each of multiple zoom levels. For example, each image may be associated with multiple downsampled versions of the image each with a different level of detail in the downsampled images, and each being subdivided into a different quantity of tiles that together may be rendered to represent the entire image. The images corresponding to a highest zoom level (e.g., most zoomed in) may be associated with a highest quantity of tiles with a low degree of downsampling while images corresponding to a lowest zoom level (e.g., most zoomed out) will have been downsampled the most and may be associated with a lower quantity of tiles. Thus, in some embodiments, at the highest zoom level (e.g., most zoomed in), there will be a high quantity of tiles, and each tile will show a not-very-downsampled view of a small subset of the overall image, and at the lowest zoom level, there will be a small quantity of tiles, each tile will be very downsampled, and each tile will show a large section of the overall image.

When the image is viewed in a browser, for example, only those tiles for a currently rendered zoom level and portion of the image are transferred to the browser for rendering, allowing the browser to quickly render the image and also allowing the user to access all portions of the image at all zoom levels as desired. The processing required by the system is substantially lower since the level of detail that will be visually rendered corresponds only or primarily to the tile at the selected zoom level. Thus, the details (e.g., artificial coloring areas of the image at one or more zoom levels, features having a characteristic within a given and/or selected range, features satisfying some other threshold parameter) can be displayed on the fly (e.g., in real time). Additionally, the viewing user interface (e.g., a browser) can display images at various zoom levels through selective access to tiles within the particular zoom level(s) requested by the user.

<FIG> and <FIG> show example images <NUM> that have been tiled at two respective zoom levels. As shown in <FIG>, the image <NUM> has been tiled at a first zoom level corresponding to a plurality of first tiles <NUM> (shown as a 3x3 grid of tiles). In <FIG>, the image <NUM> has been tiled at a second zoom level corresponding to a plurality of second tiles <NUM> (shown as a 2x2 grid of tiles). In one implementation, the tiles <NUM> each include a higher resolution of image data than the tiles <NUM>, while tiles <NUM> each include a larger area of the overall image than tiles <NUM>. Thus, at a lower zoom level (e.g., most zoomed out), the tiles <NUM> may be loaded (e.g., transferred from a network server for rendering on a user device), and then when the zoom level is increased (e.g., zooming in), one or more of the tiles <NUM> may be loaded (e.g., transferred from the network server for rendering on a user device). In this way, not all of the tiles at all zoom levels are necessary to view various portions of the image at various zoom levels, minimizing bandwidth and processing requirements of the server and user computing systems.

The tilings shown in <FIG> and <FIG> are only examples. For example, in one implementation at the outer most zoom level, the entire image may be rendered in a single tile, and each zoom level doubles in both dimensions, so a single tile is replaced by <NUM> tiles when zooming in. Various systems may be configured with different quantities of zoom levels, such as <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> or more. An image may have additional or fewer zoom levels and/or with each zoom level having different numbers of tiles. For example, an image can have a tiling corresponding to a grid of 1x8, 1x6, 1x5, 2x3, 3x4, 3x2, 4x3, 5x4, 4x4, 5x5, 6x6, 7x7, 8x8, 9x9, 10x10, 11x11, 12x12, 13x13, 14x14, and/or other arrangement of tiles. The grid can have the same number of tiles vertically as horizontally along the image. In some embodiments, the number of tiles vertically is different from the number tiles horizontally. As shown, the first zoom level includes nine first tiles <NUM>, and the second zoom level includes four second tiles <NUM>. As shown, each zoom level tiles the whole image <NUM> and each zoom level includes tiles that are nonoverlapping. In some configurations, the tiles at a particular zoom level may overlap. In some implementations, the tiles include grayscale data and colorization of features (e.g., that are identified by one or more machine learning algorithm) may be associated with a false coloring scheme to add coloring to the tiles.

The system can create tiles on a server and transfer of the size-optimized tiles for display in the interactive user interface (e.g., the graphical user interface <NUM>). This feature allows the system to minimize bandwidth requirements for displaying an information-dense image, which can allow for more rapid or even real-time image display. This also allows the image data to be accessed for only a portion of the image that is currently selected for display and at the current zoom level selected for display. For example, in some embodiments, the graphical user interface automatically tiles the image data all at once when an image is received at the server. Additionally or alternatively, the graphical user interface can tile the image data "on-demand" (e.g., at different zoom levels as requested by the user).

To create tiles, the system can access an image and subdivide the image into one or more groups of tiles at corresponding zoom levels. The tiles at the same zoom level may be non-overlapping. For example, system can subdivide the image into a first and second groups of non-overlapping tiles at respective first and second zoom levels. Additional or fewer zoom levels and/or groups of tiles are possible.

In some embodiments, each zoom level can be associated with a corresponding level of downsampling. The level of downsampling refers generally to the level of detail of features shown in that zoom level. As a zoom level increases (e.g., going to tiles that cover a smaller portion of the complete image), the density of detail (e.g., the number of features per unit area of the complete image) increases. Thus, a tile displaying a larger portion of the original image (zoomed out) will display a smaller feature density than a tile displaying a smaller portion of the original image (zoomed in). In this way, the system can display a suitable number of image elements rapidly without either waiting for long load times to see more detail than desired. Each image may include an enormous number of features and allowing a user to rapidly and in real time view the features allows time-sensitive information to be analyzed quickly and accurately. Moreover, this triaging of the features can allow the images to be viewed in a browser, thus allowing users to view the data from outside the internal network and/or allowing a plurality of users to view the images. A user can zoom in or zoom out on the image (e.g., using the image interface selectors <NUM> described above, with reference for example to <FIG>), which can display one or more tiles of the image having a different zoom level and/or a different downsampling level. An image may be tiled to allow as many as <NUM> zoom levels or more.

A graphical user interface (e.g., the graphical user interface <NUM>) can receive one or more tiles of the image from a particular zoom and/or downsampling level. The system can receive a user input representing an analysis of visual elements displayed within the one or more tiles. For example, the analysis may include setting one or more threshold parameters on various aspects of the features displayed within the one or more tiles. The threshold parameters may include, for example, setting one or more of an upper threshold value, a lower threshold value, a feature type, and/or any other threshold parameter and/or filter.

The analysis performed on the one or more tiles can be saved and associated with the tiles when they are viewed at another time and/or place. In some embodiments, the analysis can be saved and associated with every set of tiles at all zoom levels. In some embodiments, the detail of the analysis may be downsampled at a corresponding level of downsampling for each set of tiles at each zoom level. The system can allow a user to select a new zoom level (e.g., zoom out, zoom in). The system can then update the display to show one or more tiles from a different group that correspond to the area zoomed in or zoomed out. Thus, the one or more tiles at the different group will display at least some overlap of the same portion of the image before zooming in or zooming out.

The downsampling of features may change with each level of zooming in or zooming out. As noted above, cells of interested (e.g., potential tumor cells) and other features may be identified as a result of machine learning applied to the original, non-downsampled, non-tiled image. Advantageously, these identified cells of interest, along with analysis and annotations performed on the cells at any zoom level (e.g., tiles of a composite image) carry over onto every other zoom level of the image.

In some embodiments, to display the tiles, the system accesses information indicating locations of a plurality of features of interest within the image. The system may determine one or more tile arrangements indicating corresponding quantities of tiles associated with respective zoom levels. Each of the tile arrangements may include a portion of the image at a corresponding downsampling. In some embodiments, each set of tiles at a given downsampling and/or zoom level collectively represent the image. The system can determine a zoom level associated with display of a portion of the image and then determine which tiles associated with the portion of the image at the determined zoom level are to be displayed. Receipt of thresholds, analysis, annotations, etc. may be done at the determined zoom level. A user may choose to change the zoom level and/or provide additional thresholds, annotation, etc..

In an example implementation, one or more first tiles from the first zoom level may be displayed. If a user zooms out, then the system may display one or more second tiles at a second (zoomed out) zoom level. The portion of the image shown by the first tiles will have encompassed only a portion of the region of the image displayed by the second tiles at the second group (since the second tiles are at a zoomed out level). The downsampling of features is not necessarily the same and, in some cases, is different according to the level of zoom (e.g., the greater zoom level, then the greater level of detail and less downsampling). A user may provide an annotation and/or other analysis (e.g., thresholding) to the first tile. In some embodiments, the analysis done to the features displayed in the first tile tracks with the features displayed in the second tile once the user zooms out. For example, if tumor cells have been identified and/or visually annotated in the first tile, then the display of the second tile will include the identification and/or visual annotation of the tumor cells, though perhaps at a different level of detail due to a different downsampling between the two tiles. Similar functionality may apply for going from a zoomed out to a zoomed in tile.

<FIG> and <FIG> are flowcharts illustrating two embodiments of example methods that are implemented on a computer. Depending on the embodiment, the methods of <FIG> and <FIG> may include fewer or additional blocks and the blocks may be performed in an order that is different than illustrated.

<FIG> shows an example method <NUM> of tiling and displaying an image with associated features of interest (or simply "features"). Beginning at block <NUM>, a display system, for example, accesses an image. At block <NUM>, the system subdivides the image into a first group of non-overlapping tiles at a first zoom level, and at block <NUM> the system subdivides the image into a second group of non-overlapping tiles at a second zoom level. For example, the system may divide the tiles into different zoom levels having corresponding different downsampling levels, such as is described with respect to <FIG> above. However, as noted above, in some embodiments the subdividing of the image data (e.g., blocks <NUM> and <NUM>) may occur at a separate system, such as at a remote server. In some embodiments, the remote server may additionally or alternatively access the image at block <NUM>.

At block <NUM>, the system displays, using for example a graphical user interface, a tile of the image from the first group. At block <NUM>, the system receives a user input representing analysis of visual elements within the tile from the first group. For example, the analysis may include setting one or more threshold parameters on various aspects of the features displayed within the tile. The threshold parameters may include setting one or more of an upper threshold value, a lower threshold value, a feature type, and/or any other threshold parameter and/or filter. The system at block <NUM> updates the display of the tile based on the analysis of the visual elements. In some embodiments, the system can receive a new zoom level from the user. The system can then update the display to show a tile from the second group. This tile from the second group will display at least some of the same portion of the image as the tile from the first group. For example, if the tile from the second group is more zoomed in, then the portion of the image shown by the tile at the first group will have encompassed the region of the image displayed by the tile at the second group. Alternatively, if the tile from the second group is more zoomed out, then the portion of the image shown by the tile at the first group will have encompassed only a portion of the region of the image displayed by the tile at the second group. In neither scenario is the downsampling of features necessarily the same. In some embodiments, the analysis done to the features displayed in the tile of the first group tracks with the features displayed in the tile of the second group. For example, if tumor cells have been identified and/or visually annotated in the tile of the first group, the display of the tile of the second group will include the identification and/or visual annotation of the tumor cells (though perhaps at a different level of detail due to a different downsampling between the two tiles).

<FIG> shows a different method <NUM> that a system may implement for displaying images. As shown, at block <NUM>, the method <NUM> includes accessing an image (e.g., from a file). In some embodiments, the system accesses information indicating locations of a plurality of features of interest within the image, such as may have been determined using multiple machine learning algorithms applied to the image. At block <NUM>, the method <NUM> includes determining a first tile arrangement indicating a first quantity of tiles associated with a first zoom level. Each of the first tiles may include a portion of the image at a first downsampling such that the first tiles collectively represent the image. At block <NUM>, the system determines a second tile arrangement indicating a second quantity of tiles associated with a second zoom level. Each of the second tiles may include a portion of the image at a second downsampling such that the second tiles collectively represent the image. The system at block <NUM> determines a zoom level associated with display of a portion of the image.

At block <NUM> the system determines one or more tiles associated with the portion of the image at the determined zoom level and, at block <NUM>, generates the determined one or more tiles. In some embodiments, the system receives a selection of a threshold parameter associated with one or more of the features of interest. At block <NUM>, the method <NUM> includes updating the user interface to display features of interest satisfying one or more received threshold parameters (e.g., the user selection of the threshold parameter, an automatically received threshold parameter). In some embodiments, the received threshold parameters can include parameters received from a trained machine learning model.

In some configurations, the system receives the threshold parameter by receiving a minimum density, a minimum distance, a maximum number, or any combination thereof. The system may receive a user selection of a portion of the image and based on that selection, update the user interface to display results of user analytics applied to the selected portion.

In some configurations, the system, based on the selection of features and/or a subgroup of features, can identify a characteristic (e.g., cell type, protein type, feature name, minimum distance from target, etc.) relating to the selection and/or subgroup. The characteristic can be identified by a visual indicator indicating each feature of the subgroup. The visual indicator of the subgroup of features can include a coloring, a highlighting, a shading, and/or an outlining associated with the subgroup of features. Based on one or more of the selection of features, the subgroup of features, and/or the threshold parameter, the system can pass the respective selection of features, the subgroup of features, and/or the threshold parameter to a computer in communication with a second graphical user interface. In some embodiments, the second graphical user interface includes a browser. Passing the information to the second graphical user interface can allow another user to, for example, view the image with and/or without the annotations and/or modifications. In some embodiments, the method <NUM> includes receiving a modification (e.g., addition, subtraction, alteration) to the analysis.

The image may include one layer of a plurality of images derived from a medical scan. The method <NUM> can include using a trained model to automatically identifying the features of interest. The images may be obtained using one or more sensors.

Various embodiments of the present disclosure may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or mediums) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

For example, the functionality described herein may be performed as software instructions are executed by, and/or in response to software instructions being executed by, one or more hardware processors and/or any other suitable computing devices. The software instructions and/or other executable code may be read from a computer readable storage medium (or mediums).

The computer readable storage medium can be a tangible device that can retain and store data and/or instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device (including any volatile and/or non-volatile electronic storage devices), a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a solid state drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing.

Computer readable program instructions (as also referred to herein as, for example, "code," "instructions," "module," "application," "software application," and/or the like) for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and procedural programming languages, such as the "C" programming language or similar programming languages. Computer readable program instructions may be callable from other instructions or from itself, and/or may be invoked in response to detected events or interrupts. Computer readable program instructions configured for execution on computing devices may be provided on a computer readable storage medium, and/or as a digital download (and may be originally stored in a compressed or installable format that requires installation, decompression or decryption prior to execution) that may then be stored on a computer readable storage medium. Such computer readable program instructions may be stored, partially or fully, on a memory device (e.g., a computer readable storage medium) of the executing computing device, for execution by the computing device. The computer readable program instructions may execute entirely on a user's computer (e.g., the executing computing device), partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart(s) and/or block diagram(s) block or blocks.

The remote computer may load the instructions and/or modules into its dynamic memory and send the instructions over a telephone, cable, or optical line using a modem. A modem local to a server computing system may receive the data on the telephone/cable/optical line and use a converter device including the appropriate circuitry to place the data on a bus. The bus may carry the data to a memory, from which a processor may retrieve and execute the instructions. The instructions received by the memory may optionally be stored on a storage device (e.g., a solid state drive) either before or after execution by the computer processor.

In addition, certain blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states relating thereto can be performed in other sequences that are appropriate.

For example, any of the processes, methods, algorithms, elements, blocks, applications, or other functionality (or portions of functionality) described in the preceding sections may be embodied in, and/or fully or partially automated via, electronic hardware such application-specific processors (e.g., application-specific integrated circuits (ASICs)), programmable processors (e.g., field programmable gate arrays (FPGAs)), application-specific circuitry, and/or the like (any of which may also combine custom hard-wired logic, logic circuits, ASICs, FPGAs, etc. with custom programming/execution of software instructions to accomplish the techniques).

Any of the above-mentioned processors, and/or devices incorporating any of the above-mentioned processors, may be referred to herein as, for example, "computers," "computer devices," "computing devices," "hardware computing devices," "hardware processors," "processing units," and/or the like. Computing devices of the above-embodiments may generally (but not necessarily) be controlled and/or coordinated by operating system software, such as Mac OS, iOS, Android, Chrome OS, Windows OS (e.g., Windows XP, Windows Vista, Windows <NUM>, Windows <NUM>, Windows <NUM>, Windows Server, etc.), Windows CE, Unix, Linux, SunOS, Solaris, Blackberry OS, VxWorks, or other suitable operating systems. In other embodiments, the computing devices may be controlled by a proprietary operating system. Conventional operating systems control and schedule computer processes for execution, perform memory management, provide file system, networking, I/O services, and provide a user interface functionality, such as a graphical user interface ("GUI"), among other things.

For example, <FIG> is a block diagram that illustrates a computer system <NUM> upon which various embodiments may be implemented. For example, the computer system <NUM> may be implemented as the graphical user interface <NUM> (<FIG>) in some embodiments. Computer system <NUM> includes a bus <NUM> or other communication mechanism for communicating information, and a hardware processor, or multiple processors, <NUM> coupled with bus <NUM> for processing information. Hardware processor(s) <NUM> may be, for example, one or more general purpose microprocessors.

Computer system <NUM> also includes a main memory <NUM>, such as a random access memory (RAM), cache and/or other dynamic storage devices, coupled to bus <NUM> for storing information and instructions to be executed by processor <NUM>. Such instructions, when stored in storage media accessible to processor <NUM>, render computer system <NUM> into a special-purpose machine that is customized to perform the operations specified in the instructions.

A storage device <NUM>, such as a magnetic disk, optical disk, or USB thumb drive (Flash drive), etc., is provided and coupled to bus <NUM> for storing information and instructions.

Computer system <NUM> may be coupled via bus <NUM> to a display <NUM>, such as a cathode ray tube (CRT) or LCD display (or touch screen), for displaying information to a computer user. In some embodiments, the same direction information and command selections as cursor control may be implemented via receiving touches on a touch screen without a cursor.

Computing system <NUM> may include a user interface module to implement a GUI that may be stored in a mass storage device as computer executable program instructions that are executed by the computing device(s). Computer system <NUM> may further, as described below, implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system <NUM> to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system <NUM> in response to processor(s) <NUM> executing one or more sequences of one or more computer readable program instructions contained in main memory <NUM>. Execution of the sequences of instructions contained in main memory <NUM> causes processor(s) <NUM> to perform the process steps described herein.

Various forms of computer readable storage media may be involved in carrying one or more sequences of one or more computer readable program instructions to processor <NUM> for execution.

As another example, communication interface <NUM> may be a local area network (LAN) card to provide a data communication connection to a compatible LAN (or WAN component to communicated with a WAN).

As described above, in various embodiments certain functionality may be accessible by a user through a web-based viewer (such as a web browser), or other suitable software program). In such implementations, the user interface may be generated by a server computing system and transmitted to a web browser of the user (e.g., running on the user's computing system). Alternatively, data (e.g., user interface data) necessary for generating the user interface may be provided by the server computing system to the browser, where the user interface may be generated (e.g., the user interface data may be executed by a browser accessing a web service and may be configured to render the user interfaces based on the user interface data). The user may then interact with the user interface through the web-browser. User interfaces of certain implementations may be accessible through one or more dedicated software applications. In certain embodiments, one or more of the computing devices and/or systems of the disclosure may include mobile computing devices, and user interfaces may be accessible through such mobile computing devices (for example, smartphones and/or tablets).

Many variations and modifications may be made to the above-described embodiments, the elements of which are to be understood as being among other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure. The foregoing description details certain embodiments. It will be appreciated, however, that no matter how detailed the foregoing appears in text, the systems and methods can be practiced in many ways. As is also stated above, it should be noted that the use of particular terminology when describing certain features or aspects of the systems and methods should not be taken to imply that the terminology is being re-defined herein to be restricted to including any specific characteristics of the features or aspects of the systems and methods with which that terminology is associated.

Conditional language, such as, among others, "can," "could," "might," or "may," unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements, and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment.

The term "substantially" when used in conjunction with the term "real-time" forms a phrase that will be readily understood by a person of ordinary skill in the art. For example, it is readily understood that such language will include speeds in which no or little delay or waiting is discernible, or where such delay is sufficiently short so as not to be disruptive, irritating, or otherwise vexing to a user.

Conjunctive language such as the phrase "at least one of X, Y, and Z," or "at least one of X, Y, or Z," unless specifically stated otherwise, is to be understood with the context as used in general to convey that an item, term, etc. may be either X, Y, or Z, or a combination thereof. For example, the term "or" is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Thus, such conjunctive language is not generally intended to imply that certain embodiments require at least one of X, at least one of Y, and at least one of Z to each be present.

The term "a" as used herein should be given an inclusive rather than exclusive interpretation. For example, unless specifically noted, the term "a" should not be understood to mean "exactly one" or "one and only one"; instead, the term "a" means "one or more" or "at least one," whether used in the claims or elsewhere in the specification and regardless of uses of quantifiers such as "at least one," "one or more," or "a plurality" elsewhere in the claims or specification.

Claim 1:
A computer-implemented method (<NUM>) of preprocessing images (<NUM>) for efficient image data transmission to a user interface (<NUM>) configured to display the preprocessed image, the method comprising:
accessing (<NUM>) an image (<NUM>);
accessing information indicating locations of a plurality of features of interest within the image (<NUM>);
determining (<NUM>, <NUM>) a first tile arrangement indicating a first quantity of tiles associated with a first zoom level, each of the first tiles (<NUM>) including a portion of the image (<NUM>) at a first downsampling level corresponding to a first level of detail of the plurality of features of interest within the image (<NUM>) at the first zoom level such that the first tiles (<NUM>) collectively represent the image (<NUM>);
determining (<NUM>) one or more tiles associated with the portion of the image (<NUM>);
generating (<NUM>) the determined one or more tiles;
transmitting only the generated one or more tiles of the first tile arrangement to a user interface (<NUM>) for display (<NUM>);
displaying a feature type analysis indicator (<NUM>) comprising a histogram showing a graphical distribution of quantitative values associated with one or more of the plurality of features of interest;
receiving, via user selection of a feature type selector (<NUM>), a feature type associated with the one or more of the plurality of features of interest;
displaying, in response to the user selection of the feature type selector, an updated histogram showing the graphical distribution of quantitative values associated with one or more of the plurality of features of interest corresponding to the selected feature type;
receiving a selection of a portion of the updated histogram to fine-tune a threshold parameter associated with the one or more of the plurality of features of interest corresponding to the selected feature type (<NUM>); and
updating (<NUM>) the user interface (<NUM>) to display features of interest satisfying the fine-tuned threshold parameter within the displayed one or more tiles.