Recipe based method for time-lapse image analysis

A computerized recipe station for time-lapse image analysis method includes the steps of inputting an image sequence and an initial recipe to a computer storage; performing by a computer program an incremental apply using the image sequence and the initial recipe to generate an incremental output; pausing the incremental apply; using the incremental output to perform an incremental output assurance operation, which may be an intermediate result analysis to generate an analysis output, a recipe update to generate an updated recipe, or a result editing to generate an edited incremental output; and continuing the incremental apply until pausing or completion to generate a processing output. The analysis output generated by the intermediate result analysis may be used to guide the recipe update step or used to guide the result editing step.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to computerized time-lapse image analysis and more particularly to a recipe station framework to support intuitive workflow that starts with incremental recipe execution, continuous result monitoring with image guided data analysis/data guided image visualization, recipe fine-tuning, and mask and track editing.

2. Description of the Related Art

a. Description of Problem that Motivated Invention

The technology advancement has enabled the routine acquisition of movie (image sequences) from not only video cameras but also smart phones. Therefore, the demand for time-lapse (rather than fixed point) image analysis becomes more prevalent. In the bioscience field, the advent of time-lapse microscopy and live cell fluorescence probes has enabled biologists to visualize the inner working of living cells in their natural context. Expectations are high for breakthroughs in area such as cell response and motility modification by drugs, control of targeted sequence incorporation into the chromatin for cell therapy, spatial-temporal organization of the cells and its changes with time or under infection, assessment of pathogens routing into the cell, interaction between proteins, and sanitary control of pathogen evolution, etc. The breakthroughs could revolutionize the broad fields in basic research, drug discovery and disease diagnosis.

Deciphering the complex machinery of cell function and dysfunction necessitates a detailed understanding of the dynamics of proteins, organelles, and cell populations. Due to the complexity of the time-lapse image analysis tasks to cover the wide range of highly variable and intricate properties of biological material, it is difficult to have fully automated solutions except some dedicated high-volume applications such as cancer screening, wafer defect inspection. Most of the computerized image analysis applications require interactive confirmation, editing and data analysis by users.

After tackling the huge complexities involved in establishing a live cell imaging study, scientists are often frustrated by the difficulties of image quantification that requires either tedious manual operations or specialized image processing and programming skills to achieve the desired outcomes. It is highly desirable to have an intuitive, easy-to-use workflow for obtaining optimal time-lapse analysis outcomes and efficient result viewing and sharing without specialized image processing and programming knowledge.

b. How Did Prior Art Handle the Problem?

The prior art approach provides manual analysis tools or manual editing tools. However, the tools become impractical for time-lapse image analysis, as the data volume is high and the errors could accumulate over time. For example, in tracking applications of time-lapse image sequence, a wrong track assignment in an early time frame will propagate to the later time frames. This causes significant inefficiency for a user to review and correct the mistakes, as the same mistakes have to be repeatedly corrected.

Furthermore, for a meaningful spatial-temporal analysis, the time-lapse image sequence has to cover a long time duration which has high data volume that requires timely review and timely correction of analysis error or timely updates of the processing instructions (recipes) to achieve good outcome efficiently. The existing tools do not facilitate the above requirements.

Therefore, a more sophisticated computerized framework and method for time-lapse image analysis is urgently needed to address the deficiencies of the prior art methods.

BRIEF SUMMARY OF THE INVENTION

The primary objective of the invention is to provide an intuitive recipe station for a user to incrementally apply image analysis recipe and monitor the intermediate results of a time-lapse movie. The secondary objective of the invention is to allow a user to pause the time-lapse image analysis to perform intermediate result analysis. The third objective of the invention is to allow a user to optimize the time-lapse image analysis by intermediate recipe update. The fourth objective of the invention is to allow a user to optimize the time-lapse image analysis by intermediate result editing. The fifth objective of the invention is to allow the recording of processing history for reproducing the results or creating an adaptive recipe for volume data processing.

The current invention provides a recipe station framework to support intuitive workflow that starts with incremental recipe execution, continuous result monitoring with image guided data analysis/data guided image visualization, recipe fine-tuning, and mask and track editing.

The computerized recipe station for time-lapse image analysis method according to the present invention comprises the steps of:a) inputting an image sequence and an initial recipe to a computer storage;b) performing by a computer program an incremental apply using the image sequence and the initial recipe to generate an incremental output;c) pausing the incremental apply;d) using the incremental output to perform an incremental output assurance operation, which may be a group consisting of intermediate result analysis, recipe update, or result editing to generate an edited incremental output; ande) continuing the incremental apply until pausing or completion to generate a processing output.

Furthermore, the analysis output generated by the intermediate result analysis in step d) may be used to guide the recipe update step or used to guide the result editing step.

DETAILED DESCRIPTION OF THE INVENTION

I. Application Scenarios

FIG. 1shows the processing flow of the computerized recipe station for time-lapse image analysis method. An input image sequence100and an initial recipe102is loaded into a computer memory for incremental apply108performed by a computer program. The initial recipe102is stored in a computer recipe storage104that maintains the recipe106to be used. The incremental apply108step processes the input image sequence100using the recipe106to generate an incremental output110that incrementally stores the processing results of the image frames up to the current frame. The incremental output110can be viewed during the incremental apply108and the incremental apply108can be paused112. After pausing, the incremental output110up to the paused frame can be used for incremental output assurance. The incremental output assurance could be performed by intermediate result analysis114, recipe update118or result editing122. The intermediate result analysis114generates analysis output116. The recipe update118generates updated recipe120and the result editing122generates edited incremental output124. After the incremental output assurance operations, a continue apply126is invoked and the incremental apply is continued from the current frame or an earlier frame specified by a user. The processing continues until it is paused and the incremental output assurance process is repeated or until the processing is completed128for all time frames and processing output130is generated. A processing recording132step records the processing history and generates a processing profile134.

FIG. 2shows an alternative processing flow of the computerized recipe station for time-lapse image analysis method. In the processing flow, after pausing, the incremental output assurance is performed by the intermediate result analysis114that generates analysis output116. The analysis output116is used to guide the recipe update118step to generate updated recipe120. After the recipe is updated a continue apply126is invoked and the incremental apply108is continued from the current frame or a frame specified by a user.

FIG. 3shows another alternative processing flow of the computerized recipe station for time-lapse image analysis method. In the processing flow, after pausing, the incremental output assurance is performed by the intermediate result analysis114that generates analysis output116. The analysis output116is used to guide the result editing122step to generate edited incremental output124. After the edited incremental output124is generated, a continue apply126is invoked and the incremental apply108is continued from the current frame or a frame specified by a user.

II. Input Image Sequence

The input image sequence100can be acquired from any digitization methods such as a camera, a smart phone, a scanner, photomultipliers, image sensors, etc. The images can be acquired with different spectra and modalities such as bright field, dark field, X-ray, IR, ultrasound, lasers, etc. as time-lapse (X, Y, T) sequence. It could also include Z dimension (3D) and multiple spectra.

In one embodiment of the invention, microscopy image sequences are used as the input images. The microscopy images can be acquired from different microscopy modes such as Total internal reflection fluorescence microscopy (TIRF), bright-field, Phase contrast, Differential interference contrast (DIC) microscopy, FRAP, FLIM and FRET and also could be from 2D and 3D microscopy such as inverted, confocal and super-resolution microscopes.

III. Initial Recipe

A recipe contains instructions for computer image sequence processing for time-lapse image applications such as object tracking, object counting, lineage analysis, exocytosis analysis, colony analysis, etc. The recipe processing steps may contain combinations of operations selected from a group consisting of enhancement, segmentation, tracking, subset gating, decision, analysis and measurements, etc. In one embodiment of the invention, the initial recipe102could be generated using the method disclosed in the U.S. Pat. No. 7,849,024, to Lee et, al, “Imaging system for producing recipes using an integrated human-computer interface (HCl) for image recognition, and learning algorithms”.FIG. 4shows an embodiment of the initial recipe for fluorescent cell tracking. It includes an input, output specifications400,402and configuration selection buttons404,406and408.

Other embodiments may exclude the input/output specification through automatic routing. The configuration selection buttons may or may not be necessary.

IV. Incremental Apply, Pause and Continue Apply

The incremental apply108step allows the incremental execution of the recipe over consecutive time frames of the time-lapse image sequence. This can be performed by applying in step, one frame at a time, or by continuing applying to consecutive frames until being paused. After an incremental apply is paused, it could be continued from the frame it paused or continue from a user selected frame.

FIG. 5Ashows an embodiment of the control buttons for incremental apply108including apply from beginning500, continue502and step504.FIG. 5Bshows an embodiment of the pause button506during incremental apply108.FIG. 5Cshows an embodiment of a menu selection for continue apply from selected frame508. If recipe is updated, the continue apply126will apply using the updated recipe120from the recipe storage104.

Other embodiments of the incremental apply108include keyboard control rather than button control. It could also include fast forward/backward for review and/or shortcut keys.

During incremental apply108, an incremental output110is generated. The incremental output110is updated after each new frame is processed. The incremental output can be viewed to monitor the processing progress and quality.FIG. 6shows an embodiment of viewing of the incremental output110. The example recipe is for Florescence Cell Tracking600, the incremental apply108processing is on-going and the current frame is at the 46thframe602. The incremental output110can be viewed from the image display604showing detected cell masks and the trajectories for the tracked cells. In addition, the measurements up to the current frame can be shown in object data graph606shown in the chart window608. Note that the image display604and object data graph606are updated after each new frame is processed.

VI. Intermediate Result Analysis

After the incremental apply108is paused112, the intermediate result analysis114step using the incremental output110to generate analysis output116. In one embodiment, the intermediate result analysis114performs an analysis step selected from a group consisting of reviewing image display604; reviewing at least one object data graph606; and reviewing at least one object data sheet. This generates the analysis output116.

The image display604highlights the detected objects with object indicators such as object masks with different labels. The image display highlights the tracked objects with track indicators such as trajectory display of a selected number of frames. The trajectory display could be encoded so that it could fade with time and/or use different colors for time indication or for tracked object features (velocity, direction, etc.) indications. The at least one object data graph606may be selected from a group consisting of a feature histogram plot, an object histogram plot, a scatter plot and a stacked scatter plot, radial graph, mesh graph, surface graph, and volumetric graph, etc.

One embodiment of the object data sheet is shown inFIG. 7. The object data sheet can be opened by clicking on a “Spreadsheet”700button. In the data sheet, the object measurements702are displayed and the data can be sorted, charted and exported.

In one embodiment of the invention, the image display604, the at least one object data graph606, and the at least one object data sheet are linked in a way such that selecting an object in one display mode also selects the object in other display modes. This facilitates efficient intermediate result analysis in a data guided or image guided fashion.

VII. Recipe Update

The recipe update118step inputs the recipe106and performs an update step selected from a group consisting of configuration selection800, configuration update804and parameter update808. This generates the updated recipe120that could include at least one of the updated configuration selection802, updated configuration806, and updated parameter810(seeFIG. 8).

A. Configuration Selection

The recipe parameters can be grouped into different sets such as detection parameters, tracking parameters, decision parameters, measurement parameters, etc. One or more parameters could exist within each set. For the parameters in a set, their values could be pre-configured into multiple pre-set configurations that are tuned for different use cases. In one embodiment of the invention as shown inFIG. 9, the grouped sets include “Detection”900and “Tracking”902. The pre-set configurations for “Detection”900are “Low”904“medium”906and “High”908. Similarly, the pre-set configurations for “Tracking”902are also “Low”,904“medium”906and “High”908. The recipe update step118could perform configuration selection800that selects, in the example embodiment, one of the “Low”,904“medium”906and “High”908from each of the “Detection”900and “Tracking”902sets. A Preview910option can be invoked to allow the viewing of the updated incremental output for the current frame or a newly selected frame. The updated configuration selection802is stored in the updated recipe120.

Other embodiment may have more or fewer pre-set configurations. In one embodiment, the configuration can be selected by selecting different values in a slider bar.

B. Configuration Update

In addition to configuration selection800, some configurations such as measurements912can also be updated. In these configurations, there are pre-defined measurement items914that can be turned on and off. In the example embodiment, the measurements912window can be invoked by clicking the “Select Measurements”916button. After the configuration update804, the updated Configuration806is stored in the updated recipe120.

C. Parameter Update

Configuration selection800is user friendly for non-technical users. But it has its limitations, as the adjustment is limited to the pre-set configurations. The parameter update808step allows more technical competent users to perform fine adjustment of the parameters underlying the pre-set configurations.FIG. 10shows one embodiment of the parameter update808. The parameters associated with each set can be made visible and adjustable to users. InFIG. 10, parameters1000-1014correspond to the set “detection”900. The pre-set configuration “Low”904“medium”906and “High”908each defines specific values for parameters1000-1014. Users can view those pre-configured values by clicking on “Low”904, “medium”906, or “High”908button. Users can adjust at least one value of the parameters1000-1014to tune the performance. The Preview910option can also be invoked to allow the viewing of the updated incremental output for the current frame or a newly selected frame after the parameter adjustment. After the parameter update808, the updated Parameter810is stored in the updated recipe120.

VIII. Result Editing

FIG. 11shows the processing flow of the result editing step. The result editing122step inputs the incremental output110and performs a mask editing1100or a track editing1104step or both. The mask editing step1100edits mask from the incremental output110to generate edited mask1102output. The track editing step1104edits track from the incremental output110to generate edited track1106output. In another embodiment of the invention, the edited mask1102output is used to perform track editing1104to generate edited track1106output. In yet another embodiment of the invention, the edited track1106is used to perform mask editing1100to generate edited mask1102output. In an alternative embodiment of the invention, the edited mask1102and/or edited track1106are processed by an incremental update1108step to generate edited incremental output124. The incremental update1108step, taking the edited mask1102and updates the incremental output to reflect the changes in masks. This generates the edited incremental output124. For example, the object count is updated if the edited mask1102changes the number of objects. For the objects whose masks are updated, their associated measurements such as size, shape, intensity statistics are also updated. In the case of tracking applications, the tracks and tracking measurements such as object velocity, lineage, etc. are also updated if the edited masks change the tracks. For the tracks updated in the edited track1106, their associated tracking measurements such as object velocity, lineage, etc. are also updated.

An embodiment of the mask editing1100step is shown inFIG. 12. The drawing tools1200allows users to draw the desired mask region in the image and a mask action1202or1204can be selected. In the example embodiment, the action includes “Add to Mask” that adds the user drawn region(s) into the mask, “Remove from Mask” that removes the user drawn region(s) from the mask and “Cancel” that cancels the user drawing. To make the mask editing1100efficient, the drawn shapes can be copied using the “Copy Drawn Shapes”1206selection. This allows copy selected1208or copy all1210to the next time frame1212or all timeframes1214.

In other embodiment, the drawing tool includes other shapes such as circle/ellipse, square/rectangle. The editing can also be extended to 3D mask editing for 3D image processing. In this case circle becomes sphere. Square becomes cube.

B. Track Editing

An embodiment of the track editing1104step is shown inFIG. 13. A user selects the tracks to be edited by clicking the trajectory display on the image. Multiple tracks can be selected. The selected tracks can also be displayed in a grid view1300. Track editing actions can be performed in either image display604and/or the grid view1300. The editing action includes connecting trajectories and other actions from the action menu1302such as Extend Trajectory, Delete Trajectory, Delete Point, Connect Lineage, etc. The connect lineage action allows the connecting of tracks as the parent and children of a division event.

In other embodiment, new track can be created by clicking the desired time frames and locations. A mask can also be manually created for each instance of the newly created track.

IX. Processing Recording

The processing recording step132generates a processing profile134output. This records the incremental apply108process and its associated update and editing steps until its completion so that the processing result130can be reproduced. It also allows the sharing of the processing information for reference and/or improvement. The processing profile134consists of initial recipe and recipe update. Also it includes the updates in the result editing122including mask and track editing steps1100/1104. In one embodiment of the invention, the processing profile134is converted to an adaptive recipe for volume data processing. The adaptive recipe will perform automatic intermediate recipe update at the time frame according to the processing profile134. The adaptive recipe can also perform automatic intermediate editing according to the processing profile134.