CellProfiler/CellProfiler — Issue #106: JIRA [IMG-1236] Improve turnaround time for illum corrxn
Author: thouis
Created: 2011-11-23 20:50:09+00:00
State: open
Labels: next

On Jan 5, 2011, at 11:47 AM, Thouis (Ray) Jones wrote:
> Preprocessing would be to open every image and:
> - sample a handful of random pixels
> - compute the image histogram
> - (whatever else. md5sum?)
>
> And store these in a sort of data thumbnail. Then some things can be
> done from the thumbnail without opening the image (thresholdin
> calculation, illum correction). Thumbnail creation can be in parallel
> (as long as the file server can keep up), while some things (illum
> correction) work sequentially, and would be faster (I think) if done
> from thumbnail information.
>
> Then again, illum correction can be done in parallel (using a
> tree-structured computation), but it's more complicated to implement.
>
> On Wed, Jan 5, 2011 at 16:58, Anne Carpenter<anne@broadinstitute.org> wrote:
>> I see, so the workflow improvement route would not be overwhelmingly
>> complicated if we did an Illum corrxn precalc pipeline (in parallel, using
>> batch mode) followed by a short illum calc pipeline that just does one cycle
>> per needed illum function (which could be run in batch mode or regular but
>> either way would not take much time if it uses sampling). That seems pretty
>> feasible.
>>
>> (I'm a bit confused as to what the preprocessing actually is - I don't think
>> we usually do anything to the images other than open them, add/average them
>> together, then blur the result to create illum fxns)
>>
>> Instead of sampling at the illum calc pipeline stage, we might instead have
>> the precalc pipeline resize the images smaller, if that is an appropriate
>> first step for illum corrxn. This might obviate the need for sampling and
>> offer similar speed improvements?
>>
>> Anne
>>
>>
>>
>> On Jan 5, 2011, at 10:01 AM, Thouis (Ray) Jones wrote:
>>
>>> You can preprocess them in parallel, grab a hundred randomly spaced
>>> samples (jittered, not uniformly random), and compute the histogram as
>>> well. Then you can precompute the optimal Otsu threshold just by
>>> looking at the histogram.
>>>
>>>
>>> On Wed, Jan5, 2011 at 15:54, Anne Carpenter<anne@broadinstitute.org>
>>> wrote:
>>>>
>>>> (Lee, we are talking about how to improve the speed of illumination
>>>> correction calculations when there are lots of images. This is eating up
>>>> sometimes 6-12 hours for big image sets because ~4,000 images must be
>>>> opened
>>>> and averaged. In theory, we could gain speed improvements super-easily
>>>> because illum correction = averaging and we could break it into averages
>>>> of
>>>> averages - but in practice it's hard to fit averages-of-averages into
>>>> CP's
>>>> normal workflow/batch processing.)
>>>>
>>>> So the big question is whether sampling each image instead of taking the
>>>> whole thing actually saves any time, or whether reading each image is
>>>> actually the major time sink so by the time you've opened each image you
>>>> may
>>>> as well use the whole thing instead of sampling. If that's the case,
>>>> then
>>>> figuring this out from the workflow side is the only way I can see to
>>>> improve speed for this step.
>>>>
>>>> Anne
>>>>
>>>>> thouis: You could probably get a pretty good sampling just by grabbing
>>>>> a
>>>>> few hundred pixels from each image, randomly.
>>>>> But I haven't thought more about it than that
>>>>
>>>>
>>
>>
>>

Original reporter: klmadden
Jira link: https://jira.broadinstitute.org/browse/IMG-1236
Fix version: Next
Assignee: klmadden