RIDER -- NIST Variance phone call, 3/30/07

Chuck Fenimore

Alden Dima

Chuck Meyer

Tony Reeves

Larry Clarke

Barbara Croft

Chuck F -- NIST has funding for Radiation Physics, Brian Zimmerman, who
are quite interested in PET CT work. On Information Tech side, have
statisticians interested who work with such questions as the variation
in ground truth and image registration. There are people interested in
imaging processing and testing. There are people who have been involved
in facial recognition and image processing. They are interested in a
richer set of test objects.

He asked Alden to join this call re his view on this work.

He sent out a document, Reporting Benchmark Results for RIDER.

Face recognition efforts have been going on for the last 14 years or so.
This was a benchmarking effort. At the time of the first evaluation,
they measured FP rate for the best algorithm, which was 70% at FN rate
of 0.1%. Recently they are at about 1% FP rate.

The role of NIST is to develop software infrastructure to facilitate
this. This could be on-line tools or run at NIST. There are a variety of
design issues. The question comes up about how to report on the results,
which depends on agreement made with the developer. They report results
by name of the facial recognition tools. If they anonymize the inputs of
the methods, there is some risk to having such a bracing competitive
environment.

A second issue is that the tools are typically tuned on a specific set
of data. The sequestering of data is the issue that comes up. This is a
design issue in the future of this effort.

Larry -- RIDER data could be sequestered since we are early in the RIDER
effort.

Chuck F -- typically data sequestered in 1 round could be the training
set for the next round. Depending on the details of the variance we need
to be careful not to have a small data set. It might be that large
variance will mean the need for a large data set.

Larry - the FDA group is also very interested in these issues.

Chuck F -- they will meet with the FDA Monday a week. People are looking
at data that Marios distributed. The team is not up to speed, but are
very excited. They are also hoping that the tools being developed will
work well with caBIG efforts.

Larry -- see the Variance group as a core for RIDER. This will come down
to each group working from their strengths. There needs to be a
comfortable relationship as to the scope for each group.

The FDA NIST meeting is very encouraging.

Tony R -- when you do facial recognition, you know ground truth, but
with lesion recognition, it is not known.

Chuck F -- it is an exciting challenge. He is cautious about the
uncertainties and their effects on the results. He is asking for
algorithm submission. There is a level of uncertainty not seen in facial
recognition work. This might be more like machine reading of written
text. He looks forward to seeing the first phantom data.

Workshop on April 12 and 13^th^ -- Chuck F may make part of this
meeting.

Alden -- He has been looking at the problem from point of view of
framework needed to process the algorithms, using VTK, ITK, etc. It is
not clear what form of the algorithm would be made available. How would
they be set up?

Chuck M -- will come in a lot of forms, with no motivation to put them
in a particular format. What about the Fitzpatrick format? You calculate
results and send them back to Vanderbilt for evaluation.

Chuck F -- one question comes from previous software comparisons.
Someone will have a tool which works well on a particular set of data,
but in another environment, the conditionality of that original setting
affects its behavior in the test situation. They might want to receive
an executable version to try to run there. Would this community live
with this or would it make sense to Alden?

Alden -- you would want to get components which conform to an interface
and can be "played" from there. It gets more complicated when you do not
get to look at the system under test.

Chuck M -- industry has motivation to conform to the interface standard,
while academics do not.

Larry -- could you be funded to make the algorithm conform?

Alden -- you can validate it yourself; then to go to next level you
could conform to the test situation.

Chuck M -- not all of these algorithms are really polished.

Chuck F -- is there a way to specify the output from the algorithm that
would make comparison possible?

Alden -- for self validation, NIST passes out the data and they send the
results back in a particular format, perhaps .xml.

Chuck M -- this could work.

Alden -- this could be .xml. It does not have to be an API. You could
imagine wrapping the code in a driver.

Chuck F -- see that this should be discussed in RIDER and also NCIA
people may have some thoughts.

Chuck M -- but caImage - NCIA will have a different set of tools.

Chuck F -- are the academics using software or hardware?

Chuck M -- it is software coded for a particular environment. QWIS is in
Java, Cornell code runs locally, with a web application. U Mich is
something different from these.

Larry -- contract solicitation is coming up for change analysis
software.

We might get everyone on a phone call.

Chuck F -- NIST is very motivated to work with the RIDER group.

Larry -- and there was a need to use the algorithms to find their
variance in order to decide on the size of the dataset. This would
qualify the database for use.

Chuck F -- thinking in terms of a rapid implementation of a benchmarking
tool.

Larry - phantom data is part of addressing this.

Chuck M -- phantom data is simplistic usually.

Chuck F -- so it provides a lower estimate of the size of the database
needed.

Larry -- way to address role of NCIA is through a face-to-face meeting.
There is a sum of money allotted, but they are not highly experienced in
this area. He would hope for such a meeting, like April 10^th^.

Chuck F -- will be gone that day, and April 17^th^ might not work
either.

Larry -- wanted to bring Ram is going to India.

Chuck F- Have Alden, Jonathan Philips along with Chuck.

Larry -- a conference call might be best, like 9:30 am Tuesday the
10^th^.