Quarterly Progress Report March 11, 2003

AISRP: NAS5-02098

Novel Approaches to Supervised and Unsupervised Data Exploration

David Bazell, PI

[Summary of Progress]{.underline}

This quarter we have made significant progress in understanding and
applying the semisupervised class discovery program to two astronomical
data sets. This report summarizes our results and progress this quarter.

For the past several months we have been working with two data sets: the
SDSS early release data with 54007 objects including stars, galaxies,
and quasars; and the older and smaller (and apparently poorer quality)
ESOLV data sets with 5217 galaxies. Last quarter we began comparisons of
the semisupervised class discovery code with a neural network, only to
find that the neural network did not produce the output needed to make
the performance comparisons we needed to make. That issue has been
resolved. We are currently using the neural network output from the WEKA
software, which produces class conditional probabilities for each
object. This means that the network outputs the probability it
calculates for a given object to belong to a given class. We are able to
use these probabilities to do a direct comparison between the
classification accuracy of the discovery code and the neural network.
Some representative results for the ESOLV data (5 classes) are shown in
the table.

  -------------- -------------- -------------- -------------- --------------
  Unknown Class  Ncomp          Testfrac-D     Testfrac-N     \% change

  0              27             0.08859        0.09923        -10.7

  1              42             0.1716         0.1810         -5.19

  2              70             0.3797         0.4883         -22.2

  3              48             0.2183         0.2408         -9.34

  4              14             0.05110        0.07773        -34.3

  0 1            73             0.1435         0.2805         -48.8

  0 2            13             0.4553         0.6109         -25.5

  0 3            39             0.3179         0.3403         -6.58

  0 4            34             0.1463         0.1770         -17.3
  -------------- -------------- -------------- -------------- --------------

Briefly, here is a description of the procedure used and the results.
The ESOLV data consists of information or features representing
different galaxy types, classified into five different classes (E, S0,
Sa + Sb, Sc + Sd, Irr). For each run we chose one or more classes to be
the "unknown class." This means the training data sets did not contain
any examples from the unknown class. The two classifiers then assigned
each object to either one of the known classes or to an unknown class.
Ncomp is the number of components used in the mixture model by the
discovery code. The testfrac error measure reports the error in
assigning objects to the wrong "class type", i.e., assigning an object
from a known class to an unknown class or assigning an object from an
unknown class to a known class. testfrac-D is for the discovery code
while testfrac-N is for the neural network code. The % change column
shows the percent change from testfrac-N to testfrac-D, i.e., how much
better did the discovery code do.

We see from the table that the discovery code does better in assigning
objects to unknown classes, generally by more than ten percent, and
often by more than 25 percent. The SDSS data, which is a much larger and
higher quality data set, gave similar but not quite as dramatic results.
Generally, the discovery code matched or improved the error measure
compared to the neural network.

The above results were produced by finding the run, among perhaps 70
runs, that produced the best value of testfrac-D. We also are
investigating another approach, using minimum description length, MDL,
which looks for a tradeoff between low error and model complexity. We
are currently writing up the results for publication in the Ap J.

Because of the progress in this area, we have not done much in area of
clustering that was mentioned in our last progress report.

[Problems and Issues]{.underline}

None.

[Plans for Next Quarter]{.underline}

Finish the paper we have begun and submit it to the Ap J.

Continue investigation of MDL as a method of finding optimal models.

Start work on colliding galaxy data.