APPENDIX A

  This appendix contains tables of results for all the ~II~EC-2 participants. The tables in Appendix A show
varions measures of the performance on the adlioc and routing tasks. The acihoc results corne first, followed by
the routing results, with the tables in the same order as the presentation order of the papers. The definitions of
the evaluation measures are given in the Overview, section 4, and readers unfamiliar with these measures should
read that section first.
  Care should be taken in comparing the tables across systerns. These measures show performance only, with
no measure of user or system effort.
  Each table contains four major boxes of statistics and three graphs.

Box 1-- Summary Statistics

   line 1-- unique run identifier, data subset, and query construction method used

      Data subset
        flill (disks 1 and 2 for adhoc, disk 3 for routing)
        category B (the official subset of data, 1/4 of the data using the Wall Street Journal articles for adhoc
                    and the San 3ose Mercury News articles for routing)

      Query construction method
        automatic
        manual
        feedback (frozen evaluation used)

 line 2-- Number of topics included in averages.

 line 3-- Total number of documents retrieved over all topics. Here, "retrieved" means having a rank less 1001.

 line 4-- Total number of relevant documents for all topics in the collection (whether retrieved or not).

 line 5-- Total number of relevant retrieved documents for this run.

Box 2-- Recall Level Averages

   lines 1-11-- The average over all topics of the precision at each of the 11 recall points given. Note
              that this is interpolated precision: e.g., for a particular topic, if the precision at 0.50
              recall is greater than the precision at 0.40 recall, then the precision at 0.50 recall
              is used for both the 0.50 and 0A0 recall levels.

   line 12--  The average precision as calculated in a non-interpolated manner
              (see section 4 of the Overview for details on this calculation).

Box 3-- Document Level Averages

   lines 1-9-- The average recall and precision after the given number of documents have been retrieved.

   line 10-- the R precision. This is a new evaluation measure being tried that
            averages the precisions found for each topic at the document level of R,
            where R is the number of relevant documents for that topic.


                                       A-i

Box 4-- Fallout Level Averages

   lihes 1-11-- The average recall (probability of detection) at a fixed fallout (probability of false al&m rate).
           Ten equally spaeed fallout points in the high precislon end of the curve were used, and for
           each topic, the highest recall value in the region surrounding that fallout point is
           selected. The table shows the averages of these points across all topics.

Graph 1-- Recall-Precision Curve

   This is a plot of the data shown in Box 2.

Graph 2-- Fallout-Recall Curve

   This is a plot of the data shown in Box 4.

Graph 3-- Normal Deviate - Fallout-Recall

   This is a plot of the data shown in Box 4, but plotted on probability scales.


                                     A-2

ults - City University, London


     L             Summary Statistics          1                                        Recall-Precision Curve
       Run Number         cityau-full1 automatic                           1
       Num of Queries                     50 j
       Total number of documents over all queries 1
                                                                         0.8  -
       Retrieved:                      50000
       Relevant:                       10785
       Rel~ret:                         6412 j                           0.6 -

ecall Level_Averages
Recall         Precision
0.00            0.7148
0.10            0.4224
0.20            0.3528
0.30            0.2983
0.40            0.2623
0.50            0.2191
0.60            0.1835
0.10            0.1299
0.80            0.0931
0.90            0.0421
 1.00           0.0000
`age precision over all
ant docs
interpolated    0.2272

  Document Level Averages
                  Precision
At 5 docs          0.5000
At 10 docs         0.4640
At 15 docs         0.4667
At 20 docs         0.4570
At 30 docs         0.4340
At 100 docs        0.3512
At 200 docs        0.2970
At 500 docs        0.1966
At 1000 docs       0.1282
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Exact              0.2849

Precision
      0.4


      0.2


        0


        1


      0.8

0       0.2     0.4     0.6     0.8


                   Recall


             Fallout-Recall Curve

0.6  -

t Level Averages            NORMAL DEVIATE - Fallout-Recall              Recall

  )~0 0.3344
      0.4481
      0.5113
      0.5761
  BO  0.6171


  60  0.6438

  00  0.6438

0.84  0.98  0                  0.4


                               0.2

0.98

2


1                                        0.84

0                                        0.5                   0
                                                                0          0.05         0.1        0.15         0.2
-1                                       0.16                                      Fallout x 100


-2                                       0.02

-3        I     I                 I      0.001
  -3    -2    -1     0     1      2    3

~sIiIts - City University, London


                    Summary Statistics
       [ Run Number          citymf-fu II, feedback                                          Recall-Precision Curve
                                                                                1
        Num of Queries                     50
        Total number of documents over all queries
        Retrieved:                      51221                                 0.8
        Relevant:                       10785
        Rel~ret:                         6409
                                                                              0.6 -

Recall Level Averages
  Recall        Precision
  0.00           0.7328
  0.10           0.4944
  0.20           0.4180
  0.30           0.3552
  0.40           0.2809
  0.50           0.2205
  0.60           0.1584
  0.70           0.0901
  0.80           0.0482
  0.90           0.0184
  1.00           0.0000
rage precision over all
~ant docs
 interpolated    0.2324

it Level Averages
        Recall
~       0.3417
20      0.4289
~0      0.4898
50      0.5366
30      0.5682
~0      0.5932
~0      0.6012
~       0.6012
50      0.6012
30      0.6012
)0      0.6012

                                 Document Level Averages            Precision
                                                Precision                  0.4
                              At S docs          0.4920
                              At 10 docs         0.5040
                              At 15 docs         0.4827                    0.2
                              At 20 docs         0.4750
                              At 30 docs         0.4680                      0
                              At 100 docs        0.4004
                              At 200 docs        0.3162
                              At 500 docs        0.1979
                              At 1000 docs       0.1275
                              R-Precision (precision after R
                                                                             1
                              docs  retrieved (where R  is
                                         of relevant docu-
                              the number
                              ments))
                              Exact              0.2971

                           NORMAL DEVIATE - Fallout-Recall

                         i01 0.02  0.16


                       -3    -2    -1   0      1    2    3

                            2                                        0.98

                                                                   Recall

                                                                          0.8


                                                                          0.6


                                                                          0.4


                                                                          0.2

                                                                          0      0.2     0.4     0.6     0.8


                                                                                            Recall


                                                                                       Fallout-Recall Curve

                                                             I     I

                            1                                        0.84


                            0                                        0.5                     0

                                                                                              0           0.05         0.1         0.15         0.2
                           -1                                        0.16                                         Fallout x 100


                           -2                                        0.02

                           -3                                        0.001

suits - Rut~ers University

                      Summary Statistics
    Run Number        rutcombx-full, partial topics, manual 1
    Num of Queries                               25
          Total number of documents over all queries
    Retrieved:                                 25000
    Relevant:                                  4119
    Rel~ret:                                   2378

~ecall Level_Averages
 Recall         Precision
  0.00           0.1145
  0.10           0.4611
  0.20           0.3894
  0.30           0.3325
  0.40           0.2841
  0.50           0.2001
  0.60           0.1502
  0.10           0.1057
  0.80           0.0101
  0.90           0.0345
  1.00           0.0104
rage precision over all
~ant docs
-interpolated    0.2284

it Level Averages
~t     Recall
DO     0.3413
20     0.4453
40     0.5131
60     0.5584
80     0.5901
DO     0.6090
20     0.6202
40     0.6202
60     0.6202
80     0.6202
DO     0.6202

                               Document Level Averages
                                             Precision
                             At 5 docs         0.4120
                             At 10 docs        0.4600
                             At 15 docs        0.4640
                             At 20 docs        0.4520
                             At 30 docs        0.4280
                             At 100 docs       0.3396
                             At 200 docs       0.2432
                             At 500 docs       0.1489
                             At 1000 docs      0.0951
                             R-Precision (precision after R
                             docs retrieved (where  R  is
                             the number of relevant docu-
                             ments))
                             Exact             0.2185

                          NORMAL DEVIATE - Fallout-Recall

                       )01  0.02            0.84  0.98 0                  0.4


                                                                          0.2

                          2                                      0.98

                          1                                      0.84

                          0                                      0.5                    0

                                                                                      Recall-Precision Curve
                                                                        1


                                                                       0.8 -


                                                                       0.6 -
                                                                Precision
                                                                       0.4


                                                                       0.2

                                                                                   I       I       I       I
                                                                        0


                                                                        1


                                                                       0.8


                                                                       0.6
                                                                Recall

                                                                          0       0.2     0.4     0.6     0.8
                                                                                             Recall


                                                                                       Fallout-Recall Curve
                                                                                     I         I         I


                                                                                     I         I         I

                                                                                          0          0.05        0.1         0.15         0.2
                         -1                                       0.16                                       Fallout x 100


                         -2                                       0.02

                         -3        I     I           I            0.001
                           -3    -2     -1     0     1     2    3

~sults - Rutgers University


                      Summary Statistics
      Run Number       rutfmed-full, partial topics, manual                                     Recall-Precision Curve
      Num of Queries                              25                              1
          Total number of documents over all queries
      Retrieved:                                24001                            0.8
      Relevant:                                 4119
      Rel~ret:                                  2314

Recall Level_Averages
 Recall         Precision
  0.00           0.6054
  0.10           0.4215
  0.20           0.3574
  0.30           0.2726
  0.40           0.2152
  0.50           0.1459
  0.60           0.1067
  0.70           0.0753
  0.80           0.0496
  0.90           0.0204
  1.00           0.0000
`rage precision over all
vant docs
-interpolated    0.1852

  Document Level Averages
                 Precision
At 5 docs         0.3920
At 10 docs        0.3880
At 15 docs        0.3787
At 20 docs        0.3860
At 30 docs        0.3733
At 100 docs       0.2892
At 200 docs       0.2256
At 500 docs       0.1418
At 1000 docs      0.0926
R-Precision (precision after R
docs  retrieved (where R   is
the number of relevant docu-
ments))
Exact             0.2464

      0.6
Precision
      0.4


      0.2


        0


        1


      0.8

0       0.2     0.4      0.6     0.8


                   Recall


             Fallout-Recall Curve

 I     I     I

it Level Averages
ut     Recall
~      0.3238
20     0.4356
40     0.4861
60     0.5370
80     0.5625
00     0.5863
20     0.5933
40     0.5933
60     0.5933
80     0.5933
00     0.5933

NORMAL DEVIATE - Fallout-Recall

      0.6
Recall

)01  0.02                 0.98  0                   0.4


                                                    0.2 -

2                                         0.98


1                                         0.84

0                                         0.5                   0

                                                                 0          0.05         0.1        0.15         0.2
-1                                        0.16                                      Fallout x 100


-2                                        0.02

-3                                        0.001
   -3    -2   -1     0      1     2     3

,ults - Cornell University

                     Summary Statistics
          Run Number         crnlL2-full, automatic 1
          Num of Queries                     50
          Total number of documents over all queries
          Retrieved:                      50000
          Relevant:                       10785
          Rel~ret:                         8224

 ecall Level_Averages
  Recall       Precision
  0.00          0.7756
  0.10          0.5381
  0.20          0.4732
  0.30          0.4321
  0.40          0.3959
  0.50          0.3585
  0.60          0.2991
  0.70          0.2357
  0.80          0.1705
  0.90          0.0904
  1.00          0.0148
 age precision over all
 ant docs
 interpolated   0.3258

  Level Averages
 t     Recall
 Po    0.4533
 0     0.5983
 *0    0.6796
 10    0.7287
 0     0.7634
 0     0.7838
 .0    0.7864
 0     0.7864
 10    0.7864
 0     0.7864
 0     0.7864

                                Document Level Averages
                                               Precision
                             At 5 docs          0.5640
                             At 10 docs         0.5460
                             At 15 docs         0.5373
                             At 20 docs         0.5210
                             At 30 docs         0.4933
                             At 100 docs        0.4406
                             At 200 docs        0.3852
                             At 500 docs        0.2600
                             At 1000 docs       0.1645
                             R~Precision (precision after R
                             docs  retrieved (where   R is
                             the number of relevant docu-
                             ments))
                             Exact              0.3641

                          NORMAL DEVIATE - Fallout-Recall

                        Ol  0.02                   0.98 0                   0.4


                                                                            0.2

                           2                                         0.98


                           1                                         0.84

                           0                                         0.5                    0

                                                                                        Recall-Precision Curve
                                                                           1


                                                                         0.8,


                                                                         0.6
                                                                  Precision
                                                                         0.4


                                                                         0.2

                                                                                              I       I
                                                                           0


                                                                           1


                                                                         0.8


                                                                         0.6
                                                                  Recall

                                                                            0      0.2       0.4      0.6     0.8      1
                                                                                                Recall


                                                                                          Fallout-Recall Curve


                                                                                        I          I         I

                                                                                              0          0.05         0.1         0.15         0.2
                          -1                                        0.16                                         Fallout x 100


                          -2                                        0.02


                          -3                                        0.001
                            -3    -2    -1     0     1      2     3

suits - Cornell University

                      Summary Statistics         ]
         Run Number           crnlV2-full1 automatic 1
         Num of Queries                       50
         Total number of documents over all queries
         Retrieved:                        50000
         Relevant:                         10785
         Rel~ret:                           8018

~ecall Level Averages
 Recall       Precision
 0.00          0.7644
 0.10          0.5209
 0.20          0.4739
 0.30          0.4388
 0.40          0.3971
 0.50          0.3544
 0.60          0.2950
 0.70          0.2241
 0.80          0.1356
 0.90          0.0661
 1.00          0.0076
~ge precision over all  1
~nt docs
nterpolated    0.3163   J

 Level Averages
       Recall
D      0.4516
       0.5815
       0.6605
)      0.7148
       0.7514
)      0.7675
)      0.7713
)      0.7713
)      0.7713

                               Document Level Averages
                            ________________    Precision
                             At 5 docs           0.5480
                             At 10 docs          0.5220
                             At 15 docs          0.5187
                             At 20 docs          0.5130
                             At 30 docs          0.5013
                             At 100 docs         0.4434
                             At 200 docs         0.3789
                             At 500 docs         0.2529
                             At 1000 docs        0.1604
                             R-Precision (precision after R
                             docs  retrieved  (where  R is
                             the number of relevant docu-
                             ments))

                                     Exact                0.3640
                                                                                           0.6
                                 NORMAL DEVIATE - Fallout-Recall

                              101  0.02                0.84  0.9g  0
                                                                     )~999                 0.4
                           2                                          0.98

                                                                                           0.2
                                                                      0.84

                           0                                          0.5                    0

                                                                  Recall

                                                                                         Recall-Precision Curve
                                                                            1


                                                                  Precision
                                                                         0.4


                                                                         0.2


                                                                            0         L        I        I        I


                                                                            1


                                                                         0.8

                                                                            0       0.2      0.4     0.6      0.8
                                                                                                Recall


                                                                                          Fallout-Recall Curve
                                                                                        I         I


                                                                                        I         I

                                                                                              0          0.05         0.1         0.15          0.2
                          -1                                        0.16                                         Fallout x 100


                          -2                                        0.02

                          -3                                        0.001

                            -3     -2    -1    0      1     2     3

ililts - University of California, Berkeley

             Summary Statistics
Run Number          Brkly3-full1 automatic 1
                                                                     1
Num of Queries                       50
Total number of documents over all queries
Retrieved:                        50000                             0.8
Relevant:                         10185
Rel~ret:                           1166                             0.6

Recall-Precision Curve

  ecall Level Averages
  Recall      Precision
   0.00        0.8461
   0.10        0.5630
   0.20        0.5016
   0.30        0.4423
   0.40        0.3946
   0.50        0.3491
   0.60        0.2963
   0.10        0.2126
   0.80        0.1490
   0.90        0.0543
   1.00        0 0040
  age precision over all
  ant docs
  interpolated 0 3210

  t Level Averages
  it   Recall
  70   0.4522
  ?0   0.5186
  10   0.6602
  )0   0.1125
  30   0.1411
  )0   0.1611
  ?0   0.1635
  10   0.1635
  ,0   0.1635
  30   0.1635
  )0.  0.1635

                               Document Level Averages
                                               Precision
                             At S docs         0.6000
                             At 10 docs        0.5160
                             At 15 docs        0.5581
                             At 20 docs        0.5400
                             At 30 docs        0.5281
                             At 100 docs       0.4568
                             At 200 docs       0.3195
                             At 500 docs       0.2485
                             At 1000 docs      0.1553
                             R-Precision (precision after R
                             docs  retrieved (where R  is
                             the number of relevant docu-

                                                                  Precision
                                                                         0.4


                                                                         0.2


                                                                          0


                                                                          1

                                   ments))                                               0.8
                                   Exact                0.3691
                                                                                         0.6
                               NORMAL DEVIATE - Fallout-Recall
                                                                                Recall
                             Ol  0.02               0.84   0.98 0                        0.4


                          2                                        0.98
                                                                                         0.2
                          1                                        0.84

                          0                                        0.5                    0

                                                                                              I        I       I


                                                                            0       0.2     0.4       0.6     0.8
                                                                                               Recall


                                                                                          Fallout-Recall Curve
                                                                                        I          I         I


                                                                                        I          I

                                                                                             0          0.05          0.1         0.15         0.2
                         -1                                         0.16                                         Fallout x 100


                         -2                                         0.02

                         -3                                         0.001
                            -3     -2    -1     0     1     2     3

~"1ts - Universitaet Dortmiind

                     Summary Statistics        ]
         Run Number         dortl2-fu II, automatic
              of Queries                    50
         Total number
         Retrieved:                      50000
         Relevant:                       10785
       jRel~ret:7774

ecall Level_Averages
Recall       Precision
0.00          0.7324
0.10          0.5410
0.20          0.4939
0.30          0.4370
0.40          0.3850
0.50          0.3324
0.60          0.2821
0.70          0.2133
0.80          0.1340
0.90          0.0707
 1.00         0.0063
ge precision over all  1
nt docs
~terpolated   0.3151   ]

Level Averages
      Recall
      0.4380
      0.5641
      0.6401
      0.6920
      0.7313
      0.7519
      0.7557
      0.7557
      0.7557
      0.7557
      0.7557

   Document Level Averages
________________     Precision
 At 5 docs           0.5120
 At 10 docs          0.5280
 At 15 docs          0.5213
 At 20 docs          0.5210
 At 30 docs          0.5120
 At 100 docs         0.4562
 At 200 docs         0.3781
 At 500 docs         0.2439
 At 1000 docs        0.1555


L

R-Precision (precision after R
docs  retrieved (where  R  is
the number of relevant docu-
ments))
Exact             0.3604

  NORMAL DEVIATE - Fallout-Recall

Ol  0.02

0.84  0.98

                      Recall-Precision Curve
        1


      0.8


      0.6
Precision
      0.4


      0.2


        0          I        I               I


Recall

 1


0.8


0.6


0.4

2                                         0.98

                                                               0.2
1                                         0.84

0                                         0.5
                                                                0

0       0.2     0.4      0.6     0.8
                    Recall


              Fallout-Recall Curve
            I         I


            I         I

-1                                                                 0          0.05          0.1         0.15         0.2
                                          0.16
                                                                                       Fallout x 100

-2

-3

0.02

0.001

-3     -2    -1     0     1     2      3

Lilts - Universitaet Dortmund

        [           Summary Statistics
         Run Number        dortQ2-full, automatic
         Num of Queries                     50
         Total number of documents over all queries
         Retrieved:                      50000
         Relevant:                       10785
         Rel~ret:                         8259

`call Level Averages
~ecall        Precision
0.00           0.7673
0.10           0.5518
0.20           0.5060
0.30           0.4572
0.40           0.4114
0.50           0.3635
0.60           0.3048
0.70           0.2388
0.80           0.1565
0.90           0.0777
 1.00          0.0086
ge precision over all
nt docs
iterpolated    0.3340  1

                              Document Level Averages
                                              Precision
                            At 5 docs          0.5560
                            At 10 docs         0.5440
                            At 15 docs         0.5427
                            At 20 docs         0.5440
                            At 30 docs         0.5213
                            At 100 docs        0.4626
                            At 200 docs        0.3959
                            At 500 docs        0.2607
                            At 1000 docs       0.1652
                            R-Precision (precision after R
                            docs retrieved  (where R  is
                            the number of relevant docu-
                            ments))
                            Exact              0.3722

                                                                          1


                                                                        0.8


                                                                        0.6
                                                                 Precision
                                                                        0.4


                                                                        0.2


                                                                          0


                                                                          1


                                                                        0.8

                                                                            0.6
Level Averages             NORMAL DEVIATE - Fallout-Recall
                                                                     Recall
     Recall
                         ol  0.02                 0.98 0 )~999              0.4
     0.4625
     0.5959
     0.6731
     0.7216
     0.7643
     0.7850
     0.7908
     0.7908
     0.7908
     0.7908
     0.7908

                      2                                    0.98
                                                                             0.2
                      1                                    0.84

                      0                                    0.5                 0

                                                                                        Recall-Precision Curve


                                                                           0      0.2       0.4     0.6     0.8
                                                                                               Recall


                                                                                         Fallout-Recall Curve
                                                                                       I         I


                                                                                       I         I

                                                                                                0           0.05          0.1          0.15          0.2
                          -1                                          0.16                                           Fallout x 100


                          -2                                          0.02

                          -3         I     I                   I      0.001
                             -3     -2    -1     0      1      2     3

~su1ts - University of Massachusetts at Amherq.t

          [            Summary Statistics
           Run Number        INQOOl-full, automatic 1
           Num of Queries                      50
           Total number of documents over all queries
           Retrieved:                       50000
           Relevant:                        10785
           Rel~ret:                          8281

 ~ecall Level Averages
   Recall       Precision
   0.00          0.8275
   0~10          0.5979
   0.20          0.5458
   0.30          0.4832
   0.40          0.4369
   0.50          0.3758
   0.60          0.3181
   0.70          0.2382
   0.80          0.1634
   0.90          0.0825
   1.00          0.0094
 `age precision over all  I
 `ant docs
 interpolated    0.3556

 t Level Averages
 t      Recall
   0    0.4769
 .0     0.6107
 .0     0.6913
   0    0.7427
 `0     0.7713
   0    0.7890
   0    0.1933
   0    0.7933
   0    0.7933
   0    0.7933
   0    0.7933

                                Document Level Averages
                             ________________    Precision
                              At 5 docs           0.6160
                              At 10 docs          0.5960
                              At 15 docs          0.5880
                              At 20 docs          0.5840
                              At 30 docs          0.5733
                              At 100 docs         0.4934
                              At 200 docs         0.4099
                              At 500 docs         0.2665
                              At 1000 docs        0.1656
                              R-Precision (precision after R
                              docs retrieved   (where R  is
                              the number of relevant docu-
                              ments))

                                   Exact               0.3946
                                                                                      0.6
                               NORMAL DEVIATE - Fallout-Recall

                             )01 0.02               0.84 0.98   0
                                                                 )~999                0.4
                          2                                       0.98

                                                                                      0.2
                          1                                       0.84

                          0                                       0.5
                                                                                        0

                                                                   Recall

                                                                                          Recall-Precision Curve
                                                                             1


                                                                          0.8'


                                                                          0.6
                                                                   Precision
                                                                          0.4


                                                                          0.2


                                                                             0         I        I       I


                                                                             1


                                                                          0.8

                                                                             0      0.2       0.4      0.6     0.8
                                                                                                 Recall


                                                                                           Fallout-Recall Curve
                                                                                         I          I          I


                                                                                         I          I          I

                                                                              0        0.05       0.1       0.15       0.2
                      -1                                 0.16
                                                                                              Fallout x 100
                      -2                                 0.02

                      -3                                 0.001
                        -3    -2   -1    0    1    2    3

"its - University of Massachusetts at Amherst

                      Summary Statistics
         Run Number            INQOO2-full, manual                             1
         Num of Queries                      50
       L
        4otal number of documents over all queries
         Retrieved:                        50000
         Relevant:                         10185
         Rel~ret:                          8165

  mcall Level Averages
  Recall     Precision
  0.00        0.1162
  0.10        0.6299
  0.20        0.5604
  0.30        0.4920
  0.40        0.4331
  0.50        0.3122
  0.60        0.3133
  0.10        0.2268
  0.80        0.1545
  0.90        0.0815
  1.00        0.0122
  ~ge precision over all 1
  ~nt docs
  nterpolated 0.3565     ]

  Level Averages
  t~  Recall
      0.4860
      0.6161              2
      0.6885
      0.1359              1
      0.1639
                          0
      0.1802
      0.1859             -1
      0.1859
      0.1859             -2
      0.1859
                         -3
      0.1859

                                                                     0.8

                                                                      0.6
                                                               Precision
                                                                      0.4


                                                                      0.2


                                                                       0

                                                                       1


                                                                     0.8


                                                                     0.6

                             Document Level Averages
                                            Precision
                           At 5 docs         0.6000
                           At 10 docs        0.6160
                           At 15 docs        0.6040
                           At 20 docs        0.6140
                           At 30 docs        0.5880
                           At 100 docs       0.5058
                           At 200 docs       0.4089
                           At 500 docs       0.2626
                           At 1000 docs      0.1633
                           R-Precision (precision after R
                           docs  retrieved (where R is
                           the number of relevant docu-
                           ments))

                           Exact            0.3954

                        NORMAL DEVIATE - Fallout-Recall
                                                                Recall

                                                                                     Recall- Precision Curve


                                                                                  I       I               I    -

                                                                        0      0.2       0.4    0.6     0.8      1
                                                                                           Recall


                                                                                      Fallout-Recall Curve

                                                            0.98  0.                        0.4


                                                                     0.98
                                                                                            0.2
                                                                     0.84

                                                                     0.5                     0
                                                                                               0           0.05          0.1          0.15         0.2
                                                                     0.16                                           Fallout x 100

                                                                     0.02

                                                                     0.001
                            -3     -2    -1     0      1     2      3

suits - HNC, Inc.


                    Summary Statistics
        Run Number        HNCad1-full, automatic                                          Recall-Precision Curve
        Num of Queries                      50                               1
        Total number of documents over all queries
        Retrieved:                       50000                             0.8
        Relevant:                        10785
        Rel~ret:                          6944
                                                                           0.6

 ecall Level Averages
 Recall      Precision
 0.00         0.7750
 0.10         0.5762
 0.20         0.4934
 0.30         0.4215
 0.40         0.3536
 0.50         0.2672
 0.60         0.2041
 0.70         0.1271
 0.80         0.0575
 0.90         0.0215
 1.00         0.0007
 ~ge precision over all
 ~nt docs
 nterpolated  0.2787

 Level Averages
       Recall
 0     0.4087
 0     0.5261
 0     0.5867
       0.6272
       0.6533
 )     0.6681
 )     0.6720
 )     0.6720
 )     0.6720

  Document Level Averages
                 Precision
At 5 docs         0.5560
At 10 docs        0.5680
At 15 docs        0.5507
At 20 docs        0.5470
At 30 docs        0.5367
At 100 docs       0.4520
At 200 docs       0.3652
At 500 docs       0.2247
At 1000 docs      0.1389
R-Precision (precision after R
docs  retrieved (where R  Is
the number of relevant docu-
ments))

Precision
      0.4


      0.2


        0          I        I               I
         0       0.2      0.4      0.6     0.8
                             Recall


                       Fallout-Recall Curve

 1


0.8

          Exact                0.3474
                                                                0.6
      NORMAL DEVIATE      Fallout-Recall

    ~1   0.02  0.16        0.84   0.98  0
                                              99                0.4<

2                                          0.98

                                                                0.2
1                                          0.84

0                                          0.5                    0

                                                                    0          0.05          0.1         0.15         0.2
-1                                         0.16                                        Fallout x 100


-2                                         0.02

-3                                         0.001
  -3     -2    -1     0      1           3

Recall

suits - HNC, Inc.

L

          Summary Statistics
Run Number         HNCad2-full, feedback
Num of Queries                     50

1

1

Total number of documents over all queries
                                                                    0.8<
Retrieved:                        50000
Relevant:                         10785
Rel~ret:                           6911                             0.6

Recall-Precision Curve

~ecall Level Averages
 Recall         Precision
   0.00          0.7811
   0.10          0.5920
   0.20          0.5262
   0.30          0.4429
   0.40          0.3574
   0.50          0.2767
   0.60          0.2059
   0.70          0.1250
   0.80          0.0713
   0.90          0.0218
   1.00          0.0006
`rage precision over all
vant docs
-interpolated    0.2882

~t Level Averages
ut      Recall
                          31
~       0.4132
~20     0.5207            2
~0      0.5790
)60     0.6233            1
)80     0.6485
                          0
.00     0.6643
L20     0.6676           -1
L40     0.6676
L60     0.6676           -2
L80     0.6676
                         -3
?00     0.6676

  Document Level Averages
                 Precision
At 5 docs         0.5560
At 10 docs        0.5680
At 15 docs        0.5507
At 20 docs        0.5470
At 30 docs        0.5513
At 100 docs       0.4784
At 200 docs       0.3714
At 500 docs       0.2243
At 1000 docs      0.1382
R-Precision (precision after R
                (where R  is
               relevant docu-

       docs  retrieved
       the number of
       ments))
       Exact               0.3575

    NORMAL DEVIATE - Fallout-Recall

.001  0.02 0.16  0.5   0.84  0.98  0.C


       I    I           I     I

-~7--

-3     -2    -1     0     1     2     3

Precision
      0.4

                   0.2


                    0


                    1


                   0.8


                   0.6


I~999              0.4~

0.98
                   0.2
0.84

0.5                 0

Recall

                          I       I


0       0.2     0.4      0.6     0.8      1
                   Recall


             Fallout-Recall Curve


           I          I

                          0          0.05         0.1         0.15         0.2
0.16                                         Fallout x 100


0.02

0.001

esults - Belicore


                      Summary Statistics         j
         Run Number            Isial-full, automatic                                        Recall-Precision Curve
                                                                               1
         Num of Queries                       50
         Total number of documents over all queries
         Retrieved:                        50000                             0.8
         Relevant:                         10785
         Rel~ret:                           4756
                                                                             0.6<

Recall Level Averages
   Recall       Precision
   0.00         0.5897
   0.10         0.3100
   0.20         0.2392
   0.30         0.1929
   0.40         0.1482
   0.50         0.1077
   0.60         0.0616
   0.70         0.0257
   0.80         0.0159
   0.90         0.0000
   1.00         0.0000
rage precision over all  1
~ant docs
-interpolated   0.1307   1

it Level Averages
Lit     Recall
0~0     0.2200
20      0.3034
40      0.3601
~0      0.4050
BO      0.4350
~0      0.4576
~0      0.4642
~0      0.4642
,0      0.4642
30      0.4642
)0      0.4642

  Document Level Averages
                 Precision
At 5 docs         0.3720
At 10 docs        0.3340
At 15 docs        0.3200
At 20 docs        0.3210
At 30 docs        0.3060
At 100 docs       0.2664
At 200 docs       0.2092
At 500 docs       0.1406
At 1000 docs      0.0951
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))

Precision
       0.4


       0.2


        0


        1


       0.8

         L Exact               0.1937
                                                                 0.6
      NORMAL DEVIATE - Fallout-Recall

    )01  0.02              0.84   0.98  0
                                          ~~999                  0.4
2                                          0.98

                                                                 0.2
1                                          0.84

0                                          0.5                    0

                                                                    0          0.05          0.1         0.15         0.2
-1                                         0.16
                                                                                        Fallout x 100
-2                                         0.02

-3                                         0.001
   -3    -2     -1    0     1     2       3

Recall

0       0.2     0.4      0.6     0.8


                   Recall


              Fallout-Recall Curve

 I          I

~sIiIts - Beilcore


                      Summary Statistics
           Run Number         Isiasm-full, automatic                                         Recall-Precision Curve
                                                                               1
           Num of Queries                     50
           Total number of documents over all queries
           Retrieved:                      50000                              0.8 -
           Relevant:                       10785
           Rel~ret:                         7869

Recall Level Averages
  Recall        Precision
  0.00          0.7127
  0.10          0.5122
  0.20          0.4698
  0.30          0.4241
  0.40          0.3753
  0.50          0.3379
  0.60          0.2758
  0.70          0.2073
  0.80          0.1331
  0.90          0.0443
  1.00          0.0000
rage precision over all
vant docs
-interpolated   0.3018

it Level Averages
ut      Recall
00      0.4272
20      0.5560
40      0.6362
60      0.6927
80      0.7323
DO      0.7493
20      0.1540
40      0.7540
60      0.7540
80      0.7540
DO      0.7540

                               Document Level Averages
                                               Precision
                             At 5 docs          0.5240
                             At 10 docs         0.5020
                             At 15 docs         0.5027
                             At 20 docs         0.4950
                             At 30 docs         0.4873
                             At 100 docs        0.4306
                             At 200 docs        0.3716
                             At 500 docs        0.2469
                             At 1000 docs       0.1574
                             R-Precision (precision after R
                             docs  retrieved (where R  is
                             the number of relevant docu-
                             ments))
                             Exact              0.3580

                          NORMAL DEVIATE - Fallout-Recall

                                                                          0.6
                                                                   Precision
                                                                          0.4


                                                                          0.2

                                                                                      I        I        I       I
                                                                           0
                                                                             0       0.2      0.4     0.6      0.8      1
                                                                                                 Recall


                                                                                           Fallout-Recall Curve

                                                                           1


                                                                          0.8


                                                                          0.6

                                                                   Recall

                              101 0.02                     0.98  0                      0.4


                          2                                         0.98
                                                                                        0.2
                          1                                         0.84

                          0                                         0.5                   0

                                                                                           0          0.05         0.1        0.15         0.2
                          -1                                        0.16                                      Fallout x 100


                          -2                                        0.02

                          -3                                        0.001
                            -3    -2     -1     0     1     2     3

esults - Thinking Machines Corn.


         [             Summary Statistics
         [ Run Number         TMC8-full, automatic                                           Recall-Precision Curve
                                                                                1
           Num of Queries                     50
           Total number of documents over all queries
           Retrieved:                      49998                               0.8
           Relevant:                       10185
           Rel~ret:                         6497

Recall Level Averages
  Recall        Precision
  0.00          0.5482
  0.10          0.4005
  0.20          0.3411
  0.30          0.2911
  0.40          0.2525
  0.50          0.1914
  0.60          0.1341
  0.10          0.0855
  0.80          0.0509
  0.90          0.0125
  1.00          0.0000
rage precision over all  1
~docs
-interpolated   0.1939

it Level Averages
~t     Recall
DO     0.2962
20     0.4158
~      0.4942
50     0.5490
~0     0.5911
~0     0.6191
~0     0.6260
~      0.6260
,0     0.6260
30     0.6260
)0     0.6260

                                Document Level Averages
                                               Precision
                              At 5 docs          0.3520
                              At 10 docs         0.3680
                              At 15 docs         0.3733
                              At 20 docs         0.3180
                              At 30 docs         0.3820
                              At 100 docs        0.3342
                              At 200 docs        0.2835
                              At 500 docs        0.1942
                              At 1000 docs       0.1299
                              R-Precision (precision after R
                              docs  retrieved (where  R  is
                              the number of relevant docu-
                              ments))
                              Exact              0.2612

                           NORMAL DEVIATE - Fallout-Recall

                        101  0.02 0.16  ( 5   0.84  0.98 0

                                                                        0.6
                                                                 Precision
                                                                        0.4


                                                                        0.2


                                                                         0


                                                                 Recall

                                                                         1


                                                                        0.8


                                                                        0.6


                                                                        0.4

                           2                                      0.98

                                                                                       0.2
                           1                                      0.84

                           0                                      0.5                   0

                                                                           0       0.2     0.4     0.6     0.8
                                                                                              Recall


                                                                                        Fallout-Recall Curve
                                                                                      I         I         I


                                                                                      I         I

                                                                                           0          0.05         0.1        0.15         0.2
                          -1                                       0.16                                       Fallout x 100


                          -2                                       0.02

                          -3                                       0.001
                            -3    -2    -1     0      1     2     3

Lilts - Thinking Machines Corp.

          Summary Statistics
Run Number         TMC9-full, automatic
Num of Queries                    50
Total number of documents over all queries

1

Retrieved:                       49998                           0.8
Relevant:                        10185
Rel~ret:                         6181                            0.6  -

Recall-Precision Curve

`call Level_Averages
Recall       Precision
0.00          0.5141
0.10          0.3692
0.20          0.3195
0.30          0.2104
0.40          0.2203
0.50          0.1113
0.60          0.1144
0.70          0.0682
0.80          0.0357
0.90          0.0104
 1.00         0.0000
ige precision over all
mt docs
iterpolated   0.1136

                             Document Level Averages
                                            Precision
                           At 5 docs         0.2800
                           At 10 docs        0.3180
                           At 15 docs        0.3320
                           At 20 docs        0.3450
                           At 30 docs        0.3501
                           At 100 docs       0.3144
                           At 200 docs       0.2661
                           At 500 docs       0.1840
                           At 1000 docs      0.1237
                           R-Precision (precision after R
                           docs  retrieved (where R is
                           the number of relevant docu-
                           ments))
                           Exact             0.2436

                                                               Precision
                                                                     0.4


                                                                     0.2


                                                                       0

Level Averages             NORMAL DEVIATE - Fallout-Recall
                                                                  Recall
    Recall
    0.2125
    0.3846
    0.4610
    0.5126
    0.5540
    0.5827
    0.5904
    0.5904
    0.5904
    0.5904
    0.5904

                                                                         1


                                                                        0.8


                                                                        0.6

                           101  0.02   0.16  ( 5   0.84   0.98  0   9                  0.4


                        2                                         0.98
                                                                                       0.2
                        1                                         0.84

                        0                                         0.5                    0

                                                                                  I       I       I


                                                                        0      0.2      0.4      0.6     0.8
                                                                                           Recall


                                                                                      Fallout-Recall Curve
                                                                                    I         I


                                                                                    I         I

                                                                                          0         0.05         0.1        0.15         0.2
                        -1                                       0.16                                       Fallout x 100


                        -2                                       0.02

                        -3        I      I           I     I     0.001
                          -3     -2    -1     0      1     2    3

"ults - New York University


                    Summary Statistics            ]
      Run Number        nyuirl-category B, automatic                                        Recall-Precision Curve
      Num of Queries                           50                             1
       Total number of documents over all queries
      Retrieved:                            49884                           0.8 -
      Relevant:                              3929
      Reljet:                                2983
                                                                            0.6 -

~call Level Averages
Recall        Precision
0.00           0.7013
0.10           0.4874
0.20           0.4326
0.30           0.3531
0.40           0.3016
0.50           0.2637
0.60           0.2175
0.10           0.1617
0.80           0.1176
0.90           0.0684
1.00           0.0102
ge precision over all
nt docs
iterpolated    0.2649  J

Level Averages
       Recall
       0.2825
       0.4257
       0.4979
       0.5494
       0.5768
       0.6112
       0.6395
       0.6574
       0.6757
       0.6905
       0.7014

2

  Document Level Averages
                  Precision
At 5 docs          0.4920
At 10 docs         0.4420
At 15 docs         0.4240
At 20 docs         0.4050
At 30 docs         0.3640
At 100 docs        0.2720
At 200 docs        0.1886
At 500 docs        0.1026
At 1000 docs       0.0597
R-Precision (precision after R
docs  retrieved (where R  is
               relevant docu-

        the number of
        ments))
        Exact          0.3003

    NORMAL DEVIATE - ~1out-RecaII

  101 0.02


-3    -2    -1   0    1   2    3

                                                  0.2
1                                0.84

0                                0.5
                                                    0
                                                     0        0.05       0.1      0.15        0.2
-1                               0.16
                                                                     Fallout x 100
-2                               0.02


-3

0.001

           Precision
                 0.4


                 0.2


                   0


                   1


                 0.8


                 0.6


.999             0.4


0.98

Recall

0       0.2      0.4     0.6     0.8


                    Recall


              Fallout-Recall Curve

 I     I    I

ilts - New York University

              Summary Statistics
Run Number        nyuir2-category B, automatic                         1
Num of Queries                           50 3
 Total number of documents over all queries
                                                                     0.8
Retrieved:                           49876
Relevant:                             3929
Rel~ret:                              3274 I                         0.6 -

Recall-Precision Curve

  `call Level Averages
  Recall      Precision
  0.00         0.7528
  0.10         0.5567
  0.20         0.4721
  0.30         0.4060
  0.40         0.3617
  0.50         0.3135
  0.60         0.2703
  0.70         0.2231
  0.80         0.1667
  0.90         0.0915
  1.00         0.0154
  ~ge precision over all
  ~nt docs
  nterpolated  0.3111 ]

  Level Averages
  t    Recall
  0    0.3407
  0    0.4886
  ~    0.5620
  ~0   0.6193
  ~0   0.6569
  )0   0.7012
  ~0   0.7202
  10   0.7435
  )0   0.7572
  30   0.7732
  )0   0.7842

                              Document Level Averages                 Precision
                                              Precision                      0.4
                            At 5 docs         0.5360
                            At 10 docs        0.4880
                                                                             0.2
                            At 15 docs        0.4693
                            At 20 docs        0.4390
                            At 30 docs        0.4067                          0
                            At 100 docs       0.3094
                            At 200 docs       0.2139
                            At 500 docs       0.1137
                            At 1000 docs      0.0655
                            R-Precision (precision after R                    I
                            docs  retrieved (where R   is
                            the number of relevant docu-
                            ments))                                          0.8
                            Exact             0.3320
                                                                             0.6
                         NORMAL DEVIATE - Fallout-Recall
                                                                      Recall
                                            0.84 098                         0.4


                                                                             0.2

                                                             0.98

                                                                           0       0.2      0.4     0.6     0.8


                                                                                               Recall


                                                                                         Fallout-Recall Curve

                                                                       I     I

                       2


                       1                                       0.84

                       0                                       0.5                  0
                                                                                      0         0.05        0.1        0.15         0.2
                       -1                                      0.16                                     Fallout x 100


                       -2                                      0.02

                       -3                                      0.001
                         -3    -2    -1     0     1     2    3

~u1ts - New York University


                  Summary Statistics
      Run Number      nyuir3-category B, automatic                                        Recall-Precision Curve
      Num of Queries                         50                              1

           lotal number of documents over all queries
         Retrieved:                         49811                           0.8
         Relevant:                           3929
         Reljet:                             3281

ecall Level Averages
 Recall        Precision
 0.00           0.7528
 0.10           0.5514
 0.20           0.4724
 0.30           0.4076
 0.40           0.3621
 0.50           0.3142
 0.60           0.2711
 0.10           0.2237
 0.80           0.1691
 0.90           0.0916
 1.00           0.0160
lge precision over all
Int docs
nterpolated     0.3118

 Level Averages

                               Document Level Averages
                                               Precision
                             At 5 docs          0.5360
                             At 10 docs         0.4880
                             At 15 docs         0.4101
                             At 20 docs         0.4410
                             At 30 docs         0.4080
                             At 100 docs        0.3094
                             At 200 docs        0.2140
                             At 500 docs        0.1140
                             At 1000 docs       0.0656
                             R-Precision (precision after R
                             docs retrieved  (where R  is
                             the number of relevant docu-
                             ments))

                                                                         0.6
                                                                  Precision
                                                                         0.4


                                                                         0.2


                                                                           0


                                                                           1


                                                                         0.8

                                                                            0       0.2     0.4     0.6     0.8


                                                                                               Recall


                                                                                         Fallout-Recall Curve

                                                            I     I     I     I

                                  Exact              0.3321
                                                                                   0.6
                              NORMAL DEVIATE - Fallout-Recall

                                                  0.84 098   0
                                                                                   0.4
                         2                                     0.98

                                                                                   0.2
                         1                                     0.84


                         0                                     0.5                   0

                                                                                      0          0.05        0.1        0.15        0.2
                        -1                                     0.16
                                                                                                        FalloutxlOO
                        -2                                     0.02


                                                               0.001
                          -3     -2    -1    0     1    2     3

      Recall                                                 Recall
  D   0.3412
  D   0.4885
  D   0.5629
      0.6220
      0.6586
      0.6991
      0.7206
  3   0.7445
  3   0.1581
  3   0.1744
  3   0.7866

suits - Carnegie Mellon University

 I          Summary Statistics
  Run Number      CLARTA-full, automatic 1
  Num of Queries                    50

1

          Total number of documents over all queries
          Retrieved:                        50000                              0.8
          Relevant:                         10785
          Rel~ret:                           8109
                                                                               0.6

~ecall Level Averages               Document Level Averages             Precision
Recall         Precision                           Precision                   0.4
 0.00           0.7581            At 5 docs         0.5800
 0.10           0.5676            At 10 docs        0.5680
                                                                               0.2
 0.20           0.5038            At 15 docs        0.5547
 0.30           0.4489            At 20 docs        0.5490
 0.40           0.3938            At 30 docs        0.5253                       0
 0.50           0.3455            At 100 docs       0.4644
 0.60           0.2806            At 200 docs       0.3874
 0.70           0.2172            At 500 docs       0.2542
 0.80           0.1463            At 1000 docs      0.1622
 0.90           0.0827            R-Precision (precision after R
                                                                                 1
 1.00           0.0070            docs  retrieved (where R  is
~age precision over all           the number of relevant docu-
~ant docs                         ments))                                      0.8
interpolated    0.3264    ]       Exact             0.3645

             Recall-Precision Curve


                           I       I


0       0.2      0.4      0.6     0.8
                    Recall


              Fallout-Recall Curve
            I          I         I

t Level Averages
Pt      Recall
)~0     0.4407
)~0     0.5756
~       0.6613
,0      0.7169
30      0.7520
)0      0.7732
~       0.7767
~       0.7767
,0      0.7767
30      0.7767
)0      0.7767

NORMAL DEVIATE - Fallout-Recall

      0.6
Recall

    301 0.02                     0.98  0                      0.4


2                                        0.98
                                                              0.2
1                                        0.84

0                                        0.5                   0
                                                                 0          0.05         0.1        0.15         0.2
-1                                       0.16                                       Fallout x 100


-2                                       0.02

-3                                       0.001
  -3    -2     -1     0     1     2     3

`suits - Carnegie Mellon University

        L            Summary Statistics        ]
        F Run Number       CLARTM-full1 manual
         Num of Queries                     50
         Total number of documents over all queries
         Retrieved:                      50000
         Relevant:                       10785
         Rel~ret:                         8229

 ~ecall Level Averages
 Recall        Precision
  0.00          0.1455
  0.10          0.5811
  0.20          0.5240
  0.30          0.4622
  0.40          0.4135
  0.50          0.3593
  0.60          0.2983
  0.10          0.2312
  0.80          0.1516
  0.90          0.0814
  1.00          0.0062
 `age precision over all
 ant docs
 interpolated   0.3383 j

 t Level Averages
       Recall
 Fo    0.4521
 0     0.5940
 ~     0.6130
 0     0.7211
 0     0.1696
 0     0.1868
 .0    0.7932
 0     0.7932
 0     0.1932
 0     0.1932
 `0    0.1932

                               Document Level Averages
                                               Precision
                             At 5 docs         0.5840
                             At 10 docs        0.5140
                             At 15 docs        0.5640
                             At 20 docs        0.5590
                             At 30 docs        0.5433
                             At 100 docs       0.4846
                             At 200 docs       0.3915
                             At 500 docs       0.2601
                             At 1000 docs      0.1646
                             R-Precision (precision after R
                             docs  retrieved (where R  is
                             the number of relevant docu-
                             ments))
                             Exact             0.3141

                          NORMAL DEVIATE - Fallout-Recall

                       )01  0.02 0.16

                                                                                      Recall-Precision Curve
                                                                         1


                                                                       0.8


                                                                       0.6 -
                                                                Precision
                                                                       0.4


                                                                       0.2

                                                                                   I               I       I
                                                                         0


                                                                         1


                                                                       0.8


                                                                       0.6
                                                                Recall

                                          0.R4  0.98
                                                                        0.4


                                                                        0.2

                          2                                      0.98


                          1                                      0.84

                          0                                      0.5                    0

                                                                          0       0.2     0.4     0.6     0.8      1
                                                                                             Recall


                                                                                       Fallout-Recall Curve


                                                                                     I         I         I

                                                                                          0          0.05        0.1         0.15         0.2
                         -1                                       0.16                                       Fallout x 100


                         -2                                       0.02

                         -3                                       0.001
                           -3    -2    -1      0     1     2    3

"its - Swiss Federal Institute of Technology (ETH)

              Summary Statistics         I
F Run Number         schaul-full, automatic                          1
 Num of Queries                      50 j
 Total number of documents over all queries 1
                                                                    0.8 ~-,
 Retrieved:                       50000 1
 Relevant:                        10785
 Reljet:                           7081 j                           0.6 -

Recall-Precision Curve

~call Level Averages
Recall       Precision
0.00          0.1692
0.10          0.4912
0.20          0.4231
0.30          0.3649
0.40          0.3025
0.50          0.2529
0.60          0.1887
0.70          0.1213
0.80          0.0127
0.90          0.0403
 1.00         0.0000
~ge precision over all
~nt docs
nterpolated   0.2517 1

Level Averages
t     Recall
                        31
      0.3620
      0.4763            2
      0.5547
      0.6102            1
      0.6507
                        0
      0.6743
      0.6771          -1
      0.6771
      0.6771          -2
      0.6171
                      -3
      0.6771

          Document Level Averages
                        Precision
       At 5 docs          0.5360
       At 10 docs         0.4960
       At 15 docs         0.4800
       At 20 docs         0.4720
       At 30 docs         0.4513
       At 100 docs        0.4000
       At 200 docs        0.3302
       At~500 docs        0.2158
       At 1000 docs       0.1416
       R-Precision (precision after R
       docs  retrieved (where  R  is
                    of relevant docu-
       the number
       ments))
       Exact              0.3148

    NORMAL DEVIATE - Fallout-Recall

.001  0.02  0.16  0.5  0~84  0.98 0.~


       I     I          I     I

-~7-

-3    -2    -1     0     1

3

           Precision
                  0.4


                  0.2


                   0


                   1


                  0.8


                  0.6


~999              0.4

0.98
                  0.2
0.84

0.5                0


0.16


0.02

0.001

Recall

                 I        I       I


0       0.2     0.4      0.6     0.8
                   Recall


             Fallout-Recall Curve
           I         I


           I         I
0        0.05       0.1        0.15      0.2
                Fallout x 100

~su1ts - The Environment Research Institute of Michi~an

                      Summary Statistics
          Run Number         erimal-full1 automatic
          Num of Queries                      50 J
          TOtd'l number of documents over all~queries
          Retrieved:                       48809
          Relevant:                        10185
          Rel~ret:                          5349

Recall Level_Averages
 Recall         Precision
  0.00           0.6409
  0.10           0.3564
  0.20           0.2893
  0.30           0.2251
  0.40           0.1693
  0.50           0.1332
  0.60           0.0911
  0.70           0.0610
  0.80           0.0310
  0.90           0.0048
  1.00           0.0000
rage precision over all
~ant docs
-interpolated    0.1589

  Document Level Averages
                 Precision
At 5 docs         0.4360
At 10 docs        0.3960
At 15 docs        0.3827
At 20 docs        0.3810
At 30 docs        0.3660
At 100 docs       0.3016
At 200 docs       0.2378
At 500 docs       0.1598
At 1000 docs      0.1070
R-Precision (precision after R
docs  retrieved (where R  is
               relevant docu-
the number of
ments))
Exact             0.2179

                     Recall-Precision Curve
        1


      0.8 -


      0.6
Precision
      0.4


      0.2

            I        I
 0


 1


0.8

0       0.2     0.4      0.6     0.8
                    Recall


              Fallout-Recall Curve
            I                   I

it Level Averages
Lit     Recall
DO      0.2423
20      0.3411
40      0.4053
60      0.4460
BO      0.4827
DO      0.5138
20      0.5221
40      0.5221
60      0.5221
BO      0.5221
~0      0.5221

                                                         0.6
      NORMAL DEVIATE - Fallout-Recall

   )Oi                        0.98 0                     0.4


2                                    0.98
                                                         0.2
1                                    0.84


0                                    0.5                   0

                                                            0         0.05        0.1        0.15       0.2
-1                                   0.16                                     Fallout x 100


-2                                   0.02

-3                                   0.001

Recall

-3    -2    -1    0     1    2     3

ults - The Environment Research Institute of Michigan


       I            Summary Statistics                                                     Recall-Precision Curve
       ~ Run Number        erima2-full, automatic 1                           1
        Num of Queries                      50
        Total number of documents over all queries
                                                                            0.8
        Retrieved:                       47943
        Relevant:                        10785
        Rel~ret:                          5975                              0.6

ecall Level_Averages
 Recall         Precision
0.00             0.7096
0.10             0.4059
0.20             0.3287
0.30             0.2723
0.40             0.2276
0.50             0.1698
0.60             0.1302
0.70             0.0707
0.80             0.0480
0.90             0.0011
1.00             0.0000
rage precision over all   1
~ant docs
-interpolated    0.1885   1

  Document Level Averages
                 Precision
At 5 docs         0.4760
At 10 docs        0.4480
At 15 docs        0.4320
At 20 docs        0.4180
At 30 docs        0.3993
At 100 docs       0.3426
At 200 docs       0.2707
At 500 docs       0.1799
At 1000 docs      0.1195
~R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Fxact             0.2494

Precision
      0.4


      0.2 -


        0


        1


      0.8


      0.6

   it Level Averages          NORMAL DEVIATE - Fallout-Recall               Recall
   Tht  Recall                                       0.98      0                  0.4
   00   0.2717
   20   0.3824
   40   0.4449
   60   0.4979
   80   0.5353
   00   0.5611
   20   0.5678
   40   0.5678
   60   0.5678
   .80  0.5678
   !00  0.5678

2                                       0.98

0.2

1                                        0.84

0                                        0.5                   0


-1                                       0.16


-2                                       0.02

-3
  -3    -2    -1     0     1     2

3

0.001

                                          1
0Thy~~~~0.40.60.8
                    Recall


              Fallout-Recall Curve


0         0.05       0.1      0.15       0.2
                 Fallout x 100

~ults - CITRI, Royal Melbourne Institute of Technology

                     Summary Statistics         ]
          Run Number          citril-full, automatic 1
          Num of Queries                     50
          Total number of documents over all queries
          Retrieved:                      50000
          Relevant:                       10785
          Rel~ret:                         7439

  ecall Level_Averages
  Recall       Precision
   0.00         0.8023
   0.10         0.5298
   0.20         0.4570
   0.30         0.4027
   0.40         0.3482
   0.50         0.2951
   0.60         0.2271
   0.70         0.1514
   0.80         0.0887
   0.90         0.0379
   1.00         0.0000
  ~ge precision over all
  ~nt docs
  nterpolated   0.2804

  Level Averages
       Recall
  0~   0.3934
  0    0.5301
  0    0.6112
  0    0.6630
  0    0.6972
  0    0.7158
  D    0.7192
  0    0.7192
  0    0.7192
  0    0.7192
  0    0.7192

                               Document Level Averages
                                              Precision
                             At S docs         0.5680
                             At 10 docs        0.5320
                             At 15 docs        0.5173
                             At 20 docs        0.5140
                             At 30 docs        0.4913
                             At 100 docs       0.4224
                             At 200 docs       0.3541
                             At 500 docs       0.2340
                             At 1000 docs      0.1488
                             R-Precision (precision after R
                             docs  retrieved (where R  is
                             the number of relevant docu-
                             ments))
                             Exact             0.3359

                          NORMAL DEVIATE - Fallout-Recall

                        ol                 084  0.98
                                                                         0.4~


                                                                         0.2

                                                                  Recall

                        2                                     0.98

                                                                                         Recall-Precision Curve
                                                                          1


                                                                         0.8


                                                                         0.6
                                                                  Precision
                                                                         0.4


                                                                         0.2

                                                                                      I        I        I       I
                                                                          0
                                                                            0      0.2       0.4      0.6      0.8
                                                                                                 Recall


                                                                                          Fallout-Recall Curve

                                                                          1


                                                                         0.8


                                                                         0.6

                        1                                      0.84

                        0                                      0.5                   0

                                                                                       0         0.05         0.1        0.15        0.2
                        -1                                     0.16                                      Fallout x 100


                        -2                                     0.02

                        -3                                     0.001
                          -3    -2    -1    0     1     2    3

ults - CITRI, Royal Melbourne Institute of Technology

        L            Summary Statistics
         Run Number          citri2-full, automatic
         Num of Queries                      50
         Total number of documents over all queries
         Retrieved:                       50000
         Relevant:                        10785
         Rel~ret:                          1495

  ecall Level Averages
  Recall      Precision
  0.00         0.1856
  0.10         0.5335
  0.20         0.4611
  0.30         0.4086
  0.40         0.3526
  0.50         0.2932
  0.60         0.2306
  0.70         0.1613
  0.80         0.1053
  0.90         0.0510
  1.00         0.0097
  rage precision over all
  ~ant docs
  interpolated 0.2814

  it Level Averages
  Ft   Recall
  Fo   0.4050
  20   0.5313           2
  40   0.6116
  60   0.6684
  80   0.7040
  00   0.1216           0
  20   0.1291           -1
  40   0.1291
  60   0.1291           -2
  80   0.1291
  00   0.1291           -3

  Document Level Averages
                 Precision
At 5 docs         0.5600
At 10 docs        0.5280
At 15 docs        0.5133
At 20 docs        0.5000
At 30 docs        0.4900
At 100 docs       0.4354
At 200 docs       0.3606
At 500 docs       0.2358
At 1000 docs      0.1499
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))

        1


      0.8<


      0.6
Precision
      0.4


      0.2 -


        0

   Exact            0.3388

NORMAL DEVIATE - Fallout-Recall
                                         Recall

 1


0.8


0.6

                        0.84  0.98 0                      0.4~


                                      0.98
                                                          0.2
                                      0.84

                                      0.5                  0


                                      0.16


                                      0.02

       I                  I     I     0.001

-3    -2    -1     0      1     2   3

Recall-Precision Curve


              I       I

0       0.2     0.4      0.6     0.8
                   Recall


             Fallout-Recall Curve


0         0.05      0.1        0.15      0.2
                Fallout x 100

~`11ts - GE Researeh and Development Center


                    Summary Statistics           j
        Run Number         gecrd2-full, automatic 1                                         Recall-Precision Curve
                                                                              1
       L Num of Queries                     50 J
       r Total number of documents over all queries 1
        Retrieved:                       50000 1                             0.8 -
        Relevant:                        10785
        Rel~ret:                          6080
                                                                             0.6

ecall Level Averages
Recall       Precision
 0.00         0.7047
 0.10         0.4433
 0.20         0.3753
 0.30         0.3112
 0.40         0.2503
 0.50         0.2049
 0.60         0.1496
 0.70         0.0999
 0.80         0.0690
 0.90         0.0381
 1.00         0.0088
~ge precision over all
~nt docs
nterpolated   0.2183  1

  Document Level Averages
                 Precision
At 5 docs         0.4880
At 10 docs        0.4760
At 15 docs        0.4653
At 20 docs        0.4510
At 30 docs        0.4253
At 100 docs       0.3662
At 200 docs       0.2896
At 500 docs       0.1876
At 1000 docs      0.1216
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Exact             0.2816

Precision
       0.4


       0.2


        0


        1


       0.8

0       0.2     0.4      0.6     0.8


                   Recall


              Fallout-Recall Curve

 I           I     I

 Level Averages         NORMAL DEVIATE - Fallout-Recall
                                                    Recall
     Recall
 D   0.3221
     0.4269
     0.4957
     0.5489
     0.5850
 )   0.5968
 )   0.6014
 )   0.6014
 )   0.6014
 )   0.6014
 )   0.6014

                                                                         0.6

                        101  0.02        5   0.84  0.98  0
                                                                             0.4


                                                                             0.2 -

                           2                                        0.98

                           1                                        0.84

                           0                                        0.5                     0

                                                                                              0          0.05         0.1         0.15          0.2
                          -1                                        0.16                                         Fallout x 100


                          -2                                        0.02

                          -3

                            -3    -2    -1     0      1     2     3

                                                                    0.001

ults - Verity, Inc.


                      Summary Statistics
                                                                                            Recall-Precision Curve
         Run Number           TOPIC2-fuII1 manual
                                                                               1
         Num of Queries                       50
         Total number of documents over all queries
         Retrieved:                        31624                             0.8
         Relevant:                         10185
         Rel~ret:                           5614

mcall Level Averages
Recall        Precision
0.00           0.8491
0.10           0.5786
0.20           0.4943
0.30           0.3906
0.40           0.2774
0.50           0.2062
0.60           0.1221
0.10           0.0663
0.80           0.0353
0.90           0.0137
1.00           0.0000
ige precision over all   1
mt docs
nterpolated    0.2464    J

  Document Level Averages
                 Precision
At 5 docs         0.6280
At 10 docs        0.5880
At 15 docs        0.5827
At 20 docs        0.5680
At 30 docs        0.5407
At 100 docs       0.4624
At 200 docs       0.3347
At 500 docs       0.2002
At 1000 docs      0.1123
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Exact             0.3211

      0.6
Precision
      0.4


      0.2


        0


        1


      0.8

0       0.2     0.4      0.6     0.8


                   Recall


             Fallout-Recall Curve

 Level Averages
      Recall
)     0.3744
~     0.4689
~     0.5232
~     0.5446
~     0.5512
~     0.5512
~     0.5512
)     0.5512
~     0.5512
)     0.5512
~     0.5512

NORMAL DEVIATE - Fallout-Recall

      0.6
Recall

    101  0.02                    0.98  0 )~999                0.4


2                                         0.98
                                                              0.2
1                                         0.84

0                                         0.5                   0

                                                                  0         0.05         0.1        0.15         0.2
-1                                        0.16                                      Fallout x 100


-2                                        0.02

-3
  -3     -2    -1     0      1    2

3

0.001

tilts - Siemens Corporate Researchq Inc.

        [            Summary Statistics
         Run Number         siems2-fuIl, automatic
         Num of Queries                     50
         Total number of documents over all queries
         Retrieved:                      50000
         Relevant:                       10785
         Rel~ret:                         8068

`call Level_Averages
Recall         Precision
0.00           0.1964
0.10           0.5166
0.20           0.5267
0.30           0.4808
0.40           0.4244
0.50           0.3810
0.60           0.3088
0.70           0.2418
0.80           0.1416
0.90           0.0513
1.00           0.0000
ige precision over all
mt docs
riterpolated   0.3408

                             Document Level Averages
                                             Precision
                           At 5 docs         0.6000
                           At 10 docs        0.5860
                           At 15 docs        0.5733
                           At 20 docs        0.5600
                           At 30 docs        0.5447
                           At 100 docs       0.4690
                           At 200 docs       0.4030
                           At 500 docs       0.2606
                           At 1000 docs      0.1614
                           R-Precision (precision after R
                           docs retrieved  (where R  is
                           the number of relevant docu-
                           ments))
                           Fxact             0.3938

 Level Averages         NORMAL DEVIATE - Fallout-Recall
    Recall


 )  0.7722

 )  0.7163
 )  0.7763
 )  0.7763
 )  0.7763

                       Ol  0.02 0.16                                     0.4


                                                                         0.2

                         2                                       0.98


                         1                                       0.84


                         0                                       0.5                   0

                                                                                    Recall-Precision Curve
                                                                        1


                                                                      0.8


                                                                      0.6
                                                               Precision
                                                                      0.4


                                                                      0.2

                                                                                          I       I
                                                                        0


                                                                        1


                                                                      0.8


                                                                      0.6
                                                               Recall

                                                                         0       0.2     0.4     0.6     0.8
                                                                                            Recall


                                                                                      Fallout-Recall Curve
                                                                                    I         I


                                                                                    I         I

                                                                                         0          0.05        0.1         0.15         0.2
                        -1                                       0.16                                      Fallout x 100


                        -2                                       0.02

                        -3

                     -3   -2    -1    0         2   3

                                                                0.001

Its - Siemens Corporate Research, Inc.

       L           Summary Statistics         ]
        Run Number        siems3-full, automatic ]
        Num of Queries                     50
        Total number of documents over all queries
        Retrieved:                      50000
        Relevant:                       10785
        Rel~ret:                         8065

all Level Averages
ecall        Precision
).00          0.8164
).10          0.5849
).20          0.5262
).30          0.4774
).40          0.4262
).50          0.3771
).60          0.3111
).70          0.2338
).80          0.1362
).90          0.0513
1.00          0.0000
r~e precision over all
it docs
~erpolated    0.3397

evel Averages
     Recall
     0.4714
     0.5999
     0.6680
     0.7207
     0.7543
     0.7720
     0.7757
     0.7757
     0.7757
     0.7757
     0.7757

                             Document Level Averages
                                             Precision
                           At 5 docs          0.5960
                           At 10 docs         0.5920
                           At 15 docs         0.5773
                           At 20 docs         0.5630
                           At 30 docs         0.5453
                           At 100 docs        0.4692
                           At 200 docs        0.3979
                           At 500 docs        0.2598
                           At 1000 docs       0.1613
                           R-Precision (precision after R
                           docs retrieved  (where R  is
                           the number of relevant docu-
                           ments))

                                                                        1


                                                                       0.8


                                                                       0.6
                                                                Precision
                                                                       0.4


                                                                       0.2


                                                                        0

                           Exact            0.3931

                        NORMAL DEVIATE - Fallout-Recall
                                                                Recall

                                                                        1


                                                                       0.8


                                                                       0.6

                           Ol   0.02  0.16    5   0.84  0.98  0                       0.4


                        2                                        0.98
                                                                                      0.2
                        1                                        0.84

                        0                                        0.5                    0

                                                                                       Recall-Precision Curve


                                                                                    I        I       I


                                                                          0      0.2       0.4      0.6     0.8
                                                                                               Recall


                                                                                        Fallout-Recall Curve
                                                                                      I          I


                                                                                      I          I

                                                                                            0          0.05          0.1         0.15         0.2
                        -1                                         0.16                                         Fallout x 100


                        -2                                         0.02

                        -3        I     I            I      I      0.001
                          -3     -2    -1     0      1      2    3

ilts - Queens College, CUNY

                   Summary Statistics
                                             J
         Run Number        pircs3-full, automatic
         Num of Queries                    50
         Total number of documents over all queries
         Retrieved:                     50000
         Relevant:                      10185
         Rel~ret:                        1456

call Level Averages
~ecall       Precision
0.00          0.7538
0.10          0.5529
0.20          0.4127
0.30          0.4088
0.40          0.3531
0.50          0.2994
0.60          0.2311
0.10          0.1677
0.80          0.1094
0.90          0.0551
1.00          0.0145
~e precision over all
~t docs
terpolated    0.2949

 evel Averages
      Recall
      0.4118
      0.5411
      0.6092
      0.6608
      0.6990
      0.7245
      0.1288
      0.1288
      0.1288
      0.1288
      0.7288

                              Document Level Averages
                           ________________    Precision
                            At 5 docs           0.5640
                            At 10 docs          0.5560
                            At 15 docs          0.5467
                            At 20 docs          0.5460
                            At 30 docs          0.5200
                            At 100 docs         0.4466
                            At 200 docs         0.3660
                            At 500 docs         0.2342
                            At 1000 docs        0.1491
                            R-Precision (precision after R
                            docs  retrieved  (where R  is
                            the number of relevant docu-
                            ments))

                                    Exact                0.3437

                                NORMAL DEVIATE - Fallout-Recall

                              Ol  0.02  0.16         0.84   0.98   0
                                                                     ~999

                          2                                          0.98

                                                                                        Recall-Precision Curve
                                                                         1


                                                                        0.8


                                                                        0.6
                                                                 Precision
                                                                        0.4


                                                                        0.2

                                                                                     I        I        I        I
                                                                         0


                                                                 Recall

                                                                         1


                                                                        0.8


                                                                        0.6


                                                                        0.4


                                                                        0.2

                          1                                         0.84

                          0                                         0.5                    0

                                                                           0      0.2       0.4     0.6     0.8       1
                                                                                               Recall


                                                                                         Fallout-Recall Curve


                                                                                       I         I

                                                                                            0           0.05         0.1         0.15         0.2
                         -1                                         0.16                                        Fallout x 100


                         -2                                         0.02

                         -3                                         0.001
                            -3    -2    -1     0     1      2     3

`sults - Queens College, CUNY

                      Summary Statistics         )
          ~ Run Number        pircs4-fulI, automatic 1
           Num of Queries                     50
           Total number of documents over all queries
           Retrieved:                      50000
           Relevant:                       10785
           Rel~ret:                         7464

Recall Level_Averages
 Recall         Precision
    0.00         0.7621
    0.10         0.5569
    0.20         0.4773
    0.30         0.4108
    0.40         0.3554
    0~50         0.3017
    0.60         0.2343
    0.70         0.1702
    0.80         0.1128
    0.90         0.0602
    1.00         0.0154
`rage precision over all
.`vant docs
i-interpolated   0.2981

ut Level Averages
~t       Recall
)00      0.4140
)20      0.5449            2
)40      0.6128
)60      0.6637            1
)80      0.7008
                           0
100      0.7244
120      0.7293           -1
[40      0.7293
[60      0.7293           -2
[80      0.7293
                          -3
~00      0.7293

  Document Level Averages
                 Precision
At 5 docs         0.5760
At 10 docs        0.5760
At 15 docs        0.5507
At 20 docs        0.5460
At 30 docs        0.5200
At 100 docs       0.4494
At 200 docs       0.3680
At 500 docs       0.2348
At 1000 docs      0.1493
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))       _______________

        1


      0.8


      0.6
Precision
      0.4


      0.2


        0

       ~t03467

    NORM ~L DEVIATE - Fallout-Recall
                                                      Recall
  101  0.02                    0.98   0


-3     -2    -1     0     1     2      3

 1


0.8


0.6


0.4~

0.98
                      0.2
0.84

0.5                     0

            Recall-Precision Curve


         I        I        I       I


0       0.2     0.4       0.6     0.8
                   Recall


             Fallout-Recall Curve
            I          I         I


            I                    I

                          0          0.05         0.1         0.15         0.2
0.16                                         Fallout x 100


0.02

0.001

results - VPI ~ SU


                    Summary Statistics

                                              ]

          Run Number          VTcms2-full manual                                            Recall-Precision Curve
          Num of Queries                      50                              1

          Total number of documents over all queries
          Retrieved:                       50000                             0.8
          Relevant:                        10185
          Rel~ret:                          1616

 Recall Level_Averages
   Recall       Precision
    0.00         0.8693
    0.10         0.6115
    0.20         0.5409
    0.30         0.4591
    0.40         0.3834
    0.50         0.3172
    0.60         0.2451
    0.70         0.1659
    0.80         0.0909
    0.90         0.0364
    1.00         0.000
 ~rage precision over a
 vant docs       0.320:11
 i-interpolated

                                 Document Level Averages
                                                 Precision
                               At 5 docs          0.6800
                               At 10 docs         0.6280
                               At 15 docs         0.6240
                               At 20 docs         0.6130
                               At 30 docs         0.5820
                               At 100 docs        0.4790
                               At 200 docs        0.3835
                               At 500 docs        0.2430
                               At 1000 docs       0.1535
                               R-Precision (precision after R
                               docs  retrieved (where R  is
                               the number of relevant docu-
                               ments))
                               Exact              0.3708

                                                                              0.6
 ~t Level Averages             NORMAL DEVIATE - Fallout-Recall
 ut      Recall
 i00     0.4410
 `20     0.5666
 40      0.6420
 60      0.6877
 80      0.7179
 00      0.7347
 20      0.7414
 40      0.7414
 60      0.7414
 80      0.7414
 o0      0.7414

                                                                    Recall

                          3:1                            0.98   a )~999
                                                                                      0.4
                          2                                       0.98

                                                                                      0.2
                          1                                       0.84

                          0                                       0.5
                                                                                        0

                                                                           0.6
                                                                    Precision
                                                                           0.4


                                                                           0.2  -

                                                                                        I        I       I
                                                                            0


                                                                            1


                                                                           0.8

                                                                              0      0.2       0.4      0.6     0.8      1
                                                                                                  Recall


                                                                                            Fallout-Recall Curve


                                                                                          I          I          I

                                                                               0        0.05       0.1       0.15       0.2
                       -1                                 0.16
                                                                                               Fallout x 100
                       -2                                 0.02

                       -3                                 0.001
                         -3    -2   -1   0     1    2    3

`ults - Conquest Software, Inc.


    [           Summary Statistics                                          Recall-Precision Curve
       Run Number        CnQstl-full, manual                      1       I     II
       Num of Queries                 50

           ~otal num ~ 0f ~ over all queries
            Retrieved:                         50000
            Relevant:                          10185
            Rel~ret:6494I

                                      Document Level Averages
    ~ecall Level Averages
                                                     Precision
    Recall       Precision
                                   At 5 docs          0.4480
    0.00         0.6831
                                   At 10 docs         0.4360
    0.10         0.4552
                                   At 15 docs         0.4427
    0.20         0.4010
                                   At 20 docs         0.4310
    0.30         0.3538
                                   At 30 docs         0.4393
    0.40         0.2951
                                   At 100 docs        0.3954
    0.50         0.2361
                                   At 200 docs        0.2997
    0.60         0.1807
                                   At 500 docs        0.1962
    0.70         0.1147
                                   At 1000 docs       0.1299
    0.80         0.0725
    0.90         0.0454            R-Precision (p~recisionafterRI
    1.00         0.0054            docs  retrieved (where  R  is
                                   the number of relevant docu-
    `rage precision over all
                                   ments))            -1
    `vant docs
    ~-interpolated 0.2343          Exact              0.2869

                                NORMAL DEVIATE - Fallout-Recall
    ut Level Averages
    )ut  Recall             .0 Ol 0.02  0.16  0.5  0.84  0.98 0.9~999
    )00  0.3320
    ~20  0.4506            2                                     0.98
    040  0.5406
                                                                 0.84
    060  0.5993            1
    080  0.6400                                                  0.5
                           0
    100  0.6696
    120  0.6763          -1                                      0.16
    140  0.6763
                                                                 0.02
    160  0.6763          -2
    180  0.6763          -3                                      0.001
    p200 0.6763             3     -2    -1    0     1    2      3

      0.8  -


      0.6
Precision
      0.4


      0.2


        0
          0      0.2     0.4     0.6     0.8       1
                            Recall

                      Fallout-Recall Curve


        1


      0.8


      0.6
Recall
      0.4


      0.2


        0
          0        0.05      0.1       0.15       0.2
                         Fallout x 100

slilts - Conquest Software, Inc.

                       Summary Statistics        ]
         Run Number            CnQst2-full, manual 1
         Num of Queries                       50 J
         Total number of documents over all queries 1
         Retrieved:                         50000 1
         Relevant:                          10785
         Rel~ret:                           6788

~ecall Level Averages
  Recall      Precision
  0.00         0.7202
  0.10         0.5005
  0.20         0.4411
  0.30         0.3787
  0.40         0.3242
  0.50         0.2681
  0.60         0.2010
  0.70         0.1373
  0.80         0.0947
  0.90         0.0465
  1.00         0.0045
~age precision over all 1
,ant docs
interpolated   0.2633

t Level Averages
it     Recall
Po     0.3737
?0     0.4851
~      0.5676
)0     0.6290
30     0.6653
)0     0.6931
~0     0.6960
~      0.6960
)0     0.6960
10     0.6960
~0     0.6960

                                Document Level Averages
                                               Precision
                              At 5 docs         0.5080
                              At 10 docs        0.4880
                              At 15 docs        0.4920
                              At 20 docs        0.4960
                              At 30 docs        0.4793
                              At 100 docs       0.4178
                              At 200 docs       0.3256
                              At 500 docs       0.2102
                              At 1000 docs      0.1358
                              R-Precision (precision after R
                              docs  retrieved (where R  is
                              the number of relevant docu-
                              ments))

                                    Exact              0.3140

                                NORMAL DEVIATE - Fallout-Recall

                          g)  Ol  0.02


                          2                                       0.98

                                                                                         Recall-Precision Curve
                                                                           1


                                                                         0.8


                                                                         0.6
                                                                   Precision
                                                                         0.4


                                                                         0.2


                                                                           0          I        I       I


                                                                   Recall

                                                                           1


                                                                         0.8


                                                                         0.6


                                                                         0.4


                                                                         0.2

                          1                                      0.84


                          0                                      0.5                   0

                                                                            0       0.2      0.4      0.6     0.8
                                                                                                Recall


                                                                                          Fallout-Recall Curve
                                                                                        I          I         I


                                                                                        I          I         I

                                                                                          0          0.05         0.1        0.15         0.2
                         -1                                       0.16                                       Fallout x 100


                         -2                                       0.02

                         -3                                       0.001
                           -3    -2    -1     0      1     2     3

suitS - PRC, Inc.

                       Summary Statistics
          Run Number             prceol-full, manual 1
                                                                                    1
         LTNum of Queries                       50
             otal number of documents over all queries
          Retrieved:                         50000                                0.8
          Relevant:                          10185
          Rel~ret:                            3323                                0.6~

~ecall Level Averages               Document Level Averages                Precision
 Recall         Precision                          Precision                      0.4
  0.00           0.5963           At 5 docs         0.4120
  0.10           0.3270           At 10 docs        0.4000
                                                                                  0.2
  0.20           0.2306           At 15 docs        0.3920
  0.30           0.1430           At 20 docs        0.3140
  0.40           0.0866           At 30 docs        0.3521                          0
  0.50           0.0425           At 100 docs       0.2122
  0.60           0.0323           At 200 docs       0.1914
  0.70           0.0246           At 500 docs       0.1090
  0.80           0.0209           At 1000 docs      0.0665
  0.90           0.0131           R-Precision (precision after R                    1
  1.00           0 0041           docs  retrieved (where R  is
rage precision over all           the number of relevant docu-
~ant docs                         ments))                                         0.8
-interpolated    0 1120           Exact             0.1809
                                                                                  0.6
it Level Averages
LIT     Recall
00      0.2103
20      0.2531
40      0.2811
60      0.3006
80      0.3166
00      0.3294
20      0.3335
40      0.3335
60      0.3335
80      0.3335
00      0.3335

NORMAL DEVIATE - ~11out-Reca11

Recall

1)                         0.84  0.98  0                       0.4

2


1


0


-1

-2

          I     I
-3

0.98

-3     -2    -1     0     1     2     3

0.2~

0.84

0.5                     0


0.16


0.02

0.001

            Recall-Precision Curve


0       0.2      0.4     0.6     0.8
                   Recall


             Fallout-Recall Curve
           I          I         I


                      I         I

0         0.05       0.1       0.15       0.2
                 Fallout x 100

suits - PRC, Inc.


                     Summary Statistics
        Run Number            prceo2-full, manual                                        Recall-Precision Curve
        Num of Queries                      50                              1
       L
        lotal number of documents over all queries
        Retrieved:                       50000                             0.8
        Relevant:                        10785
        Rel~ret:                          3561
       F
                                                                           0.6

~ecall Level_Averages
 Recall        Precision
 0.00          0.4768
 0.10          0.2613
 0.20          0.1807
 0.30          0.1202
 0.40          0.0875
 0.50          0.0504
 0.60          0.0328
 0.70          0.0169
 0.80          0.0142
 0.90          0.0077
 1.00          0.0031
age precision over all
ant docs
nterpolated    0.0904

                                Document Level Averages
                                               Precision
                              At 5 docs         0.2120
                              At 10 docs        0.2480
                              At 15 docs        0.2707
                              At 20 docs        0.2760
                              At 30 docs        0.2753
                              At 100 docs       0.2418
                              At 200 docs       0.1756
                              At 500 docs       0.1086
                              At 1000 docs      0.0712
                              R-Precision (precision after R
                              docs  retrieved (where R is
                              the number of relevant docu-
                              ments))
                              Exact             0.1703

                                                                Precision
                                                                       0.4


                                                                       0.2


                                                                        0

  Level Averages           NORMAL DEVIATE - ~11out-Reca11
  t    Recall                                                      Recall
                          101 0.02 0.16  5   a84  0.98 0
  0    0.1861
  0    0.2402
  0    0.2860
  0    0.3141
  0    0.3387
  0    0.3564
  0    0.3623
  0    0.3623
  0    0.3623
  0    0.3623
  0    0.3623

                            2                                          0.98

                                                                        1


                                                                       0.8


                                                                       0.6


                                                                       0.4


                                                                       0.21

                             1                                          0.84


                             0                                          0.5                     0

                                                                         0       0.2      0.4     0.6     0.8
                                                                                             Recall


                                                                                       Fallout-Recall Curve
                                                                                     I         I


                                                                                     I         I

                                                                             0        0.05       0.1       0.15       0.2
                      -1                                 0.16
                                                                                             Fallout x 100
                      -2                                 0.02

                      -3                                 0.001
                        -3   -2    -1   0    1    2    3

ults - Mead Data Central, Inc.

                     Summary Statistics
          Run Number        UREKA2-full, manual 1
          Num of Queries                     50
          Total number of documents over all queries
          Retrieved:                      50042
          Relevant:                       10785
          Rel~ret:                         2690

  ecall Level Averages
   Recall     Precision
  0.00         0.2740
  0.10         0.0977
  0.20         0.0774
   0.30        0.0701
  0.40         0.0617
   0.50        0.0467
   0.60        0.0338
   0.70        0.0242
   0.80        0.0187
   0.90        0.0000
   1.00        0.0000
  rage precisior~ over all
  `-~rtteidp0:l:ted0.05021


  it Level Averages
  ut    Recall
  00    0.0733
  20    0.1100          2
  40    0.1403
  ~0    0.1713          1
  80    0.1970
                        0
  .00   0.2173
  .20   0.2271          -1
  .40   0.2271
  .60   0.2271          -2
  .80   0.2271
                        -3
  ?00   0.2271

     Document Level Averages
                    Precision
   At 5 docs        0.1360
   At 10 docs       0.1140
   At 15 docs       0.1067
   At 20 docs       0.1100
   At 30 docs       0.1033
   At 100 docs      0.0912
   At 200 docs      0.0831
   At 500 docs      0.0664
   At 1000 docs     0.0538
   R-Precision (precision after R
   docs  retrieved (where R      is
   the number of relevant docu-
   ments))
   Exact            0.0793

NORMAL DEVIATE - Fallout-Recall              Recall

              5

 C          ).    .--  .   .9


     I                 I

101  0.02

0.84

0.98  0.

0.98


0.84

0.16

-3   -2   -1    0          2   3

                       Recall-Precision Curve
        1


      0.8


      0.6
Precision
      0.4


      0.2

                   I        I
        0
         0       0.2      0.4      0.6     0.8      1
                              Recall


                        Fallout-Recall Curve
        1            I          I


      0.8


      0.6


      0.4


      0.2

0.5                    0
                         0           0.05         0.1         0.15         0.2
0.16                                         Fallout x 100


0.02


0.001

~su1ts - Mead Data Central, Inc.


          Run Num be
                                                                                            Recall-Precision Curve
                                              50                               1
         [
          Total numb
          Retrieved:
                                                                             0.8
          Relevant:                        10785
          Rel~ret:3282j
         t
~ecall Level Averages
  Recall        Precision
   0.00          0.4228
   0.10          0.1880
   0.20          0.1385
   0.30          0.1045
   0.40          0.0840
   0.50          0.0643
   0.60          0.0445
   0.70          0.0365
   0.80          0.0183
   0.90          0.0000
   1.00          0.0000
  age precision over all
  ant docs
  interpolated   0.0819

   Level Averages
  t     Recall
  0     0.1242
  0     0.1144
  0     0.2072
  0     0.2337
  0     0.2571
  0     0.2740
  0     0.2805
  0     0.2805
  D     0.2805
  ~     0.2805
  ~     0.2805

                             31

                             2


                             1


                             0

                                 Document Level Averages
                                                  Precision
                               At 5 docs           0.2520
                               At 10 docs          0.2320
                               At 15 docs          0.2267
                               At 20 docs          0.2200
                               At 30 docs          0.2073
                               At 100 docs         0.1688
                               At 200 docs         0.1383
                               At 500 docs         0.0954
                               At 1000 docs        0.0656
                               R-Precision (precision after R
                               docs  retrieved  (where  R  is
                               the number of relevant docu-
                               m ents))
                               Exact               0.1231

                            NORMAL DEVIATE - Fallout-Recall

                         )01  0.02 0.16   0. 5  0.84  0.98 0


                                             7

                                                                          0.6

                                                                   Precision
                                                                          0.4

                                                                           0.2


                                                                             0


                                                                             1


                                                                           0.8


                                                                           0.6

                                                         ~999              0.4

                                                         0.98

                                                                           0.2
                                                         0.84


                                                         0.5                 0

                                                                   Recall

                                                                             0       0.2      0.4     0.6      0.8
                                                                                                 Recall


                                                                                           Fallout-Recall Curve
                                                                                         I          I


                                                                                         I                    I

                            -1                                                                 0           0.05         0.1         0.15         0.2
                                                                      0.16
                                                                                                                   Fallout x 100
                            -2                                        0.02

                            -3                                        0.001
                               -3    -2    -1     0     1     2      3

ults - University of Cahfornia, Los An~e1es


                    Summary Statistics
       Run Number      uclaal-category B, automatic                                       Recall-Precision Curve
       Num of Queries                        50                             1
         Total number of documents over all queries
       Retrieved:                         50000                           0.8'
       Relevant:                           3929
       Reljet:                             3204

ecall Level Averages
Recall       Precision
0.00          0.8126
  0.10        0.6075
0.20          0.5292
0.30          0.4385
  0.40        0.3861
  0.50        0.3210
  0.60        0.2767
  0.10        0.2284
  0.80        0.1646
  0.90        0.0938
 1.00         0.0163
ige precision over all
mt docs
nterpolated   0.3345

  Level Averages
       Recall
()     0.3638
D      0.4872
D      0.5666
D      0.6114
~      0.6617
~      0.6944
D      0.7136
~      0.7335
~      0.1451
~      0.7604
~      0.1748

                               Document Level Averages
                                              Precision
                             At 5 docs         0.5840
                             At 10 docs        0.5380
                             At 15 docs        0.4973
                             At 20 docs        0.4660
                             At 30 docs        0.4333
                             At 100 docs       0.3098
                             At 200 docs       0.2216
                             At 500 docs       0.1140
                             At 1000 docs      0.0641
                             R-Precision (precision after k
                             docs  retrieved (where R  is
                             the number of relevant docu-
                             ments))

                                                                        0.6
                                                                  Precision
                                                                        0.4


                                                                        0.2


                                                                          0


                                                                          1


                                                                        0.8

                                                                           0       0.2     0.4      0.6     0.8


                                                                                              Recall


                                                                                         Fallout-Recall Curve

                                                            I     I     I

                                    Exact               0.3629
                                                                                         0.6
                                NORMAL DEVIATE - Fallout-Recall

                              Ol  0.02                     0.98  0 )~999                 0.4


                          2                                         0.98

                                                                                         0.2
                          1                                         0.84

                          0                                         0.5                    0

                                                                                             0          0.05          0.1        0.15          0.2
                          -1                                        0.16                                         Fallout x 100


                          -2                                        0.02

                          -3                                        0.001
                            -3     -2    -1     0     1     2      3

                                                                  Recall

suits - University of California~ Los An~e1es


      f             Summary Statistics
       Run Number      uclaa2-category B1 automatic                                     Recall-Precision Curve
       Num of Queries                        50                            1
         Total number of documents over all queries
       Retrieved:                         50000                           0.8
       Relevant:                           3929
       Reljet:                             3059
                                                                          0.6 -

~ecall Level Averages
 Recall        Precision
 0.00           0.7464
 0.10           0.5880
 0.20           0.5024
 0.30           0.4058
 0.40           0.3429
 0.50           0.2698
 0.60           0.2168
 0.10           0.1803
 0.80           0.1209
 0.90           0.0695
 1.00           0.0098
*age precision over all
ant docs
interpolated    0.2951 ]

  Document Level Averages
                  Precision
At 5 docs          0.5440
At 10 docs         0.5180
At 15 docs         0.4920
At 20 docs         0.4650
At 30 docs         0.4201
At 100 docs        0.2160
At 200 docs        0.1958
At 500 docs        0.1051
At 1000 docs       0.0612
RLprecision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Exact              0.3289

Precision
      0.4


      0.2


        0


        1


      0.8

0       0.2     0.4     0.6     0.8


                   Recall


             Fallout-Recall Curve

 I          I

t Level Averages
~t    Recall
Fo    0.3245
~0    0.4324
~     0.4885
~     0.5313
~0    0.5838
~     0.6135
0     0.6339
~     0.6561
~     0.6698
0     0.6852
0     0.7066

                                                  0.6
   NORMAL DEVIATE - ~Iout-RecaI1

101  0.02           0.84   0.98
                                                  0.4


                                                  0.2

Recall

2                                     0.98

1                                      0.84


0                                      0.5                  0

                                                              0         0.05        0.1        0.15        0.2
-1                                     0.16
                                                                                Fallout x 100
-2                                     0.02


-3                                     0.001
  -3    -2    -1    0     1    2     3

Lilts - University of California, Los Angeles


                   Summary Statistics
                                                                                             Recall-Precision Curve
un Number       uclafi-category B, feedback, frozen evaluation                  1
um of Queries                                      50
          Total number of documents over all queries
                                                                              0.8<
etrieved:                                       50473
elevant:                                         3929
cljet:                                           3047 J                       0.6

  ecall Level Averages
  Recall      Precision
  0.00         0.8125
  0.10         0.5951
  0.20         0.5218
  0.30         0.4113
  0.40         0.3484
  0.50         0.2813
  0.60         0.2289
  0.10         0.1915
  0.80         0.1351
  0.90         0.0764
  1.00         0.0105
  age precision over all
  ant docs
  interpolated 0.3091  1

  L Level Averages
  it   Recall
  Fo   0.3425
  ?0   0.4483           2
  ~    0.5041
  ,0   0.5489           1
  30   0.5898
                        0
  )0   0.6216
  ~0   0.6431           -1
  ~    0.6653
  50   0.6864           -2
  BO   0.1030
                        -3
  00   0.1112

     Document Level Averages                Precision
                    Precision                      0.4
   At 5 docs        0.5880
   At 10 docs       0.5220
                                                   0.2
   At 15 docs       0.4893
   At 20 docs       0.4640
   At 30 docs       0.4360                          0
   At 100 docs      0.2884
   At 200 docs      0.1992
   At 500 docs      0.1058
   At 1000 docs     0.0609
   R-Precision (precision after R                   1
   docs  retrieved (where R  is
   the number of relevant docu-
   ments))                                         0.8
   Exact            0.3459

NORMAL DEVIATE - Fa11out-R~ca11
                                            Recall

0       0.2     0.4     0.6     08


                   Recall


             Fallout-Recall Curve

 I     I

0.6  -

                               0.98  0                       0.4


                                        0.98
                                                             0.2
                                        0.84

                                        0.5                    0
                                                                 0          0.05         0.1         0.15         0.2
                                        0.16                                        Fallout x 100


                                        0.02


                                        0.001

-3     -2    -1     0     1     2     3

suits - University of Illinois at Chicago


                    Summary Statistics
        Run Number           uicah-full, automatic 1                                        Recall-Precision Curve
        Num of Queries                      50 J                               1
        Total number of documents over all queries 1
        Retrieved:                       45108 1                             0.8
        Relevant:                        10785
        Rel~ret:                          4421
                                                                             0.6

~ecall Level Averages
Recall       Precision
 0.00         0.5665
 0.10         0.3272
 0.20         0.2527
 0.30         0.1937
 0.40         0.1413
 0.50         0.0712
 0.60         0.0393
 0.70         0.0069
 0.80         0.0000
 0.90         0.0000
 1.00         0.0000
age precision over all  1
ant docs
interpola~~.1200

 Level Averages
t
~0
0
0
0
0
0
0
0
0
0
0

Recall
0.2095
0.3057
0.3532
0.3860
0.4125
0.4359
0.4467
0.4467
0.4467
0.4467
0.4467

     Document Level Averages
                    Precision
   At 5 docs         0.3280
   At 10 docs        0.3340
   At 15 docs        0.3280
   At 20 docs        0.3380
   At 30 docs        0.3233
   At 100 docs       0.2784
   At 200 docs       0.2107
   At 500 docs       0.1392
   At 1000 docs      0.0884
   R-Precision (precision after R
   docs  retrieved (where  R  is
   the number of relevant docu-
   ments))
   Exact         I   0.1950

NORMAL DEVIATE - Fallout-Recall

Precision
       0.4


       0.2


         0


         1


       0.8


       0.6
Recall

)01  0.02             0.84 0.98  0
                                                     0.4 -


                                                     0.2

2                                         0.98


1                                         0.84

0                                         0.5                     0

                                                                    0          0.05          0.1         0.15         0.2
-1                                        0.16                                          Fallout x 100


-2                                        0.02

-3

0.001

-3     -2    -1     0     1           3

0       0.2     0.4      0.6     0.8


                   Recall


             Fallout-Recall Curve

 I    I

A

results - City University, London

             Summary Statistic          s]
Run Number           cityri-full, automatic
                                                                     1
Num of Queries                       50 J
Total number of documents over all queries
Retrieved:                        50000                             0.8<
Relevant:                         10489
Rel~ret:                           6801                             0.6

Recall-Precision Curve

Recall Level Averages
    Recall     Precision
    0.00        0.1888
    0.10        0.6028
    0.20        0.5101
    0.30        0.4380
    0.40        0.3851
    0.50        0.3070
    0.60        0.2494
    0.70        0.1826
    0.80        0.1269
    0.90        0.0606
    1.00        0.0048
~rage precision over all
vant docs       0.3149
i-interpolated

ut Level Averages
~ut      Recall
)~       0.3610
)20      0.4444
)40      0.5009
)60      0.5433
)80      0.5791
L00      0.6074
[20      0.6294
[40      0.6476
[60      0.6636
[80      0.6784
~0       0.6918

  Document Level Averages
                 Precision
At 5 docs         0.6280
At 10 docs        0.5940
At 15 docs        0.5867
At 20 docs        0.5690
At 30 docs        0.5333
At 100 docs       0.4316
At 200 docs       0.3385
At 500 docs       0.2154
At 1000 docs      0.1360
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))

Precision
      0.4


      0.2


        0


        1


      0.8

         Exact            0.3607
                                                       0.6 -
     NORMAL DEVIATE - Fallout-Recall
                                                Recall
    ol 0.02           0.84  0.98  0.   999             0.4


2                                     0.98
                                                       0.2
1                                     0.84

0                                     0.5                0
                                                          0        0.05       0.1       0.15       0.2
-1                                    0.16                                Fallout x 100


-2                                    0.02

-3                                    0.001
  -3   -2   -1    0    1     2      3

0       0.2     0.4      0.6     0.8


                   Recall


             Fallout-Recall Curve

 I     I     I

results - City University, London

         L            Summary Statistics
           Run Number         cityr2-fulI, automatic
           Num of Queries                     50
           Total number of documents over all queries
           Retrieved:                      50000
           Relevant:                       10489
           Rel~ret:                         7134

Recall Level Averages
  Recall        Precision
  0.00           0.8313
  0.10           0.6341
  0.20           0.5491
  0.30           0.4990
  0.40           0.4356
  0.50           0.3618
  0.60           0.2908
  0.10           0.2305
  0.80           0.1551
  0.90           0.0900
  1.00           0.0085
rage precision over all
~ant docs
-interpolated    0.3562 J

it Level Averages
it      Recall
~       0.3949
~0      0.4731
~       0.5331
50      0.5722
30      0.6050
)0      0.6278
~0      0.6510
~       0.6696
,0      0.6851
30      0.7065
)0      0.7181

j


1
I

 1


0.8


0.6

        Document Level Averages                 Precision
      _______________   Precision                     0.4
      At 5 docs          0.6920
      At 10 docs         0.6500
      At 15 docs         0.6213                       0.2
      At 20 docs         0.6060
      At 30 docs         0.5613                         0
      At 100 docs        0.4494
      At 200 docs        0.3588
      At 500 docs        0.2261
      At 1000 docs       0.1427
      R- Precision (precision after R
                                                        1
      docs  retrieved  (where R    is
      the number of relevant docu-
      ments))                                         0.8
      Exact              0.3877
                                                      0.6
   NORMAL DEVIATE - Fallout-Recall
                                                Recall
101  0.a2 0.16  ( 5

0.84   0.98  0
                                0.4


                                0.2

2                                        0.98


1                                        0.84

0                                        0.5                  0


-1                                       0.16


-2                                       0 02

-3                                       0.001
  -3    -2    -1     0     1     2     3

             Recall-Precision Curve


0       0.2      0.4     0.6     0.8


                    Recall


              Fallout-Recall Curve


0         0.05       0.1       0.15       0.2
                 Fallout x 100

results - Rutgers University

                        Summary Statistics
            Run Number         rutcombx-full, manual
            Num of Queries                      50
            Total number of documents over all queries
            Retrieved:                       49983
            Relevant:                        10489
            Rel~ret:                          6244

Recall Level Averages
    Recall       Precision
    0.00          0.7739
    0.10          0.5601
    0.20          0.4537
    0.30          0.4042
    0.40          0.3458
    0.50          0.2860
    0.60          0.2074
    0.70          0.1708
    0.80          0.1180
    0.90          0.0668
    1.00          0.0132
erage precision over all   1
evant docs
n-interpolated    0.2905

)ut Level Averages
out      Recall
000      0.3188
020      0.3966
040      0.4374
060      0.4733
080      0.4980
100      0.5403
120      0.5587
140      0.5765
160      0.5927
~180     0.6044
`200     0.6169

  Document Level Averages
                 Precision
At 5 docs         0.6000
At 10 docs        0.5640
At 15 docs        0.5427
At 20 docs        0.5300
At 30 docs        0.5000
At 100 docs       0.3880
At 200 docs       0.3023
At 500 docs       0.1930
At 1000 docs      0.1249

        1


      0.8<


      0.6
Precision
      0.4


      0.2


        0

R-Precision (precision a+ter R                    1
              (where  R  is

docs  retrieved
the number of
ments))

relevant docu-

0.8


0.6

   Exact            0.3375

NORMAL DEVIATE - Fallout-Recall
                                         Recall

             Recall-Precision Curve


0       0.2     0.4      0.6     0.8


                   Recall


              Fallout-Recall Curve

                                      0.98  0.                         0.4


2                                              0.98
                                                                       0.2
1                                              0.84

0                                              0.5                       0
                                                                           0           0.05          0.1          0.15           0.2
-1                                             0.16                                             Fallout x 100


-2                                             0.02

-3         I                    I              0.001
   -3     -2     -1      0      1      2     3

results - Cornell University


                  Summary Statistics
                                                                                       Recall-Precision Curve
        Run Number         crnlRl-full, automatic 1
                                                                          1
        Num of Queries                    50

          Jotal number of documents over all queries
          Retrieved:                      50000                           0.8
          Relevant:                       10489
          Rel~ret:                        7164

Recall Level Averages
 Recall        Precision
  0.00          0.8268
  0.10          0.1060
  0.20          0.6282
  0.30          0.5636
  0.40          0.4900
  0.50          0.4246
  0.60          0.3416
  0.10          0.2483
  0.80          0.1543
  0.90          0.0615
  1.00          0.0062
.`rage precision over all
vant docs
-interpolated   0.3952

  Document Level Averages
                  Precision
At 5 docs          0.6800
At 10 docs         0.6180
At 15 docs         0.6640
At 20 docs         0.6560
At 30 docs         0.6241
At 100 docs        0.4950
At 200 docs        0.3914
At 500 docs        0.2526
At 1000 docs       0.1553
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Exact              0.4213

      0.6
Precision
      0.4


      0.2


        0


        1


      0.8

0       0.2     0.4     0.6     0.8


                   Recall


             Fallout-Recall Curve

 I          I          I          I

it Level Averages
~ut     Recall
~00     0.4126
`20     0.5260
`40     0.6041
`60     0.6485
`80     0.6786
00      0.1056
20      0.1241
40      0.1433
60      0.1588
80      0.1685
O0      0.1194

NORMAL DEVIATE - Fallout-Recall

      0.6
Recall

    101 0.02   0.16    5   0.84  0.98  0                      0.4


2                                         0.98
                                                              0.2
1                                         0.84

0                                         0.5                   0

                                                                 0          0.05         0.1        0.15         0.2
-1                                        0.16                                      Fallout x 100


-2                                        0.02

-3                                        0.001
  -3    -2     -1     0     1     2     3

suits - Cornell University


                    Summary Statistics
                                                                                       Recall-Precision Curve
       Run Number          crnlCl-full1 automatic 1
                                                                          1
       Num of Queries                      50
       Total number of documents over all queries
       Retrieved:                       50000                           0.8
       Relevant:                        10489
       Reljet:                           7808                           0.6

 call Level Averages
 Recall      Precision
 0.00         0.8355
 0.10         0.1202
 0.20         0.6516
 0.30         0.5816
 0.40         0.5142
 0.50         0.4518
 0.60         0.3542
 0.70         0.2519
 0.80         0.1547
 0.90         0.0677
 1.00         0.0062
 ige precision over all 1
 mt docs                j
 ~terpolated  0.4091    J

 Level Averages
        Recall
 )      0.4336
 )      0.5602
 ~      0.6110
 )      0.6592
 ~      0.6886
 ~      0.7144
 ~      0.7338
 ~      0.7489
 D      0.1588
 D      0.7681
 D      0.7763

  Document Level Averages             Precision
                  Precision                  0.4
At 5 docs          0.7240
At 10 docs         0.7000
                                             0.2
At 15 docs         0.6840
At 20 docs         0.6660                       0      0.2      0.4     0.6     0.81
At 30 docs         0.6240                      0
At 100 docs        0.5156
At 200 docs        0.4104                                          Recall
At 500 docs        0.2570
At 1000 docs       0.1562
                                                             Fallout-Recall Curve
R-Precision (precision afjer R
                                               1
docs  retrieved (where R  is
               relevant docu-

   the number of
   ments))
   Exact           0.4367

NORMAL DEVIATE - F~1out-Reca11

0.8


0.6  -

Recall

    Ol  0.02  0.16              0.98  0                      0.4~


2                                       0.98
                                                             0.2
1                                       0.84

0                                       0.5                    0
                                                                0          0.05         0.1        0.15         0.2
-1                                      0.16                                       Fallout x 100


-2                                      0.02

-3                                      0.001
  -3     -2   -1     0     1     2     3

snlts - University of California, Berkeley


                     Summary Statistics
        Run Number           Brkly4-full, automatic                                             Recall-Precision Curve
                                                                                 1
        Num of Queries                        50
        Total number of documents over all queries
        Retrieved:                         50000                               0.8
        Relevant:                          10489
        Rel~ret:                            6339
                                                                               0.6

~call Level Averages
Recall        Precision
0.00           0.7606
0.10           0.5512
0.20           0.4543
0.30           0.4004
0.40           0.3522
0.50           0.2797
0.60           0.2032
0.70           0.1665
0.80           0.1166
0.90           0.0631
 1.00          0.0147
Ige precision over all
mt docs
nterpolated    0.2905

  Document Level Averages                Precision
                  Precision                     0.4
At 5 docs          0.6080
At 10 docs         0.5920
                                                0.2
At 15 docs         0.5707
At 20 docs         0.5580
At 30 docs         0.5160                         0
At 100 docs        0.3934
At 200 docs        0.3117
At 500 docs        0.1984
At 1000 docs       0.1268
~-~recision (precision after R
                                                  1
docs  retrieved (where  R  is
                relevant docu-

the number of
ments))
Exact             0.3332

0.8

0        0.2     0.4      0.6      0.8


                     Recall


              Fallout-Recall Curve

 I     I     I     I

  Level Averages
       Recall
()     0.3395
D      0.4263
0      0.4791
0      0.5212
0      0.5500
0      0.5747
0      0.5939
0      0.6096
0      0.6255
0      0.6388
0      0.6530

NORMAL DEVIATE - Fallout-Recall

0.6

Recall

31 ol                           0.98 0                       0.4

2                                       0.98
                                                             0.2
1                                       0.84


0                                       0.5                    0

                                                                 0         0.05         0.1        0.15         0.2
-1                                      0.16                                       Fallout x 100


-2                                      0.02

-3                                      0.001
  -3    -2    -1     0     1     2    3

~sults - University of California, Berkeley

             Summary Statistics
 Run Number
 N um of Queri                       50
 Total number
 Retrieved:                       50000
 Relevant:                        10489
fRel~ret:7237I

 1


0.8<


0.6  -

        Recall-Precision Curve


KY

   ecall ~vel Averages
   Recall      Precision
   0.00         0.8361
   0.10         0.6331
   0.20         0.5384
   0.30         0.4781
   0.40         0.4231
   0.50         0.3611
   0.60         0.2812
   0.70         0.2400
   0.80         0.1714
   0.90         0.0928
    1.00        0.0190
   rage precision over all
   ~ant docs
   -interpolated 0.3538

   it Level Averages
   ut    Recall
   00    0.3969
   20    0.4892
   40    0.5489
   160   0.5881
   180   0.6227
   .00   0.6458
   L20   0.6690
   140   0.6864
   L60   0.7007
   [80   0.7158
   ~0    0.1277

  Document ~vel Averages
                 Precision
At 5 docs         0.6600
At 10 docs        0.6360
At 15 docs        0.6160
At 20 docs        0.6040
At 30 docs        0.5753
At 100 docs       0.4608
At 200 docs       0.3650
At 500 docs       0.2310
At 1000 docs      0.1447
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))

Precision
      0.4


      0.2


        0


        1


      0.8


      0.6

          Exact              0.3852

      NORMAL DEVIATE - Fallout-Recall                 Recall


2


1                                       0.84

0                                       0.5                    0


-1                                      0.16

-2                                      0.02

-3

0.84   0.98 0                   0.4


                                0.2

0.98

3     -2    -1    0     1     2     3

0.001

0       `).2     0.4     0.6      0.8


                    Recall


              Fallout-Recall Curve


0         0.05       0.1       0.15       0.2
                 Fallout x 100

`esults - Universitaet Dortmund


         [             Summary Statistics
          Run Number           dortPl-full, automatic                                          Recall-Precision Curve
          Num of Queries                         50                               1
         L
          Total number of documents over all queries
          Retrieved:                         50000                              0.8~
          Relevant:                          10489
          Rel~ret:                            7148
         {
                                                                                0.6
  ecall Level Averages              Document Level Averages              Precision
  Recall       Precision          _______________   Precision
                                                                                0.4
  0.00          0.7899            At 5 docs          0.6600
  0.10          0.6598            At 10 docs         0.6380
  0.20          0.5844            At 15 docs         0.6387                     0.2
  0.30          0.5331            At 20 docs         0.6300
  0.40          0.4765            At 30 docs         0.6080
                                                                                  0
  0.50          0.4134            At 100 docs        0.4816
  0.60          0.3289            At 200 docs        0.3904
  0.70          0.2502            At 500 docs        0.2500
  0.80          0.1665            At 1000 docs       0.1550
  0.90          0.0801            R-Precision (precision after R
   1.00         0.0041            docs  retrieved (where R  is                    1
  Ige precision over all 1        the number of relevant docu-
  Int docs                        ments))                                       0.8
  nterpolated   0.3800            Exact              0.4195

  Level Averages                                                                0.6
                               NORMAL DEVIATE - Fallout-Recall

        Recall
  )     0.3933
  )     0.5213
  )     0.5807


        0.7370
        0.7511
        0.7666
        0.7743

                                                                            0       0.2     0.4      0.6      0.8


                                                                                                Recall


                                                                                         Fallout-Recall Curve

                                                            I     I     I

                                                                Recall

                         3:1 ol                         0.98  0
                                                                                    0.4.
                         2                                      0.98

                                                                                    0.2
                         1                                      0.84


                         0                                      0.5
                                                                                      0
                        -1                                                              0         0.05        0.1        0.15        0.2
                                                                0.16
                                                                                                         Fallout x 100
                        -2                                      0.02

                        -3                                      0.001
                          -3      -2   -1     0    1     2     3

~su1ts - Universitaet Dortmund


      L             Summary Statistics          I                                         Recall-Precision Curve
      F Run Number         dortVl-full, automatic ~                          1
      ~ Num of Queries                     50 j
        Total number of documents over all queries 1
                                                                           0.8
        Retrieved:                      50000 1
        Relevant:                       10489
        Rel~ret:                         7386                              0.6

   ecall Level Averages
    Recall    Precision
   0.00        0.7864
   0.10        0.6082
   0.20        0.5373
   0.30        0.5023
   0.40        0.4446
   0.50        0.3788
   0.60        0.2997
   0.70        0.2415
   0.80        0.1467
   0.90        0.0590
    1.00       0.0049
   age precision over all
   ant docs
   interpolated 0.3516

   t Level Averages
   it   Recall
   )0   0.3912
   )~0  0.4894           2
   ~    0.5521
   ,0   0.5974           1
   30   0.6316
                         0
   )0   0.6602
   ~0   0.6863          -1
   ~    0.7020
   50   0.7152          -2
   30   0.7279
                        -3
   ~    0.7389

     Document Level Averages                 Precision
                    Precision                       0.4
   At S docs         0.6280
   At 10 docs        0.5980
                                                    0.2
   At 15 docs        0.5960
   At 20 docs        0.5790
   At 30 docs        0.5653                          0
   At 100 docs       0.4520
   At 200 docs       0.3661
   At 500 docs       0.2352
   At 1000 docs      0.1477
   R-Precision (precision after R                    1
   docs  retrieved (where R  is
   the number of relevant docu-
   ments))                                          0.8
   Exact             0.3947

NORMAL DEVIATE - Fallout-Recall
                                             Recall

0       0.2     0.4      0.6     0.8


                   Recall


             Fallout-Recall Curve

 I     I           I

0.6  -

                            0.84  0.98   0                         0.4


                                            0.98
                                                                   0.2
                                            0.84

                                            0.5                      0
                                                                       0            0.05          0.1          0.15           0.2
                                            0.16                                            Fallout x 100


                                            0.02

        I      I             I              0.001
-3     -2     -1      0      1            3

~su1ts - University of Massachusetts at Amherst

        [           Summary Statistics
        [ Run Number      INQOO3-full, automatic
         Num of Queries                    50
         Total number of documents over all queries
         Retrieved:                     50000
         Relevant:                      10489
         Reljet:                         1161

  ecall Level Averages
  Recall      Precision
   0.00        0.1852
   0.10        0.6315
   0.20        0.5296
   0.30        0.4162
   0.40        0.4246
   0.50        0.3621
   0.60        0.2195
   0.10        0.2309
   0.80        0.1135
   0.90        0.1016
   1.00        0.0122
  `age precision~ over all
  `ant docs
  interpolated 0.3538

  t Level Averages
  it   Recall
  )0   0.3148
  ~0   0.4880
  10   0.5412
  )0   0.5823
  ~0   0.6106
  )0   0.6341
  ?0   0.6516
  10   0.6180
  )0   0.6926
  30   0.1063
  )0   0.1155

  Document Level Averages
                  Precision
At S docs          0.6440
At 10 docs         0.6320
At 15 docs         0.6021
At 20 docs         0.5820
At 30 docs         0.5593
At 100 docs        0.4452
At 200 docs        0.3493
At 500 docs        0.2295
At 1000 docs       0.1432
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))

        1


      0.8


      0.6
Precision
      0.4


      0.2


        0

 1


0.8

   Exact            0.3893

NORMAL DEVIATE - Fallout-Recall
                                        Recall

            Recall-Precision Curve


0       0.2     0.4     0.6     0.8


                   Recall


             Fallout-Recall Curve

 I         I         I         I        -`

0.6 -

~1  0~02        5   0.84 0.98  0                   0.4


                                                   0.2

2                                         0.98

1                                          0.84


0                                          0.5                     0
                                                                    0           0.05         0.1         0.15         0.2
-1                                         0.16                                         Fallout x 100

-2                                         0.02

-3        I      I            I     I      0.001
  -3     -2    -1     0       1           3

~esn1ts - University of Massachusetts at Amherst

             Summary Statistics        ]
I Run Number          INQOO4-fulI, manual 1                           1
 Num of Queries                      50
 Total number of documents over all queries
 Retrieved:                       50000                             0.8
 Relevant:                        10489
 Rel~ret:                          7386                             0.6

Recall-Precision Curve

Recall Level Averages                 Document Level Averages            Precision
 Recall        Precision                            Precision                   0.4
  0.00          0.8489              At S docs        0.6680
  0.10          0.6450              At 10 docs       0.6580
                                                                                0.2
  0.20          0.5327              At 15 docs       0.6200
  0.30          0.4821              At 20 docs       0.5940                        0       0.2     0.4     0.6     0.81
  0.40          0.4304              At 30 docs       0.5747                       0
  0.50          0.3633              At 100 docs      0.4474
  0.60          0.3054              At 200 docs      0.3547                                           Recall
  0.70          0.2325              At 500 docs      0.2360
  0.80          0.1762              At 1000 docs     0.1477
                                                                                                Fallout-Recall Curve
  0.90          0.1054              R-Precision (precision after R
                                                                                  1
  1.00          0.0151              docs retrieved (where R  is
..rage precision over all           the number of relevant docu-
vant docs                           ments))                                     0.8
-interpolated   0.3612              Exact            0.3951

Jt Level Averages
ut      Recall
~       0.3956
20      0.4966
40      0.5621
60      0.6083
80      0.6396
.00     0.6649
.20     0.6860
.40     0.7025
.60     0.7203
.80     0.7313
~00     0.7435

NORMAL DEVIATE     Fallout-Recall

      0.6
Recall

    )01                          0.98  0 )~999               0.4~


2                                        0.98
                                                             0.2
1                                        0.84

0                                        0.5                   0
                                                                 0          0.05        0.1         0.15        0.2
-1                                       0.16                                      Fallout x 100


-2                                       0.02

-3                                       0.001
  -3     -2   -1     0      1     2     3

results - HNC, Inc


                      Summary Statistics
         Run Number         HNCrt1-full, automatic                                          Recall-Precision Curve
         Num of Queries                      50 J                             1
         Total number of documents over all queries 1
         Retrieved:                       49970 1                            0.8~
         Relevant:                        10489
         Rel~ret:                          5833
                                                                             0.6

Recall Level Averages
   Recall        Precision
   0.00           0.7867
   0.10           0.5873
   0.20           0.5083
   0.30           0.4187
   0.40           0.3311
   0~50           0.2691
   0.60           0.1962
   0.70           0.1130
   0.80           0.0451
   0.90           0.0180
    1.00          0.0023
erage precision over all
-vant docs
ri-interpolated   0.2810

ut Level Averages
)ut      Recall
~        0.3211
D20      0.3893
~40      0.4274
~0       0.4554
D80      0.4791
~0       0.5037
120      0.5152
~0       0.5260
~0       0.5373
~0       0.5458
200      0.5504

  Document Level Averages
                 Precision
At 5 docs         0.6120
At 10 docs        0.6020
At 15 docs        0.5853
At 20 docs        0.5740
At 30 docs        0.5480
At 100 docs       0.4228
At 200 docs       0.3223
At 500 docs       0.1924
At 1000 docs      0.1167
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Exact             0.3383

Precision
      0.4


      0.2


        0


        1


      0.8


      0.b

0       0.2     0.4     0.6      0.8


                   Recall


             Fallout-Recall Curve

 I    I

y

      NORMAL DEVIATE - Fallout-Recall

    Ol  0.02                     0.98  0                        0.4


2                                         0.98
                                                                0.2
1                                         0.84

0                                         0.5                     0

                                                                    0          0.05         0.1         0.15         0.2
-1                                        0.16                                         Fallout x 100


-2                                        0.02

-3  ________________        I      I      0.001

Recall

-3     -2    -1     0     1     2     3

results - HNC, Inc.

                       Summary Statistics
           Run Number         HNCrt2-full, automatic 1
                                                                               1
           Num of Queries                      50
           Total number of documents over all queries
           Retrieved:                       50000                             0.8~
           Relevant:                        10489
           Rel~ret:                          6407

Recall Level Averages
    Recall       Precision
    0.00          0.8028
    0.10          0.5734
    0.20          0.5058
    0.30          0.4316
    0.40          0.3134
    0.50          0.2933
    0.60          0.2190
    0.70          0.1373
    0.80          0.0535
    0.90          0.0137
    1.00          0.0029
erage precision over all
evant docs
n-interpolated    0.2921

)ut Level Averages
out      Recall
000      0.3238
020      0.4185
040      0.4718
`060     0.5285
`080     0.5523
100      0.5731
`120     0.5884
140      0.6024
160      0.6117
180      0.6200
~200     0.6281

                                                                               0.6
                                Document Level Averages                 Precision
                                                Precision                      0.4
                              At 5 docs          0.6200
                              At 10 docs         0.6100
                                                                               0.2
                              At 15 docs         0.5827
                              At 20 docs         0.5560
                              At 30 docs         0.5300                         0
                              At 100 docs        0.4090
                              At 200 docs        0.3296
                              At 500 docs        0.2095
                              At 1000 docs       0.1281
                              R-~recision (precision after R                    1
                              docs  retrieved (where  R  is
                              the number of relevant docu-
                              ments))                                          0.8
                              Exact              0.3638

                           NORMAL DEVIATE - Fallout-Recall
                                                                        Recall

                                                                          0.6

                                                                                         Recall-Precision Curve


                                                                             0      0.2     0.4      0.6     0.8


                                                                                               Recall


                                                                                          Fallout-Recall Curve

                                                                                      I        I

                               )01 0.02                     0.98 0                      0.4


                           2                                        0.98
                                                                                        0.2
                                                                    0.84


                           0                                        0.5                   0
                                                                                            0         0.05         0.1        0.15         0.2
                           -1                                       0.16                                      Fallout x 100

                           -2                                       0.02

                           -3        I     I           I     I      0.001
                              -3    -2    -1     0     1     2     3

~su1ts - Thinking Machines Corp.


                     Summary Statistics
         Run Number         TMC6-full, automatic                                           Recall-Precision Curve
         Num of Queries                      50                              1
         Total number of documents over all queries
         Retrieved:                       46075                            0.8  -
         Relevant:                        10489
         Rel~ret:                         6061

~ecall Level Averages
 Recall        Precision
 0.00           0.6276
 0.10           0.4638
 0.20           0.4142
 0.30           0.3724
 0.40           0.3194
 0.50           0.2670
 0.60           0.1950
 0.70           0.1482
 0.80           0.1123
 0.90           0.0547
 1.00           0.0211
age precision over all   1
ant docs
interpolated    0.2553   J

 Level Averages
t      Recall
~0     0.2924
0      0.3747
0      0.4267
0      0.4887
0      0.5237
0      0.5472
0      0.5666
D      0.5883
0      0.6021
~      0.6172
~      0.6274

   Document Level Averages
                   Precision
 At 5 docs         0.4320
 At 10 docs        0.4180
 At 15 docs        0.4187
 At 20 docs        0.4200
 At 30 docs        0.4060
 At 100 docs       0.3396
 At 200 docs       0.2839
 At 500 docs       0.1868
 At 1000 docs      0.1213
 R-Precision (precision after R
 docs  retrieved (where R  is
 the number of relevant docu-
 ments))

L

      Exact          0.2991

    NORMAL DEVIATE - Fallout-Recall

*(   -.-      ).            (

)01  0.02

0.16

5    0.84  0.98

0

 2


 1


-1

0


-2

-3

)~999
0.98
                     0.2
0.84


0.5                    0

                        0          0.05         0.1        0.15         0.2
0.16
                                           Fallout x 100
0.02

0.001

-3    -2    -1     0     1            3

      0.6
Precision
      0.4


      0.2


        0


        1


      0.8


      0.6


      0.4

Recall

0       0.2     0.4      0.6     0.8


                   Recall


              Fallout-Recall Curve

 I         I

re8ults - Thinking Machines Corp.


                      Summary Statistics
                                                                                         Recall-Precision Curve
         Run Number          TMC1-full1 automatic 1
                                                                            1                        -I
         Num of Queries                     50
         Total number of documents over all queries
         Retrieved:                       46122                           0.8
         Relevant:                        10489
         Reljet:                           5112
                                                                          0.6 -

Recall Level Averages
 Recall         Precision
  0.00           0.5085
  0.10           0.3828
  0.20           0.3318
  0.30           0.3064
  0.40           0.2632
  0.50           0.1964
  0.60           0.1581
  0.10           0.1240
  0.80           0.0184
  0.90           0.0453
  1.00           0.0202
brage precision over all 1
`vant docs                j
i-interpolated   0.2045   j

  Document Level Averages
                  Precision
At5docs            0.3120
At 10 docs         0.3360
At 15 docs         0.3341
At 20 docs         0.3360
At 30 docs         0.3281
At 100 docs        0.2920
At 200 docs        0.2440
At 500 docs        0.1688
At 1000 docs       0.1142
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Exact              0.2564

Precision
      0.4


      0.2


        0

0       0.2     0.4     0.6     0.8


                   Recall


             Fallout-Recall Curve

 I     I     I

 1


0.8 -

ut Level Averages
ut      Recall
~       0.2333
~20     0.3224
~0      0.3114
~0      0.4123
~0      0.4433
.00     0.4121
.20     0.4961
.40     0.5158
.60     0.5360
.80     0.5482
~0      0.5611

NORMAL DEVIATE - Fallout-Recall

      0.6
Recall

    101 0.02   0.16  ( 5   flR4   0.98 0
                                                              0.4

2                                         0.98
                                                              0.2
1                                         0.84

0                                         0.5                  0

                                                                 0         0.05        0.1        0.15         0.2
-1                                        0.16                                     Fallout x 100


-2                                        0.02

-3                                        0.001
  -3    -2     -1    0      1      2    3

results - Beilcore


                      Summary Statistics              )
            Run Number         Isiri-full, automatic 1                                           Recall-Precision Curve
                                                                                   1
          L Num of Queries                    50
            Total number of documents over all queries 1
            Retrieved:                     50000                                  0.8
            Relevant:                      10489
            Rel~ret:                        6522 J

Recall Level Averages
   Recall       Precision
   0.00          0.7569
   0.10          0.5281
   0.20          0.4302
   0.30          0.3779
   0.40          0.3222
   0.50          0.2446
   0.60          0.1935
   0.70          0.1474
   0.80          0.0847
   0.90          0.0295
   1.00          0.0043
rage precision over al  1
`,ant docs       0.266:
-interpolated

it Level Averages
Lit     Recall
0~0     0.3042
20      0.4031
40      0.4660
60      0.5058
80      0.5365
o0      0.5635
20      0.5838
40      0.6034
        0.6276
BO      0.6431
        0.6554

                                Document Level Averages
                                                Precision
                              At 5 docs         0.6120
                              At 10 docs        0.5480
                              At 15 docs        0.5387
                              At 20 docs        0.5110
                              At 30 docs        0.4880
                              At 100 docs       0.3798
                              At 200 docs       0.3037
                              At 500 docs       0.2011
                              At 1000 docs      0.1304
                              R-Precision (precision after R
                              docs  retrieved (where R  is
                              the number of relevant docu-
                              ments))
                              Exact             0.3050

                           NORMAL DEVIATE - Fallout-Recall

                                                                          0.6
                                                                   Precision
                                                                          0.4


                                                                          0.2


                                                                           0


                                                                           1


                                                                          0.8


                                                                          0.6

                                                                   Recall

                         )01  0.02             0.84 0.98  0
                                                                              0.4


                                                                              0.2 -

                           2                                       0.98

                                                                             0      0.2      0.4      0.6     0.8


                                                                                                 Recall


                                                                                          Fallout-Recall Curve

                                                             I          I

                           1                                       0.84

                           0                                       0.5                   0

                                                                                           0         0.05         0.1         0.15        0.2
                          -1                                       0.16                                      Fallout x 100


                          -2                                       0.02

                          -3                                       0.001
                            -3     -2    -1    0     1           3

~u1ts - Beilcore

                    Summary Statistics
         Run Number          lsir2-full, automatic I                       1
         Num of Queries                    50 j
         Total number of documents over all queries 1
                                                                          0.8
         Retrieved:                     50000
         Relevant:                      10489
         Rel~ret:                        1155 J                           0.6

call Level Averages
~ecall        Precision
0.00           0.8533
0.10           0.6270
0.20           0.5435
0.30           0.4122
0.40           0.4080
0.50           0.3546
0.60           0.2913
0.70           0.2032
0.80           0.1282
0.90           0.0717
 1.00          0.0044
ige precision over all
tnt docs
,terpolated    0.3442

Level Averages
      Recall
      0.3478
      0.4676
      0.5269
      0.5817
      0.6136
      0.6407
      0.6632
      0.6192
      0.6973
      0.7079
      0.7174

         Document Level Averages
                        Precision
      At 5 docs          0.6960
      At 10 docs         0.6660
      At 15 docs         0.6253
      At 20 docs         0.5960
      At 30 docs         0.5740
      At 100 docs        0.4524
      At 200 docs        0.3562
      At 500 docs        0.2274
      At 1000 docs       0.1431
      R- Precision (precision after R
      docs  retrieved  (where R  is
      the number of relevant docu-
      ments))
      Exact              0.3804

   NORMAL DEVIATE - Fallout-Recall

)01  0.02  0.16  (

5

0.84   0.98   0

(

2


1

Precision
      0.4


      0.2


        0


        1


      0.8


      0.6

Recall

~999              0.4

0.98
                  0.2 -
0.84

0                                       0.5                   0


-1                                      0.16

-2                                      0.02

-3  -7                                  0.001
         I     I           I     I

  -3     -2   -~     0     1     2     3

            Recall-Precision Curve


0       0.2     0.4      0.6     0.8


                   Recall


             Fallout-Recall Curve


0         0.05      0.1        0.15      0.2
                Fallout x 100

~su1ts - New York University


                    Summary Statistics
       Run Number       nyuirl-category B, automatic 1                                  Recall-Precision Curve
       Num of Queries                        50                            1
         Total number of documents over all queries
       Retrieved:                         50000                           0.8
       Relevant:                           2064
       Rel~ret:                            1390

`call Level Averages
Recall        Precision
0.00           0.5400
0.10           0.3937
0.20           0.3423
0.30           0.2512
0.40           0.2263
0.50           0.2032
0.60           0.1614
0.10           0.1295
0.80           0.0905
0.90           0.0442
 1.00          0.0284
ge precision over all
nt docs
iterpolated    0.2038

Level Averages
       Recall
                        1)
       0.2413
       0.2950
                        2
       0.3314
       0.3126           1
       0.3872
                        0
       0.4046
       0.4181           -1
       0.4328
       0.4415           -2
       0.4600
       0.4698

                              Document Level Averages
                                              Precision
                            At 5 docs          0.3360
                            At 10 docs         0.3240
                            At 15 docs         0.2933
                            At 20 docs         0.2790
                            At 30 docs         0.2401
                            At 100 docs        0.1412
                            At 200 docs        0.0939
                            At 500 docs        0.0489
                            At 1000 docs       0.0218
                            R-Precision (precision after R
                            docs  retrieved (where R  is

                                                                      0.6
                                                               Precision
                                                                      0.4


                                                                      0.2


                                                                       0


                                                                       1

                            the number of relevant docu-
                            ments))                                    0.8  -
                            Exact            0.2267
                                                                       0.6

                                                                         0      0.2      0.4     0.6     0.8


                                                                                            Recall


                                                                                      Fallout-Recall Curve

                                                            I     I     I     I

                              NORMAL DEVIATE - Fallout-Recall

                                                  0.84   0.98 0                       0.4


                                                                 0.98
                                                                                      0.2
                                                                 0.84

                                                                 0.5                   0

                                                                                         0          0.05         0.1        0.15         0.2
                                                                 0.16                                       Fallout x 100


                                                                 0.02

                                                                 0.001

                                                               Recall

                          -3    -2    -1     0      1     2     3

suits - New York University


  L

                   Summary Statistics             j
     Run Number        nyuir2-category B, automatic 1                                      Recall-Precision Curve
                                                                             1
     Num of Queries                           50
      Total number of documents over all queries
     Retrieved:                            50000                            0.8
     Relevant:                              2064
     Rel~ret:                               1610
                                                                            0.6

call Level Averages
~ecall       Precision
0.00          0.6435
0.10          0.4610
0.20          0.3105
0.30          0.3031
0.40          0.2631
0.50          0.2282
0.60          0.1934
0.10          0.1542
0.80          0.1002
0.90          0.0456
1.00          0.0186
~e precision over all
~t docs
terpolated    0.2331

  Document Level Averages
                 Precision
At S docs         0.4280
At 10 docs        0.4000
At 15 docs        0.3613
At 20 docs        0.3260
At 30 docs        0.2740
At 100 docs       0.1108
At 200 docs       0.1018
At 500 docs       0.0515
At 1000 docs      0.0322
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Exact             0.2513

Precision
      0.4


      0.2


        0


        1


      0.8

0       0.2     0.4      0.6     0.8


                   Recall


             Fallout-Recall Curve

 I          I          I          I

evel Averages
   Recall
   0.2600
   0.3484
   0.3946
   0.4336
   0.4551
   0.4858
   0.5033
   0.5126
   0.5246
   0.5355
   0.5486

NORMAL DEVIATE - Fallout-Recall

      0.6
Recall

    101  a.02              0.84  0.98  0 111111999               0.4


  2                                       0.98
                                                                 0.2
  1                                       0~84

  0                                       0.5                      0

                                                                     0         0.05         0.1         0.15        0.2
-1                                        0.16                                         Fallout x 100


-2                                        0.02

-3  _______________          I     I      0.001
   -3    -2     -1    0      1     2    3

~sults - Carnegie Mellon University


                    Summary Statistics
      [Run Number         CLARTA-full, automatic                                          Recall-Precision Curve
      [ Num of Queries                       50                              1
        Total number of documents over all queries
        Retrieved:                        50000                            0.8 L
        Relevant:                         10489
        Rel~ret:                           6811

                                                                              0.6
~call Level Averages               Document Level Averages              Precision
Recall       Precision                          Precision                     0.4
0.00          0.8101             At 5 docs       0.6200
0.10          0.5781             At 10 docs      0.6060
0.20          0.4941             At 15 docs      0.5893                       0.2
0.30          0.4368             At 20 docs      0.5730                          0       0.2     0.4     0.6      0.81
0.40          0.3851             At 30 docs      0.5460                         0
0.50          0.3358             At 100 docs     0.4346
0.60          0.2749             At 200 docs     0.3416                                             Recall
0.70          0.2074             At 500 docs     0.2184
0.80          0.1561             At 1000 docs    0.1362
0.90          0.0933                                                                          Fallout-Recall Curve
                                 R-Precision {precision after R
 1.00         0.0141                                                            1
                                 docs retrieved (where R   is
ge precision over all            the number of relevant docu-
nt docs                          ments))                                      0.8
iterpolated   0.3269             Exact           0.3646

Level Averages
    Recall
    0.3545
    0.4571
    0.5408
    0.5801
    0.6132
    0.6422
    0.6603
    0.6822
    0.6959
    0.7096
    0.7172

                                                   0.6
   NORMAL DEVIATE - Fallout-Recall

101  0.02            0.84  0.98 0
                                                   0.4


                                                   0.2

Recall

2                                       0.98

1                                        0.84

0                                        0.5                   0

                                                                0          0.05         0.1        0.15         0.2
-1                                       0.16                                      Fallout x 100


-2                                       0.02

-3                                       0.001
  -3    -2    -1     0     1     2     3

~q~iltg - Garne~e Mellon University

                       Summary Statistics

            Run Number        CLARTM-full, manual
            Num of Queries                       501
            Total number of documents over all queries
            Retrieved:                       50000
            Relevant:                        10489
            Reljet:                           6785

                                   Document Level Averages
    ~ecall Level Averages
                                                    Precision
    Recall       Precision
                                 At 5 docs          0.6120
    0.00         0.7191
                                 At 10 docs         0.5940
    0.10         0.5825
                                 At 15 docs         0.5800
    0.20         0.4987
                                 At 20 docs         0.5700
    0.30         0.4505
                                 At 30 docs         0.5527
    0.40         0.3848
                                 At 100 docs        0.4396
    0.50         0.3295
                                 At 200 docs        0.3436
    0.60         0.2739
                                 At 500 docs        0.2176
    0.70         0.2109
                                 At 1000 docs       0.1357
    0.80         0.1683
    0.90         0.0951          R-Precision (precision after R
    1.00         0.0126          docs  retrieved   (where R is
                                 the number of relevant docu-
       precision over all
                                 ments))
    vant docs
    i-interpolated 0.3302        Fxact              0.3642

                              NORMAL DEVIATE - Fallout-Recall
    ut Level Averages
    )ut  Recall             001 0.02  0.16  C      0.84 0.98 0.9~999
    )00  0.3602
    ~20  0.4615           2                                   0.98
    ~0   0.5383
    060  0.5783           1                                   0.84
    080  0.6143
                          0                                   0.5
    100  0.6371
    120  0.6611           -1                                  0.16
    140  0.6781
    160  0.6946           -2                                  0.02
    `180 0.7071           -3                                  0.001
    200  0.7199            -3   -2    -1    0      1    2    3

                     Recall-Precision Curve


        1


      0.8


      0.6
Precision
      0.4


      0.2


        0
         0      0.2     0.4     0.6     0.8
                           Recall

                      Fallout-Recall Curve


        1


      0.8


      0.6
Recall
      0.4


      0.2


        0
         0        0.05                0.15        0.2

    0.1
Fallout x 100

g results - Institute for Decision Systems Research

                      Su~y Statistics
  Run Number         idsra2-category B, partial topics, manual
  Num of Queries                                    10!
           Total number of documents over all queries
  Retrieved:                                     10000
 fRelevant:                                        423
 IRel~ret:                                         267

Recall Level Averages
 Recall         Precision
  0.00           0.5279
  0.10           0.4577
  0.20           0.4183
  0.30           0.3665
  0.40           0.3159
  0.50           0.2956
  0.60           0.2425
  0.70           0.1672
  0.80           0.0971
  0.90           0.0115
  1 00           0.0000
     precision over all
~!~docsI
i-interpolated   0.2504

 Document Level Averages
                  Precision
At 5 docs          0.4000
At 10 docs         0.4200
At 15 docs         0.3933
At 20 docs         0.3600
At 30 docs         0.3333
At 100 docs        0.1870
At 200 docs        0.1040
At 500 docs        0.0482
At 1000 docs       0.0267
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Exact              0.3085

it Level Averages             NORMAL DEVIATE - Fallout-Recall
ut     Recall
                          ~.001 0.02 0.16  0.5  0.84 0.98  0.9~999
00     0.4079
20     0.5099
                          2                                  0.98
40     0.5397
                          1
50     0.5591                                                0.84
~0     0.5642
)0     0.5717             0                                  0.5
~0     0.5834
                         -1                                  0.16
10     0.5834
)0     0.5920            -2                                  0.02
10     0.5927
~0     0.5979            -3
                                                             0.001
                           -3   -2   -1    0     1    2     3

           1


         0.8


         0.6
Precision  <
         0.4


         0.2


           0

            Recall-Precision Curve
         I               I


0       0.2     0.4     0.6     0.8
                                         1

                           Recall

                      Fallout-Recall Curve
        1


      0.8


      0.6
Recall
      0.4


      0.2


        0
         0        0.05       0.1       0.15        0.2
                         Fallout x 100

`esults - Swiss Federal Institute of Technology (ETH)

          [            Summary Statistics        I
          I Run Number        schaul-full, automatic 1
          L Num of Queries                    50
           Total number of documents over all queries 1
           Retrieved:                      50000 1
           Relevant:                       10489
           Reljet:                          5345 j

Recall Level Averages
   Recall        Precision
   0.00           0.5925
   0.10           0.4192
   0.20           0.3618
   0.30           0.3083
   0.40           0.2453
   0.50           0.1982
   0.60           0.1319
   0.10           0.0824
   0.80           0.0485
   0.90           0.0165
   1.00           0.0023
`rage precision over all
vant docs
i-interpolated    0.1994

ut Level Averages
~t      Recall
~       0.2488
120     0.3126
140     0.3618
160     0.4031
)80     0.4549
LOO     0.4855
L20     0.5102
L40     0.5341
L60     0.5501
L80     0.5684
?00     0.5811

  Document Level Averages
                  Precision
At 5 docs          0.4000
At 10 docs         0.3960
At 15 docs         0.3680
At 20 docs         0.3550
At 30 docs         0.3500
At 100 docs        0.2910
At 200 docs        0.2350
At 500 docs        0.1561
At 1000 docs       0.1069
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))

        1


      0.8


      0.6
Precision
      0.4


      0.2


        0

   Fxact              0.2108

NORMAL DEVIATE - Fallout-Recall
                                              Recall
 0.02  0.16   C 5

 1


0.8


0.6

0.84  0.98  0                  0.4


                               0.2

2                                       0.98

            Recall-Precision Curve


0       0.2     0.4     0.6     0.8


                   Recall


             Fallout-Recall Curve

1                                        0.84

0                                        0.5                    0
                                                                 0          0.05         0.1        0.15         0.2
-1                                       0.16                                       Falloutx 100


-2                                       0.02

-3        I      I          I      I     0.001
  -3    -2     -1     0     1      2    3

results - The Environment Research Institute of Michi~an


                       Summary Statistics
          Run Number          erimrl-fulI, automatic                                          Recall-Precision Curve
          Num of Queries                       50                               1
          Total number of documents over all queries
          Retrieved:                        41829                             0.8
          Relevant:                         10489
          Rel~ret:                           4152
                                                                              0.6~

Recall Level_Averages
   Recall        Precision
   0.00          0.5948
   0.10          0.3190
   0.20          0.2334
   0.30          0.1817
   0.40          0.1280
   0.50          0.0841
   0.60          0.0441
   0.70          0.0210
   0.80          0.0115
   0.90          0.0020
   1.00          0.0020
erage precision over all
evant docs
n-interpolated   0.1219

~t Level Averages
)ut     Recall
000     0.1771
020     0.2227
040     0.2581
060     0.2834
080     0.3169
100     0.3383
120     0.3545
140     0.3794
160     0.3886
180     0.3995
~0      0.4104

  Document Level Averages
                 Precision
At 5 docs         0.3720
At 10 docs        0.3580
At 15 docs        0.3347
At 20 docs        0.3340
At 30 docs        0.3113
At 100 docs       0.2304
At 200 docs       0.1802
At 500 docs       0.1203
At 1000 docs      0.0830
R-Precision (precision after
docs  retrieved (where  R
the number of relevant
ments))

   R
   Is
docu-

Exact            0.1814

Precision
      0.4


      0.2


        0


        1


      0.8


      0.6

      NORMAL DEVIATE - Fallout-Recall

    p01  0.02                    0.98   0 )~999
                                                               0.4

2                                          0.98

                                                               0.2
                                           0.84

0                                          0.5                   0

                                                                  0          0.05         0.1        0.15         0.2
-1                                         0.16                                      Fallout x 100


-2                                         0.02

-3                                         0.001
   -3    -2    -1     0     1     2      3

Recall

0       0.2     0.4     0.6      0.8


                   Recall


             Fallout-Recall Curve

 I    I

`esults - The Environment Research Institute of Michigan

             Summary Statistics
Run Number          erimr2-full, automatic
                                                                     1
Num of Queries                       50
Total number of documents over all queries
Retrieved:                        45274                             0.8 -
Relevant:                         10489
Rel~ret:                           4475                             0.6

Recall-Precision Curve

Recall Level Averages
   Recall       Precision
   0.00          0.6448
   0.10          0.3426
   0.20          0.2643
   0.30          0.2098
   0.40          0.1551
   0.50          0.1116
   0.60          0.0568
   0.70          0.0300
   0.80          0.0168
   0.90          0.0074
   1.00          0.0006
`rage precision over all
vant docs
-interpolated    0.1415

Jt Level Averages
ut      Recall
~       0.1891
20      0.2408
~0      0.2940
60      0.3245
~0      0.3517
.00     0.3797
.20     0.3958
.40     0.4092
.60     0.4219
.80     0.4343
~00     0.4445

  Document Level Averages
                 Precision
At 5 docs         0.4720
At 10 docs        0.4240
At 15 docs        0.3933
At 20 docs        0.3810
At 30 docs        0.3420
At 100 docs       0.2524
At 200 docs       0.2012
At 500 docs       0.1394
At 1000 docs      0.0895
R-Precision (precision after R
docs  retrieved (where  R  is
the number of relevant docu-
ments))

Precision
       0.4


       0.2


        0

 1


0.8

   Exact            6~2031

NORMAL DEVIATE - Fallout-Recall
                                         Recall

0       0.2     0.4      0.6     08


                   Recall


             Fallout-Recall Curve

0.6  -

    Ol   0.02  0.16   1 5   0.84  0.98   0.                       0.4


2                                           0.98
                                                                  0.2
1                                           0.84

                                                                                  I            I
0                                           0.5                    0
                                                                     0          0.05         0.1         0.15          0.2
-1                                          0.16                                         Fallout x 100


-2                                          0.02

-3                                          0.001
   -3    -2    -1     0      1     2      3

resjilts - GE Research and Development Center


         [            Summary Statistics       ]
         [ Run Number          gecrdl-full1 manual 1                                        Recall-Precision Curve
                                                                               1
          Num of Queries                     50
          Total number of documents over all queries
          Retrieved:                      44004                               0.8
          Relevant:                       10489
          Rel~ret:                         6118

~ecall Level_Averages
 Recall        Precision
  0.00          0.8112
  0.10          0.6131
  0.20          0.5320
  0.30          0.4518
  0.40          0.3988
   0.50         0.3305
  0.60          0.2198
  0.10          0.2196
  0.80          0.1393
   0.90         0.0868
   1.00         0.0130
age precision over all  1
ant docs                J
interpolated    0.3308  ]

  Level Averages
t       Recall
  0     0.3899
  0     0.4141
  0     0.5113
  0     0.5400
  0     0.5625
  0     0.5186
  0     0.5932
  0     0.6086
  0     0.6220
  0     0.6291
  0     0.6381

                                                                       0.6

                               Document Level Averages             Precision
                                               Precision                  0.4
                             At 5 docs          0.6080
                             At 10 docs         0.6060
                             At 15 docs         0.5821                    0.2
                             At 20 docs         0.5660
                             At 30 docs         0.5440
                                                                            0
                             At 100 docs        0.4528
                             At 200 docs        0.3364
                             At 500 docs        0.2045
                             At 1000 docs       0.1236
                             R-Precision (precision after R
                                                                            1
                             docs  retrieved (where R  is

                                                                          0      0.2      0.4     0.6     0.8


                                                                                             Recall


                                                                                       Fallout-Recall Curve

                                                           II>

                              the number of relevant docu-
                              ments))                                    0.8
                              Exact            0.3812
                                                                         0.6
                           NORMAL DEVIATE - Fallout-Recall

                         )01            5   0.84 o.gs
                                                       ~~999             0.4<
                       2                                0.98

                                                                         0.2
                       1                                0.84

                       0                                0.5                0

                                                                            0        0.05      0.1       0.15      0.2
                     -1                                 0.16
                                                                                           Fallout x 100
                     -2                                 0.02


                     -3                                 0.001

                                                                Recall

                           -3    -2     -1    0     1     2     3

results - GE Research and Development Center

             Summary Statistics
Run Number          gecrd2-full, automatic                            1
Num of Queries                       50
Total number of documents over all queries
                                                                     0.8
Retrieved:                        49261
Relevant:                         10489
Rel~ret:                           4485                              0.6

Recall-Precision Curve

Recall Level_Averages
 Recall         Precision
   0.00          0.6802
   0.10          0.3796
   0.20          0.3111
   0.30          0.2581
   0.40          0.2108
   0.50          0.1600
   0.60          0.1015
   0.10          0.0611
   0.80          0.0360
   0.90          0.0150
   1.00          0.0009
~rage precision over all
`vant docs
i-interpolated   0.1772

ut Level Averages
~       Recall
~00     0.2109
020     0.2171             2
040     0.3179
060     0.3513             1
080     0.3156
                           0
100     0.3987
120     0.4178            -1
140     0.4340
160     0.4458            -2
180     0.4559
                          -3
200     0.4671

     Document Level Averages                Precision
                    Precision                      0.4
   At 5 docs         0.4760
   At 10 docs        0.4460
                                                   0.2
   At 15 docs        0.4280
   At 20 docs        0.4130                           0      0.2     0.4     0.6     0.81
   At 30 docs        0.3907                         0
   At 100 docs       0.2930
   At 200 docs       0.2275                                             Recall
   At 500 docs       0.1385
   At 1000 docs      0.0897
                                                                  Fallout-Recall Curve
   R-Precision (precision after R                   1
   docs  retrieved (where R  is
   the number of relevant docu-
   ments))                                         0.8
   Exact             0.2342

NORMAL DEVIATE - Fallout-Recall
                                            Recall

0.6 -

  P01                   0.84   0.98  0 )~999                0.4


                                        0.98
                                                            0.2
                                        0.84

                                        0.5                   0
                                                                0         0.05         0.1        0.15         0.2
                                        0.16                                      Fallout x 100


                                        0.02

                                        0.001

-3    -2    -1     0     1      2     3

results - TRW Systems and Development Division


                     Summary Statistics
         Run Number             trwl-full, manual 1                                         Recall-Precision Curve
         Num of Queries                      50                               1
         Total number of documents over all queries
         Retrieved:                       36485                              0.8
         Relevant:                        10489
         Rel~ret:                          5681
                                                                             0.6 -

~ecall Level Averages
 Recall       Precision
 0.00          0.1181
 0.10          0.4823
 0.20          0.4319
 0.30          0.3706
 0.40          0.3093
 0.50          0.2586
 0.60          0.1878
 0.70          0.1471
 0.80          0.1094
 0.90          0.0528
 1.00          0.0113
age precision over all
ant docs
interpolated   0.2525

   Document Level Averages
________________    Precision
 At 5 docs           0.4640
 At 10 docs          0.4100
 At 15 docs          0.4507
 At 20 docs          0.4300
 At 30 docs          0.4247
 At 100 docs         0.3770
 At 200 docs         0.2929
 At 500 docs         0.1847
 At 1000 docs        0.1136
 R-Precision (precision after R
 docs  retrieved   (where R  is
 the number of relevant docu-
 ments))
 Exact               0.3220

Precision
      0.4


      0.2


        0


        1


      0.8

0       0.2      0.4     0.6     0.8


                    Recall


              Fallout-Recall Curve

 I     I     I     I

                                                                             0.6
  L Level Averages            NORMAL DEVIATE - Fallout-Recall
  t    Recall
       0.3071
  0    0.4006
  *0   0.4389
  0    0.4745
  0    0.5004
  0    0.5218
  0    0.5389
  0    0.5507
  0    0.5614
  0    0.5685
  0    0.5743

                           g) 101   0.02

                                                          0.98  0 )~999

                           2                                       0.98

                                                                   Recall

                                                                          0.4


                                                                          0.2  -

                           1                                       0.84

                           0                                       0.5                   0

                                                                                           0          0.05        0.1         0.15         0.2
                          -1                                       0.16
                                                                                                              Fallout x 100
                          -2                                       0.02

                          -3                                       0.001
                             -3    -2    -1    0     1     2     3

esults - TRW Systems and Development Division

L
L

             Summary Statistics
Run Number              trw2-full, manual
Num of Queries                       50
Total number of documents over all queries
Retrieved:                        48432
Relevant:                         10489
Rel~ret:                           7383

~ecall Level Averages
   Recall       Precision
   0.00          0.8320
   0.10          0.6329
   0.20          0.5431
   0.30          0.4694
   0.40          0.4096
   0.50          0.3558
   0.60          0.2848
   0.70          0.2132
   0.80          0.1499
   0.90          0.0899
   1.00          0.0210
rage precision over all
~ant docs
-interpolated    0.3459

it Level Averages
ut      Recall
00      0.3768
20      0.4753            2
40      0.5289
`60     0.5694            1
`80     0.6029
                          0
.00     0.6321
.20     0.6599           -1
.40     0.6770
.60     0.6930           -2
.80     0.7035
                         -3
!00     0.7147

  Document Level Averages
                 Precision
At 5 docs         0.6680
At 10 docs        0.6240
At 15 docs        0.6107
At 20 docs        0.5810
At 30 docs        0.5453
At 100 docs       0.4554
At 200 docs       0.3623
At 500 docs       0.2350
At 1000 docs      0.1477
R-Precision (precision after R
                (where R  is
               relevant docu-

       docs retrieved
       the number of
       ments))
      ~t03807

   NORMAL DEVIATE - Fallout-Recall
                                               Recall
                     0.84  0.98. - 0


-3    -2   -1   0     1    2      3

                     Recall-Precision Curve
        1


      0.8<


      0.6
Precision
      0.4


      0.2

                                    I       I
        0
         0       0.2     0.4     0.6       0.8      1
                            Recall


                      Fallout-Recall Curve
        1


      0.8


      0.6  -


      0.4<

0.98
                      0.2
0.84


0.5                     0
                          0          0.05         0.1         0.15         0.2
0.16                                         Fallout x 100


0.02

0.001

results - Verity, Inc.

          ~um mary Statistics
Run Number         TOPIC2-full, manual
Num of Queries                    50

]

1

Total number of documents over all queries
Retrieved:                       40869                           0.8
Relevant:                        10489
Rel~ret:                          5600

Recall-Precision Curve

Recall Level Averages
   Recall        Precision
   0.00           0.7300
   0.10           0.5963
   0.20           0.4923
   0.30           0.4160
   0.40           0.3450
   0.50           0.2139
   0.60           0.1183
   0.10           0.1350
   0.80           0.0963
   0.90           0.0289
   1.00           0.0120
.~rage precision over all
.~ant docs
i-interpolated    0.2830

~t Level Averages
ut       Recall
00       0.3293
`20      0.4000
`40      0.4399
~0       0.4617
80       0.4843
00       0.4966
 20      0.5094
40       0.5243
 60      0.5355
 80      0.5500
 DO      0.5631

                                             0.6 -
  Document Level Averages             Precision
                  Precision
                                             0.4
At 5 docs          0.6080
At 10 docs         0.5800
At 15 docs         0.5513                    0.2
At 20 docs         0.5390
At 30 docs         0.5221
                                              0
At 100 docs        0.4028
At 200 docs        0.3048
At 500 docs        0.1805
At 1000 docs       0.1120
R-Precision (precision after R
docs  retrieved (where R  is                  1

   the number of relevant docu-
   ments))                                     0.8
   Fxact            0.3418

                                               0.6
NORMAL DEVIATE - Fallout-Recall

              5  0.84 0.98  0
                                               0.4


                                               0.2 -

Recall

2                                       0.98

0       02      0.4     0.6     0.8


                   Recall


             Fallout-Recall Curve

 I    I

1                                       0.84


0                                       0.5                   0

-1                                                              0          0.05         0.1        0.15         0.2
                                        0.16
                                                                                   Fallout x 100
-2                                      0.02

-3                                      0.001
  -3    -2    -1     0      1    2     3

esults - Siemens Corporate Research, Inc.

        ~ary Statistics
         Run Number            siemsl-full, automatic I
         Num of Queries                        50
                         of documents over all queries
                                            50000
         Retrieved:                         10489
         Relevant:                           6666
         Rel~ret:                                 I

Recall Level Averages
Recall           Precision
0.00              0.1391
0.10              0.5851
0.20              0.5002
0.30              0.4330
0.40              0.3631
0.50              0.3050
0.60              0.2333
0.10              0.1555
0.80              0.0860
0.90              0.0214
1.00              0.0042
~erage precision over all
levant docs
)n-interpolated   0.2984

  Document Level Averages
                 Precision
At 5 docs         0.6080
At 10 docs        0.5960
At 15 docs        0.5693
At 20 docs        0.5450
At 30 docs        0.5260
At 100 docs       0.4010
At 200 docs       0.3229
At 500 docs       0.2092
At 1000 docs      0.1333
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Exact             0.3482

                                NORMAL DEVIATE      F~Iout~RecaII
     lout Level Averages
     ilout Recall           ~.o0i  0.02  0.16  0.5  0.84 0.98  0.9~999
     0000 0.3305
     0020 0.4414            2                                     0.98
     0040 0.4944
                            1                                     0.84
     0060 0.5411
     ~0080 0.5731           0                                     0.5
     )0100 0.6022
     )0120 0.6232          -1                                     0.16
     )0140 0.6427
                           -2                                     0.02
     ~0160 0.6628
     D0180 0.6759          -3                                     0.001
     00200 0.6865            3     -2    -1    0      1   2     3

        1


      0.8


      0.6
Precision
      0.4


      0.2


        0


        1


      0.8


      0.6
Recall
      0.4


      02


        0

            Recall~Precision Curve


0       0.2     0.4     0.6      0.8      1
                   Recall


             Fallout-Recall Curve


0         0.05      0.1       0.15       0.2
                Fallout x 100

results - Queens College, CUNY


       Run Number          pircsl-full, automatic                              Recall-Precision Curve
      [ Num of Queries                 50                           1
       Total number of documents over all queries

          Retrieved:                       50000                             0.8
          Relevant:                        10489
          Rel~ret:                          5913

Recall Level Averages
 Recall         Precision
  0.00           0.6360
  0.10           0.4732
  0.20           0.3947
  0.30           0.3523
  0.40           0.2999
  0.50           0.2457
  0.60           0.1911
  0.70           0.1443
  0.80           0.1094
  0.90           0.0571
  1.00           0.0087
rage precision over all
~ant docs
-interpolated    0.2488

t Level Averages
it     Recall
~0     0.2840
~0     0.3622
10     0.4133
,0     0.4448
30     0.4690
)0     0.4951
~0     0.5229
~      0.5603
)0     0.5746
30     0.5853
)0     0.5942

                                                                             0.6
                                Document Level Averages               Precision
                                                Precision                    0.4
                              At 5 docs          0.4920
                              At 10 docs         0.4840
                              At 15 docs         0.4547                      0.2
                              At 20 docs         0.4420
                              At 30 docs         0.4147                       0
                              At 100 docs        0.3382
                              At 200 docs        0.2758
                              At 500 docs        0.1828
                              At 1000 docs       0.1183
                              R-Precision (precision after R
                                                                              1
                              docs  retrieved (where R   is
                                             relevant docu-

                              the number of
                              ments))

                                                                          0.8

                                                                             0      0.2      0.4      0.6     0.8


                                                                                                 Recall


                                                                                          Fallout-Recall Curve

                                        Exact                0.2950
                                                                                                0.6
                                   NORMAL DEVIATE - Fallout-Recall

                                101   0.02  0.16                 0.98  0 1~999
                                                                                                0.4
                             2                                            0.98

                                                                                                0.2
                             1                                            0.84

                             0                                            0.5                     0

                                                                                                    0           0.05         0.1          0.15          0.2
                            -1                                            0.16                                          Fallout x 100


                            -2                                            0.02

                            -3                                            0.001

                                                                   Recall

                              -3     -2    -1     0      1     2      3

esults - Queens College, CUNY

             Summary Statistics            ]
[ Run Number         pircs2-full, automatic                             1
L Num of Queries                     501
 Total number of documents over all queries
                                                                       0.8 -
 Retrieved:                       50000
 Relevant:                        10489
 Rel~ret:                          6135                                0.6

Recall- Precision Curve

~ecall Level Averages
   Recall       Precision
   0.00          0.6609
   0.10          0.4959
   0.20          0.4145
   0.30          0.3735
   0.40          0.3219
   0.50          0.2638
   0.60          0.2046
   0.70          0.1586
   0.80          0.1199
   0.90          0.0656
   1.00          0.0085
rage precision over all
~ant docs
-interpolated    0.2664 1

it Level Averages
ut      Recall
                           31
~       0.3010
`20     0.3821             2
`40     0.4263
60      0.4648             1
80      0.4954
                           0
.00     0.5175
.20     0.5603           -1
.40     0.5795
.60     0.5958           -2
.80     0.6056
                         -3
~0      0.6155

                               Document Level Averages                 Precision
                                              Precision                       0.4
                             At 5 docs         0.5440
                             At 10 docs        0.5120
                                                                              0.2
                             At 15 docs        0.4907
                             At 20 docs        0.4690                            0      0.2      0.4     0.6     0.81
                             At 30 docs        0.4460                          0
                             At 100 docs       0.3588
                             At 200 docs       0.2895                                               Recall
                             At 500 docs       0.1901
                             At 1000 docs      0.1227
                                                                                              Fallout-Recall Curve
                             R-Precision (precision after R                    1
                                            (where R    is
                                         of relevant docu-

                              docs  retrieved
                              the number
                              ments))
                              Exact               0.3155

                           NORMAL DEVIATE - Fallout-Recall

                       .001  0.02  0.16  0.5  0.84  0.98  0.~


                             4-4~r


                              I                I     I

                   ~7--

                            -3    -2    -1     0            2     3

                                                                  Recall

                                                                      I~999

                                                                         0.8


                                                                         0.6  -


                                                                         0.4

                                                                    0.98
                                                                                          0.2
                                                                    0.84

                                                                    0.5                     0
                                                                                              0          0.05         0.1         0.15         0.2
                                                                    0.16                                         Fallout x 100


                                                                    0.02

                                                                    0.001

results - VPI & SU


                      Summary Statistics           J

            Run Number       VTcms2-ful I, manual 1                                          Recall-Precision Curve
            Num of Queries                   50 J                               1
            Total number of documents over all queries 1
            Retrieved:                     50000                              0.8
            Relevant:                      19489
            Reljet:                        5898

Recall Level_Averages
   Recall      Precision
   0.00         0.7524
   0.10         0.5342
   0.20         0.4510
   0.30         0.3828
   0.40         0.3063
   0.50         0.2539
   0.60         0.1817
   0.70         0.1270
   0.80         0.0809
   0.90         0.0496
   1.00         0.0060
rage precision over all
~ant docs
-interpolated   0.2681  J

it Level Averages
Lit      Recall
00       0.3015
20       0.3795
40       0.4320
60       0.4660
BO       0.4986
DO       0.5250
20       0.5421
40       0.5577
60       0.5745
BO       0.5916
DO       0.6021

                               Document Level Averages
                                              Precision
                             At 5 docs         0.6080
                             At 10 docs        0.5760
                             At 15 docs        0.5387
                             At 20 docs        0.5160
                             At 30 docs        0.4913
                             At 100 docs       0.3806
                             At 200 docs       0.2953
                             At 500 docs       0.1882
                             At 1000 docs      0.1180
                             R-Precision (precision after R
                             docs  retrieved (where R is
                             the number of relevant docu-
                             ments))

                                                                       0.6
                                                                 Precision
                                                                       0.4


                                                                       0.2


                                                                         0


                                                                         1


                                                                       0.8

                              Exact             0.3143

                                                                           0.6
                           NORMAL DEVIATE - Fallout-Recall

                         Ol  0~02            0.84 0.98  0
                                                                           0.4


                                                                           0.2  -

                           2                                      0.98

                                                                 Recall

                                                                          0      0.2      0.4     0.6     0.8


                                                                                             Recall


                                                                                       Fallout-Recall Curve

                                                             I     I     I    I

                           1                                       0.84


                           0                                       0.5                   0

                                                                                           0          0.05        0.1         0.15        0.2
                          -1                                       0.16
                                                                                                             Fallout x 100
                          -2                                       0.02


                          -3                                       0.001
                            -3    -2    -1           1     2     3

r~q.iiltq. - Advanced Decision Systems

             Summary Statistics


maPonlYautomatioc
 Total number of documents over all queries
 Retrieved                        40423
 Relevant:                         5677
 Rel~ret:                           822

                          Document Level Averages
                                        Precision
                        At 5 docs        0.0560
                        At 10 docs       0.0480
                        At 15 docs       0.0573
                        At 20 docs       0.0640
                        At 30 docs       0.0687
                        At 100 docs      0.0468
                        At 200 docs      0.0336
                        At 500 docs      0.0239
                        At 1000 docs     0.0164
                        R-Precision (precision after R
                        docs  retrieved (where R is
                        the number of relevant docu-
                        ments))
                        Exact            0.0390

Run Number
Num of Queries

Recall Level Averages
 Recall          Precision
 0.00             0.1620
 0.10             0.0542
 0.20             0.0468
 0.30             0.0273
 0.40             0.0201
 0.50             0.0056
 0.60             0.0056
 0.70             0.0000
 0.80             0.0000
 0.90             0.0000
 1.00             0.0000
~erage precision over all
levant docs
)n-interpolated   0.0195

Precision

iout Level Averages             NORMAL DEVIATE - Fallout-Recall              Recall
lout      Recall              ooi  0.02 0.16 0.5   0.84 0.98    0.9~999
0000      0.0236
0020      0.0311            2                                      0.98
0040      0.0346
0060      0.0389            1                                      0.84
0080      0.0429
                            0                                      0.5
`0100     0.0457
0120      0.0476            -1                                     0.16
0140      0.0497
)0160     0.0526            -2                                     0.02
)0180     0.0539            -3                                     0.001
)0200     0.0553             -3    -2   -1    0      1   2       3

                Recall-Precision Curve


 1


0.8


0.6


0.4


0.2


 0
   0       0.2     0.4      0.6     0.8
                      Recall


                 Fallout-Recall Curve

 1


0.8 -


0.6


0.4


0.2


 0<
   0         0.05                0.15

   0.1
Fallout x 100

0.2

results - Advanced Decision Systems

             Summary Statistics
Run Number        ADS2-ap only, automatic                            1
    of Queries                       50
Total number of documents over all queries
                                                                   0.8
Retrieved:                        33034
Relevant:                          5671
Rel~ret:l468I                                                      0.6

Recall-Precision Curve

Recall level Averages
     Recall      Precision
     0.00         0.2199
     0.10         0.1572
     0.20         0.1304
     0.30         0.1160
     0.40         0.0885
     0.50         0.0884
     0.60         0.0114
     0.70         0.0612
     0.80         0.0362
     0.90         0.0201
     1.00         0.0196
ucrage precision over all  1
levant docs
)n-interpolated   0.0821   ]

lout Level Averages
lout      Recall
0000      0.0506
0020      0.0817             2
0040      0.1070
0060      0.1248             1
0080      0.1403
                             0
0100      0.1568
0120      0.1643            -1
0140      0.1687
0160      0.1742            -2
`0180     0.1801
                            -3
`0200     0.1813

     Document Level Averages                Precision
                    Precision                      0.4
   At 5 docs         0.1280
   At 10 docs        0.1260
                                                   0.2 -     ~
   At 15 docs        0.1320
   At 20 docs        0.1300                           0       0.2     04      0.6     081
   At 30 docs        0.1300                          0
   At 100 docs       0.1118
   At 200 docs       0.0838                                              Recall
   At 500 docs       0.0530
   At 1000 docs      0.0294
                                                                   Fallout-Recall Curve
   R-Precision ~precision after R                    1
   docs  retrieved (where R  is
              of relevant docu-
   the number
   ments))
   Exact             0.1092

NORMAL DEVIATE - Fallout-Recall
                                            Recall

0.8  -


0.6

 101 0.02                     0.98  0                      0.4


                                       0.98
                                                           0.2
                                       0.84


                                       0.5                  0
                                                              0          0.05        0.1         0.15        0.2
                                       0.16                                      Fallout x 100


                                       0.02

                                       0.001

3     -2     -1    0     1     2     3

;ults - Daihousie University

                    Summary Statistics
        Run Number     DalTx2-category B, manual 1                        1
        Num of Queries                      50
         Total number of documents over all queries
                                                                        0.8
        Retrieved:                       11987
        Relevant:                         2064
        Rel~ret:                           881                          0.6

  call Level_Averages
  ~ecall      Precision
  0.00        0.2984
  0.10        0.2029
  0.20        0.1661
  0.30        0.1415
  0.40        0.1056
  0.50        0.0714
  0.60        0.0691
  0.10        0.0488
  0.80        0.0111
  0.90        0.0002
  1.00        0.0002
  3ge precision over all
  ~nt docs
  nterpolated 0.0893 1

  Level Averages
  t   Recall
  0   0.1023
  .0  0.1559
  ~   0.2034
  0   0.2353
  ~o  0.2516
  )0  0.2135
  ~0  0.2181
  LO  0.2854
  )0  0.2890
  30  0.3151
  )0  0.3305

  Document Level Averages
                 Precision
At 5 docs        0.1320
At 10 docs       0.1280
At 15 docs       0.1301
At 20 docs       0.1310
At 30 docs       0.1213
At 100 docs      0.1040
At 200 docs      0.0645
At 500 docs      0.0334
At 1000 docs     0.0114

Precision
      0.4


      0.2


        0

          ~L Precision (precision after R
          docs  retrieved (where   R   is
          the number of relevant docu-
          ments))
          Fxact               0.1199

      NORMAL DEVIATE - Fallout-Recall                  Recall

    Ol  a.02  0.16   ( 5  0.84   0.98  0. 9~999                0.4
:1:1
                      7' 0.98
2


1                                        0.84

0                                        0.5


-1                                       0.16

-2                                       0.02


-3 -7"                                   0.001
          I     I     _______________

   3    -2    -1     0     1      2     3

               Recall-Precision Curve
            I       I       I


                                            1
   0       0.2     0.4     0.6     0.8
                      Recall


                Fallout-Recall Curve
 1            I         I


0.8


0.6


0.2

              I         I         I
 0
   0        0.05                 0.15

   0.1
Fallout x 100

0.2

~su1ts - Systems Environment Corporation

          Summary Statistics
Run Number       SEC-category B, automatic
Num of Queries                        50

]

1

 Total number of documents over all queries
Retrieved:                          19602                           0.8
Relevant:                            2064
Rel~ret:                               37

Recall-Precision Curve

`call Level_Averages
Recall        Precision
0.00           0.0213
0.10           0.0000
0.20           Q.0000
0.30           0.0000
0.40           0.0000
0.50           0.0000
0.60           0.0000
0.70           0.0000
0.80           0.0000
0.90           0.0000
1.00           0.0000
   precision over all

terpolateci~005J

Level Averages
        Recall
        0.0030
        0.0046
        0.0056
        0.0057
        0.0073
        0.0080
        0.0069
        0.0071
        0.0076
        0.0080
        0.0082

           Document Level Averages
                          Precision
        At 5 docs           0.0040
        At 10 docs          0.0020
        At 15 docs          0.0027
        At 20 docs          0.0050
        At 30 docs          0.0060
        At 100 docs         0.0038
        At 200 docs         0.0025
        At 500 docs         0.0015
        At 1000 docs        0.0007
       { R- Precision (pThecisionafterR
        docs   retrieved (where  R   is
         he number of relevant docu- j

        Exact               0.0042     ~

     NORMAL DEVIATE - Fallout-Recall

   )01 0.02   0.16   .5 0.84   0.98  0


  2                                    0.98

  1                                    0.84


  0                                    0.5

-1                                     0.16


-2                                     0.02

-3                                     0.001
  -3   -2     -1     0   1     2      3

      0.6
Precision
      0.4


      0.2


        0<                 A       A
          0       0.2     0.4      0.6     0.8      1


Recall

                      Recall


                 Fallout-Recall Curve
 1


0.8


0.6


0.4


0.2


 0
   0         0.05       0.1       0.15      0.2
                   Fallout x 100

suits - University of Central Florida

               Summary Statistics

Run Number        UCFIS1-category B, automatic                         1
Num of Queries                            50 J
  Total number of documents over all queries
                                                                      0.8
Retrieved:                             50000
Relevant:                               2064
Rel~ret:                                 102                          0.6

Recall-Precision Curve

~call Level_Averages
Recall       Precision
0.00         0.0891
0.10         0.0286
0.20         0.0062
0.30         0.0015
0.40         0.0015
0.50         0.0011
0.60         0.0000
0.10         0.0000
0.80         0.0000
0.90         0.0000
 1.00        0.0000
3ge precision over all
~nt docs~
-nterpolatedo.00601

  Document Level Averages
                 Precision
At 5 docs         0.0400
At 10 docs        0.0260
At 15 docs        0.0187
At 20 docs        0.0170
At 30 docs        0.0127
At 100 docs       0.0070
At 200 docs       0.0039
At 500 docs       0.0024
At 1000 docs      0.0020
R-Precision (precision after R
docs  retrieved (where R  is
the number of relevant docu-
ments))
Exact             0.0128   1

Precision
      0.4


      0.2


        0

0       0.2     0.4      0.6     0.8


                    Recall


              Fallout-Recall Curve

A

 1


0.8  -


0.6

A      A  _

  Level Averages
  t    Recall
  0    0.0307
  0    0.0339
  .0   0.0369
  0    0.0314
  0    0.0384
  0    0.0427
  0    0.0427
  ~    0.0427
  ~0   0.0436
  10   0.0436
  )0   0.0439

  NORMAL DEVIATE - Fallout-Recall
                                                  Recall
Ol  0.02              0.84  0.98  0.                      0.4


                                                          0.2

2                                        0.98


1                                        0.84

0                                        0.5                   0<

-1                                       0.16


-2                                       0.02

-3              I           I            0.001
  -3     -2    -1    0      1    2     3

I ~  ~  Y

0         0.05       0.1       0.15       0.2
                 Fallout x 100

results - University of Illinois at Chicago


      [


      L

                      Summary Statistics
           Run Numbe
                                                                                            Recall-Precision Curve
           NumofQueries                       50                              1
           Total number of documents over all queries
           Retrieved:                      50000
                                                                             0.8 -
           Relevant:                       10489
           Rel~ret:                         3067
                                                                             0.6
~ecall Level Averages
   Recall       Precision
   0.00          0.3965
   0.10          0.2189
   0.20          0.1543
   0.30          0.1025
   0.40          0.0807
   0.50          0.0578
   0.60          0.0361
   0.10          0.0195
   0.80          0.0088
   0.90          0.0000
   1.00          0.0000
   ~ge precision over all
   mt docs
   nterpolated   0.0780

   Level Averages
        Recall
   )    0.1231
   )    0.1693
        0.1977
        0.2278
        0.2414
        0.2664
        0.2814
        0.2918
        0.3115
        0.3272
        0.3373

                                  Document Level Averages
                                                 Precision
                               At 5 docs          0.1960
                               At 10 docs         0.2020
                               At 15 docs         0.2133
                               At 20 docs         0.2090
                               At 30 docs         0.2053
                               At 100 docs        0.1652
                               At 200 docs        0.1311
                               At 500 docs        0.0876
                               At 1000 docs       0.0613
                               R-Precision (precision after R
                               docs  retrieved (where R  is
                               the number of relevant docu-
                               ments))
                               Exact              0.1440

                            NORMAL DEVIATE - Fallout-Recall

                       ~.001  0.02  0.16  a


                       2


                       1

                       0

                       -1


                       -2

                       -3

                                                 .5    0.84  0.98   a


                                                 7

                              -3     -2    -1     0      1      2     3

                                                                   Precision
                                                                          0.4<


                                                                          0.2


                                                                           0

                                                                             0      0.2      0.4      0.6     08


                                                                                                 Recall


                                                                                          Fallout-Recall Curve

                                                                                                         I                          -

                                                                            1


                                                                           0.8 -


                                                                           0.6

                                                       ~~999               0.4


                                                        0.98
                                                                           0.2
                                                        0.84

                                                        0.5                 0

                                                                              0        0.05       0.1       0.15       0.2
                                                        0.16
                                                                                              Fallout x 100
                                                        0~02

                                                        0.001

                                                                   Recall