Experiment Design and Results for the TREC-6 Interactive Track Pre-Experiment
=============================================================================

1. Pre-experiment as designed

The TREC-6 Interactive Track experimental design was constructed to support
the comparison of multiple experimental systems, each at a different site
Each site runs its experimental system and a control system common to all 
sites. By subtracting the response on the control system from that of the 
experimental system, the design seeks to eliminate the site effect, allowing 
comparison of experimental systems at different sites without confounding of 
or contamination by site-to-site differences and without the high cost of 
running even a fraction of the experimental systems at all sites.

An effective control system is one that captures the site effect, 
which includes the influence of at least the following on the 
response: 

 - the physical surroundings in which the experiment takes place

 - the operational characteristics of the experimental system which
   are dependent on its operating environment

 - the experimenter's interaction directly and indirectly with the
   experimental participant

The purpose of running the  pre-experiment is to assess in advance
of the main experiment whether the recommended control system adequately 
captures the site effect, i.e., the extent to which it is capable of 
eliminating site-to-site differences.  In order to assess this we need 
a design which lets us compare two or more experimental systems in
two ways:

 - directly (by running multiple experimental systems per site - this 
   should be doable if the number of such systems is small compared
   to that of the main experiment) 

 - indirectly (in the manner to be used in the main experiment, i.e., 
   through use of the control running at each site)

The following designs for 2 sites is based on latin square designs 
and are similar to the design proposed for the main experiment. In the 
pre-experiments, unlike in the main experiment, the experimental 
system from a given site is run at all other sites as well.  This allows 
us to compare experimental systems directly for each site, i.e. without
any confounding due to site, and so without the need for a control
system.  

- Site i, for i = 1, ..., I sites
- P(i,j) is participant j at site i, j = 1, ..., J
- T(k)   is topic k, k = 1, ..., K topics, common to all sites
- En     is experimental system n
- Co     is the control system
- Order  is order in which systems will be run

The design for N experimental systems is a v x v latin square 
for EACH site, where v = N + 1. 

The number of sites must be at least two, but is otherwise 
unlimited."

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 sites, 2 systems:

                     Order
                  1    2    3
               _________________
               |
Site 1 P(1,1)  | E2   E1   Co

       P(1,2)  | Co   E2   E1

       P(1,3)  | E1   Co   E2

Site 2 P(2,1)  | Co   E1   E2   

       P(2,2)  | E1   E2   Co

       P(2,3)  | E2   Co   E1

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
How to run this experiment:

A. Each site randomly assigns participants to rows and topics T(1), T(2),
   and T(3) to columns (I removed topic labels from the columns to avoid
   confusion here). If both sites in a pair comes up with the same assign-,
   ment of topics, one can try again until they get a different random order.

B. The order that systems will be run is left to right as given above.

C. Concerning sites with more than the minimal 3 participants
   (ideally all), it seems desirable, because of possible carry-over effects
   from one system to the next, for sites, where possible, to try to balance 
   the number of participants assigned to each of the 6 possible system 
   sequences, which are spread over the 2 latin squares in the 2-site design. 
   This could mean using the latin square for the other site for every other 
   group of 3 participants and trying for multiples of 6 participants. 

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Features of the experiment:

A. The two latin squares given above are orthogonal latin squares and 
   have the property that each system immediately follows every other
   system twice.

B. Note that the second latin square was formed by reversing the order
   of systems from the respective row of the first square.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Proposed evaluation:

A. For each site i, estimate

	E2(i) - E1(i)  direct comparison of the experimental systems
	E2(i) - Co(i)  comparison of 2nd experimental system & the control
	E1(i) - Co(i)  comparison of 1st experimental system & the control

B. Estimate the site-INDEPENDENT comparison E2 - E1 by averaging 
   the direct site-dependent comparison E2(i) - E1(i) over all sites.

C. Estimated the site-INDEPENDENT comparison  E2 - E1 using the control
   as will be done in the main experiment:[E2(2) - Co(2)] - [E1(1) - Co(1)].

D. Assess the effectiveness of the control system in removing the site 
   effect by comparing the two estimates of the site-independent difference 
   E2 - E1 from steps 2. and 3. above.  We can estimate the "true"
   difference in the two estimates with a confidence interval.
   With only a few sites participating, the true difference will be estimated
   with fairly substantial uncertainty
.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Topics as seen by the searchers:

============
Number: 258i 
Topic: Computer Security

Description: 
Identify instances of illegal entry into sensitive computer 
networks by non-authorized personnel.
  
Please save at least one document that identifies EACH DIFFERENT 
case of illegal entry into a computer network. If one document
discusses several cases or instances, then you need not save other 
documents which mention the same ones, since your goal is to identify 
the different instances of illegal entry.

Narrative:  
Illegal entry into sensitive computer networks is a serious 
and potentially menacing problem.  Both 'hackers' and 
foreign agents have been known to acquire unauthorized entry 
into various networks. Items relevant to this subject would 
include but not be limited to instances of illegally 
entering networks containing information of a sensitive 
nature to specific countries, such as defense or technology 
information, international banking, etc. Items of a personal 
nature (e.g. credit card fraud, changing of college test 
scores) should not be considered relevant. 

============
Number: 274i 
Topic: Electric Automobiles 

Description:  
What are the latest developments in the production of electric 
automobiles? 

Please save at least one document that identifies EACH DIFFERENT recent
development in the field of electric automobiles. If one document discusses
several developments,  then you need not save other documents that
repeat those developments, since your goal is to identify the different ones that have been discussed.

Narrative: 
The economic feasibility of electric automobiles appears to 
be limited by a number of factors, including the limited range of 
operation between recharges of batteries.  What progress has been made 
in addressing these factors?

============
Number: 292i 
Topic: Worldwide Welfare

Description: 
Identify social programs for poor people in countries other 
than the U.S. 

Please save at least one document that identifies EACH DIFFERENT social 
program in EACH DIFFERENT COUNTRY. If one document discusses several 
programs in different countries, then you need not save other 
documents which mention those same programs,  since your goal is to
identify the different such programs which exist.

Narrative: 
To be relevant a document would identify a welfare program 
in a foreign country and explain how it works to aid 
citizens who have little or no income.  It would include 
those who can't work because of a disability and people who 
have the extra burden of small children.  The document 
should indicate how these people are supported or not 
supported.  A relevant document should identify the source 
of the monies used to support such welfare programs. 

-------------------------------------------------------------------------------
2. Pre-experiment as run

Two sites, Rutgers and City University, London provided experimental
systems: RU-Inquery(Ru) and Okapi(Ok), respectively. NIST provided
a control system: ZPRISE(Zp). All three sites installed and ran all
3 systems.

The following combinations of the 6 possible system sequences and
3 topic-sequences were run at the 3 sites with the specified searcher:
(Site : Searcher)

	      274  292  258         258 274 292 	     292 258 274 
	      --------------  ------------------------  -----------------------
Ok  Ru  Zp \           s001         bra		              ns8          s007
Zp  Ok  Ru  > Rutgers: s002   City: att	                NIST: ns2 Rutgers: s008
Ru  Zp  Ok /           s003  	    col 		      ns3          s009
	      --------------  ------------------------  -----------------------
Zp  Ru  Ok \			    mcf           s004	      ns4
Ru  Ok  Zp  >		      City: lea  Rutgers: s005  NIST: ns5
Ok  Zp  Ru /			    aji	          s006        ns6
	      --------------  ------------------------  -----------------------

-------------------------------------------------------------------------------
3. Results 

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
A. For each of the 3 sites, estimated

	E2(i) - E1(i)  direct comparison of the experimental systems
	E2(i) - Co(i)  comparison of 2nd experimental system & the control
	E1(i) - Co(i)  comparison of 1st experimental system & the control

Let:

E1 = Ru (Ru-Inquery)
E2 = Ok (Okapi) 
Co = Zp (ZPRISE)
		
	       Differences in             Differences in
	       Avg. Precision             Avg. Recall	
            ----------------------     ---------------------    
	    Ok-Ru   Ok-Zp   Ru-Zp      Ok-Ru   Ok-Zp   Ru-Zp    # of searchers
Sites	    ------  ------  ------     ------  ------  -----    -
-----  
City        -0.231  -0.026   0.205     0.112  -0.006  -0.118   6
  
Rutgers      0.048   0.000  -0.048     0.040   0.017  -0.023   9

NIST         0.264   0.333   0.069     0.061   0.158   0.097   6

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
B. Estimated the site-INDEPENDENT comparison E2 - E1 by averaging 
   the direct site-dependent comparison E2(i) - E1(i) over all 3 sites.

   Precision:      0.030  = 	( (6)(-0.231) + (9)0.048 + (6)0.264 ) / 21
               
   Recall:    0.067 	  =	( (6)0.112  + (9)0.040 + (6)0.061 ) / 21 
 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
C. Estimated the site-INDEPENDENT comparison  E2 - E1 using the control
   as will be done in the main experiment:[E2(2) - Co(2)] - [E1(1) - Co(1)].

   	City(Ok-Zp) - Rutgers(Ru-Zp): 	

   Precision:	   0.022  = 	-0.026 - (-0.048)

   Recall:    0.017 	  =	-0.006 - (-0.023)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
D. Assessed the effectiveness of the control system in removing the site 
   effect by comparing the two estimates of the site-independent difference 
   E2 - E1 from steps 2. and 3. above.  We can estimate the "true"
   difference in the two estimates with a confidence interval...
   With only a few sites participating, the true difference will be estimated
   with fairly substantial uncertainty...

   Confidence intervals [95% t.025,v=(n-1)=2] for "true" difference E2-E1:

    "True" E2-E1 >=    (mean of y's) - ( t(0.05/2) (stddev/sqrt(n)) )
                 and
                 <=    (mean of y's) + ( t(0.05/2) (stddev/sqrt(n)) )

    Precision:	 	 y's       	= {-0.231, 0.048, 0.264}
	                 n 		= 3
			 t(0.05/2) df=2	= 4.303
			 mean of y's 	= 0.027
			 stddev		= 0.248

	-0.589 <=  E2-E1 <= 0.643

    Recall:	 	 y's       	= {0.112, 0.040, 0.061}
	                 n 		= 3
			 t(0.05/2) df=2	= 4.303
			 mean of y's 	= 0.071
			 stddev		= 0.037

	-0.021 <=  E2-E1 <= 0.163
		
   On the one hand, the confidence intervals for the "true" difference E2-E1 
   in terms of precision and recall do contain the estimates of the 
   differences based on the control - more narrowly for recall than precision.
   On the other hand, the confidence intervals are so wide that the results of
   this small experiment provide no strong support for the effectiveness of 
   the control in isolating the site-effect.

-------------------------------------------------------------------------------
4. Supporting results data

Within-searcher system differences in precision and recall

			 Difference in		   Difference in
			 Avg. Precision    	   Avg. Recall
		        ---------------------     ---------------------
Site       Searcher     Ok-Ru   Ok-Zp   Ru-Zp     Ok-Ru   Ok-Zp   Ru-Zp

city       bra         -0.750   0.000   0.750    -0.755  -0.275   0.480 
city       att          0.700   0.150  -0.550     0.604   0.049  -0.555
city       col         -0.446  -0.596  -0.150     0.198  -0.116  -0.314
city       mcf         -1.000  -0.250   0.750    -0.091  -0.385  -0.294
city       lea         -0.286   0.714   1.000     0.237   0.545   0.308
city       aji          0.395  -0.176  -0.571     0.478   0.147  -0.331
		        -----   -----   -----  	  -----   -----   -----
mean		       -0.231   0.026   0.204     0.111  -0.006  -0.118 
std			0.658   0.442   0.710	  0.488   0.334   0.411

rutgers    s001         0.175  -0.100  -0.275     0.384  -0.007  -0.391
rutgers    s002         0.000  -0.800  -0.800     0.000  -0.636  -0.636
rutgers    s003        -0.067  -0.043   0.024    -0.014  -0.066  -0.052
rutgers    s004         0.077   0.077   0.000     0.071   0.071   0.000
rutgers    s005         0.031   0.657   0.626     0.265   0.656   0.391
rutgers    s006         0.381  -0.119  -0.500     0.176  -0.356  -0.532
rutgers    s007        -0.200  -0.208  -0.008    -0.483  -0.169   0.314
rutgers    s008        -0.786   0.047   0.833    -0.769   0.160   0.929
rutgers    s009         0.818   0.485  -0.333     0.727   0.496  -0.231
		        -----   -----   -----  	  -----   -----   -----	
mean		        0.048   0.000  -0.048     0.040   0.016  -0.023  
std			0.432   0.416   0.517     0.446   0.398   0.500

nist       ns8         -0.500  -0.833  -0.333    -0.324  -0.513  -0.189
nist       ns2          0.000   1.000   1.000    -0.328   0.308   0.636
nist       ns3          0.917   1.000   0.083     0.422   0.636   0.214
nist       ns4          1.000   1.000   0.000     0.455   0.455   0.000
nist       ns5          0.667   0.167  -0.500     0.231   0.140  -0.091
nist       ns6         -0.500  -0.333   0.167    -0.091  -0.077   0.014
		        -----   -----   -----  	  -----   -----   -----
mean		        0.264   0.333   0.069     0.061   0.158   0.097
std			0.688   0.796   0.523     0.357   0.411   0.296

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
All measures by search

Site	
|	 Searcher
|	 |     System
|	 |     |     System Sequence
|	 |     |     |   Topic
|	 |     |     |   |    Search	
|	 |     |     |	 |    | 	  Number of docs saved
|	 |     |     |	 |    | 	  |    Elapsed time (secs)
|	 |     |     |	 |    |		  |    |     Precision
|	 |     |     |	 |    |	    	  |    |     |     Recall
|	 |     |     |	 |    |	       	  |    |     |     |    Covered aspects
|        |     |     |   |    |           |    |     |     |     | 1 bit per,
								   leftmost =#1
city     aji   ok    1   258  a31         13   1172  0.538 0.692 1101110111010
city     aji   ru    3   292  a33          7   1212  0.143 0.214 00010011000000
city     aji   zp    2   274  a32          7   1204  0.714 0.545 11100111000
city     att   ok    2   274  b22         20   1186  0.900 0.818 11111011110
city     att   ru    3   292  b23          5   1197  0.200 0.214 00010011000000
city     att   zp    1   258  b21          8   1203  0.750 0.769 1011111011110
city     bra   ok    1   258  b11          8   1197  0.250 0.154 0000000001001
city     bra   ru    2   274  b12         13   1203  1.000 0.909 11110111111
city     bra   zp    3   292  b13         16   1245  0.250 0.429 00011111100000
city     col   ok    3   292  b33         13   1141  0.154 0.429 11110011000000
city     col   ru    1   258  b31          5   1096  0.600 0.231 1000000101000
city     col   zp    2   274  b32          4   1133  0.750 0.545 11110001100
city     lea   ok    2   274  a22          7   1136  0.714 0.545 11110001100
city     lea   ru    1   258  a21          2   1118  1.000 0.308 0001110000010
city     lea   zp    3   292  a23          6   1167  0.000 0.000 00000000000000
city     mcf   ok    3   292  a13         17    994  0.000 0.000 00000000000000
city     mcf   ru    2   274  a12          1   1182  1.000 0.091 10000000000
city     mcf   zp    1   258  a11          8    998  0.250 0.385 1010001001100
nist     ns2   ok    2   258  ns2-ok-258   2   1200  1.000 0.308 0001110010000
nist     ns2   ru    3   274  ns2-ru-274   6   1290  1.000 0.636 11110001110
nist     ns2   zp    1   292  ns2-zp-292   3   1200  0.000 0.000 00000000000000
nist     ns3   ok    3   274  ns3-ok-274   6   1200  1.000 0.636 11111001100
nist     ns3   ru    1   292  ns3-ru-292  12   1200  0.083 0.214 00010011000000
nist     ns3   zp    2   258  ns3-zp-258   1    660  0.000 0.000 0000000000000
nist     ns4   ok    3   274  ns4-ok-274   4   1200  1.000 0.455 10100001110
nist     ns4   ru    2   258  ns4-ru-258   1   1200  0.000 0.000 0000000000000
nist     ns4   zp    1   292  ns4-zp-292   4   1200  0.000 0.000 00000000000000
nist     ns5   ok    2   258  ns5-ok-258   3   1200  0.667 0.231 1000001001000
nist     ns5   ru    1   292  ns5-ru-292   3   1200  0.000 0.000 00000000000000
nist     ns5   zp    3   274  ns5-zp-274   2   1200  0.500 0.091 10000000000
nist     ns6   ok    1   292  ns6-ok-292   0   1200  0.000 0.000 00000000000000
nist     ns6   ru    3   274  ns6-ru-274   2   1265  0.500 0.091 10000000000
nist     ns6   zp    2   258  ns6-zp-258   3   1200  0.333 0.077 0000000000001
nist     ns8   ok    1   292  ns8-ok-292   6   1210  0.167 0.214 00010011000000
nist     ns8   ru    2   258  ns8-ru-258   9   1205  0.667 0.538 1001110101010
nist     ns8   zp    3   274  ns8-zp-274   7   1187  1.000 0.727 11111011100
rutgers  s001  ok    1   274  s001.t274   10   1237  0.300 0.455 10100011100
rutgers  s001  ru    2   292  s001.t292    8   1262  0.125 0.071 00000000100000
rutgers  s001  zp    3   258  s001.t258   10   1235  0.400 0.462 1010001001101
rutgers  s002  ok    2   292  s002.t292   11   1000  0.000 0.000 00000000000000
rutgers  s002  ru    3   258  s002.t258    7   1290  0.000 0.000 0000000000000
rutgers  s002  zp    1   274  s002.t274    5   1337  0.800 0.636 11111001010
rutgers  s003  ok    3   258  s003.t258   10   1077  0.100 0.077 0000000000001
rutgers  s003  ru    1   274  s003.t274    6   1360  0.167 0.091 10000000000
rutgers  s003  zp    2   292  s003.t292   14   1171  0.143 0.143 00000000001001
rutgers  s004  ok    3   292  s004.t292   13   1904  0.077 0.071 00010000000000
rutgers  s004  ru    2   274  s004.t274    1   1363  0.000 0.000 00000000000
rutgers  s004  zp    1   258  s004.t258    1   1212  0.000 0.000 0000000000000
rutgers  s005  ok    2   274  s005.t274   10   1320  0.800 0.727 11111001110
rutgers  s005  ru    1   258  s005.t258   13   1610  0.769 0.462 0011110011000
rutgers  s005  zp    3   292  s005.t292    7   1178  0.143 0.071 00010000000000
rutgers  s006  ok    1   258  s006.t258    7   1229  0.714 0.462 1001110101000
rutgers  s006  ru    3   292  s006.t292    3    700  0.333 0.286 00010100110000
rutgers  s006  zp    2   274  s006.t274    6   1169  0.833 0.818 11111111100
rutgers  s007  ok    1   292  s007.t292   12   1284  0.167 0.286 00010000001110
rutgers  s007  ru    2   258  s007.t258   30   1014  0.367 0.769 1011111011110
rutgers  s007  zp    3   274  s007.t274    8   1094  0.375 0.455 11010001001
rutgers  s008  ok    2   258  s008.t258   14   1418  0.214 0.231 0001000011000
rutgers  s008  ru    3   274  s008.t274   15   1402  1.000 1.000 11111111111
rutgers  s008  zp    1   292  s008.t292    6   1296  0.167 0.071 00010000000000
rutgers  s009  ok    3   274  s009.t274   11   1263  0.818 0.727 11110001111
rutgers  s009  ru    1   292  s009.t292    5   1398  0.000 0.000 00000000000000
rutgers  s009  zp    2   258  s009.t258    3    992  0.333 0.231 0001110000000

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Precision and Recall 
by searcher, topic, and system

Precision:

Searcher
 |
 | 	258			274			292
 |	---------------------   ---------------------   ---------------------
 |	ok	ru	zp	ok	ru	zp	ok	ru	zp
  	-----   -----   -----   -----   -----   -----   -----   -----   -----
bra     0.250                           1.000                           0.250
att                     0.750   0.900                           0.200     
col             0.600                           0.750   0.154             

mcf                     0.250           1.000           0.000             
lea             1.000           0.714                                   0.000
aji     0.538                                   0.714           0.143     

        ---------------------   ---------------------   ---------------------
s001                    0.400   0.300                           0.125          
s002            0.000                           0.800   0.000              
s003    0.100                           0.167                           0.143

s004                    0.000           0.000           0.077              
s005            0.769           0.800                                   0.143
s006    0.714                                   0.833           0.333      

s007            0.367                           0.375   0.167              
s008    0.214                           1.000                           0.167
s009                    0.333   0.818                           0.000      
        ---------------------   ---------------------   ---------------------
ns8             0.667                           1.000   0.167              
ns2     1.000                           1.000                           0.000
ns3                     0.000   1.000                           0.083      

ns4             0.000           1.000                                   0.000
ns6                     0.333           0.500           0.000              
ns5     0.667                                   0.500           0.000      
        ---------------------   ---------------------   ---------------------

Recall: 

	258			274			292
	---------------------   ---------------------   ---------------------
	ok	ru	zp	ok	ru	zp	ok	ru	zp
	-----   -----   -----   -----   -----   -----   -----   -----   -----
bra     0.154  		      	 	0.909  				0.429
att     		0.769   0.818  				0.214
col     	0.231  	      			0.545   0.429
mcf    			0.385  		0.091  		0.000
lea     	0.308  		0.545  					0.000
aji     0.692                                   0.545           0.214

        ---------------------   ---------------------   ---------------------
s001        		0.462   0.455  				0.071
s002    	0.000  				0.636   0.000
s003    0.077  				0.091  				0.143
s004    		0.000  		0.000  		0.071
s005    	0.462  		0.727  					0.071
s006    0.462  					0.818  		0.286
s007    	0.769  				0.455  	0.286
s008    0.231  				1.000  				0.071
s009    		0.231   0.727  				0.000
        ---------------------   ---------------------   ---------------------
ns8     	0.538  				0.727  	0.214
ns2     0.308  				0.636  				0.000
ns3     		0.000   0.636  				0.214
ns4     	0.000  		0.455  					0.000
ns5     0.231  					0.091  		0.000
ns6     		0.077  		0.091  		0.000

        ---------------------   ---------------------   ---------------------

-------------------------------------------------------------------------------
5. Various higher-level views of the data for exploration

 
Precision and recall 
by site 
across searchers within site, systems, and topics

		Precision		Recall		
		---------------- 	-------------   
		mean	std		mean	std	# of searches
		-----   -----           -----   -----   --
City		.512	.357		.404	.278	18
Rutgers		.339	.316		.319	.302	27
Nist		.440    .424		.234	.256	18

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Precision and recall 
by system 
across sites, searchers within sites, and topics

ok         	0.456   0.358 
ru         	0.426   0.292 
zp         	0.369   0.307

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Precision and recall 
by system sequence number 
across sites, searchers, systems and topics

1          	0.320   0.259 
2          	0.503   0.362 
3          	0.428   0.337

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Precision and recall 
by topic 
across sites, searchers within sites, and systems

258       	0.426   0.304 
274        	0.722   0.524 
292        	0.102   0.129

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Precision and recall 
by site and system 
across topics and searchers within sites

                Precision                       Recall                    
					                             
		Ok	Ru	Zp	        Ok      Ru      Zp   
					                             
City    	0.426 	0.657 	0.452 	        0.440   0.328   0.446
					                             
Rutgers         0.354 	0.307 	0.355 	        0.337   0.298   0.321
					                             
NIST            0.639 	0.375 	0.305 	        0.307   0.247   0.149

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Precision and recall 
by system and topic 
across sites and searchers within sites

	        Precision			Recall                        
                                                                       
        	258	274 	292 		258     274     292  

  ok   		0.498	0.790   0.081		0.308  	0.623   0.143

  ru   		0.486	0.667	0.126		0.330  	0.403   0.143
							               
  zp   		0.295	0.710	0.100		0.275  	0.545   0.102

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Precision and recall 
by site and topic 
across systems and searchers within sites

	Precision

		258	274 	292             258	274 	292            
					                             
 city   	0.565	0.846	0.124	        0.423   0.576   0.214
					                             
rutgers   	0.322	0.566	0.128	        0.299   0.545   0.111
					                             
 nist   	0.444	0.833	0.042	        0.192   0.439   0.071

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Precision and recall 
by searcher within site
across various system-x-topic combinations

		Precision
Searcher		Recall

bra            	0.500   0.497
att            	0.617   0.600 
col            	0.501   0.402  
mcf            	0.417   0.159 
lea            	0.571   0.284 
aji            	0.465   0.484 

s001           	0.275   0.329 
s002           	0.267   0.212 
s003           	0.137   0.104 
s004           	0.026   0.024 
s005           	0.571   0.420 
s006           	0.627   0.522 
s007           	0.303   0.503 
s008           	0.460   0.434 
s009           	0.384   0.319

ns8            	0.611   0.493 
ns2            	0.667   0.315 
ns3            	0.361   0.283 
ns4            	0.333   0.152 
ns5            	0.389   0.107 
ns6            	0.278   0.056 

-------------------------------------------------------------------------------
6. Assessment results 

Topic-document-aspect matrix (aspectual content as assessed by NIST)

Topic
|    TREC Document #
|    |	         Covered aspects 1 bit per possible aspect; 
|    |		 |	 	"1" = "covered by this document"; 
|    |		 |	 	leftmost bit = Aspect #1
|    |           | 
258  FT911-3918	 0000000000000
258  FT921-10278 0000000000000
258  FT921-11682 0000000000000
258  FT921-7240	 0000000000000
258  FT922-13695 0000000000000
258  FT922-2176	 1100000000000 
258  FT922-2486	 0000000000000
258  FT922-2528	 0000000000000
258  FT922-468	 0000000000000
258  FT922-5072	 0000000000000
258  FT923-10088 0000000000000
258  FT923-11228 0000000000000
258  FT923-14087 0000000000000
258  FT923-2796  0000000000000
258  FT923-449	 0010001000100
258  FT923-8456	 0000000000000
258  FT924-13926 0000000000000
258  FT924-5109	 0000000000000
258  FT924-8561	 0000000000000
258  FT931-11384 0000000000000
258  FT931-1338	 0000000000000
258  FT931-1831	 0001000000000
258  FT931-2005	 0000000000000
258  FT931-2806	 0001110000000
258  FT931-2937	 0001010000000
258  FT931-2938  0001010000000
258  FT931-3482	 0000001000000
258  FT931-7090	 0001000000000
258  FT931-7297	 0000010000000
258  FT931-7553	 0001110000000
258  FT932-14128 1000000100000
258  FT932-2922	 0000000000000	
258  FT932-3590	 0000000000000
258  FT932-7637	 0010100011000
258  FT932-7643	 0010100011000
258  FT932-7897	 0000000010000
258  FT932-9585	 0000000000000	
258  FT933-10240 0000000000000
258  FT933-11258 0000000000000
258  FT933-13340 0000000000000
258  FT933-13759 0000000000000
258  FT933-8055	 0000000000000
258  FT933-9071	 0000000000000
258  FT934-12464 0000000000000
258  FT934-13338 0000000000000
258  FT934-15659 0000000000000
258  FT934-5580	 0000000000000
258  FT934-6622	 0000000000000
258  FT941-11158 0000000000000
258  FT941-4379	 0000000000000
258  FT941-6108	 0000000000000
258  FT942-2223	 0000000000000
258  FT942-5241	 0000000000000
258  FT942-578	 0000000000000
258  FT942-7826	 0000000000000
258  FT942-7890	 0000000000000
258  FT943-1215	 0000000000000
258  FT943-12981 0000000000000
258  FT943-5307	 0000000000000
258  FT943-9656	 0000000000000
258  FT944-11064 0000000000000
258  FT944-15438 0000000000000
258  FT944-15440 0000000000000
258  FT944-16521 0000000000010
258  FT944-1726	 0000000000001
258  FT944-7040	 1000000001000
258  FT944-7314	 0000000001000
258  FT944-7649	 0000000001000
258  FT944-9024	 0000000000000
258  FT944-9459	 0000000000000
274  FT921-15508 11111000000
274  FT921-1556	 11000111000
274  FT921-15760 00000000000
274  FT921-2636	 00000000000
274  FT921-5193	 01100001100
274  FT921-5494	 00000000000
274  FT923-10730 01100001000
274  FT923-11292 10000000000
274  FT923-13374 00000000000
274  FT923-14293 10000000000
274  FT924-2212	 00100001000
274  FT924-2358	 00000010100
274  FT924-5561	 00000000010
274  FT924-6107	 00000000000
274  FT924-7255	 00000000000
274  FT931-12869 00000001000
274  FT931-13738 00000000000
274  FT931-14618 10000001000
274  FT931-14778 10000001000
274  FT932-11975 00000000000
274  FT933-11925 00000000000
274  FT933-12570 00000000000
274  FT934-14844 10100000000
274  FT934-2530	 00000000000
274  FT934-4790	 00000000000
274  FT934-4995	 00000000000
274  FT934-9597	 00100000000
274  FT941-13218 00000000000
274  FT941-16334 00000000000
274  FT941-16609 10100000010
274  FT941-17640 00100001100
274  FT941-7609	 00000001000
274  FT941-9725	 00000000000
274  FT942-12506 00000000000
274  FT942-12711 00000000000
274  FT942-15407 00000000001
274  FT942-15408 00000000001
274  FT942-9950	 10000000000
274  FT943-11205 00000000000
274  FT943-11627 00100000100
274  FT943-11755 00000000000
274  FT943-11934 10000001110
274  FT943-14882 00000000000
274  FT943-16085 00000000000
274  FT943-3737	 00000000000
274  FT943-8478	 10000000000
274  FT943-985	 00000000000
274  FT944-16551 00000000000
274  FT944-16970 10010001000
274  FT944-17283 01000000000
274  FT944-17513 11010001000
274  FT944-18127 10100000000
274  FT944-18230 11100001100
274  FT944-3280	 11010001000
274  FT944-4751	 01000000100
292  FT911-1070	 00000000000000
292  FT911-1691	 00000000000000
292  FT921-10164 00000000000000
292  FT922-1215	 00000000000000
292  FT922-12153 00000000000000
292  FT922-1568	 11100000000000
292  FT922-2754	 00000000000000
292  FT922-4425	 00010000000000
292  FT922-5307	 00000000000000
292  FT922-5936	 00000000000000
292  FT922-7329	 00000000000000
292  FT922-770	 00000000000000
292  FT922-9180	 00000000000000
292  FT922-9224	 00000000000000
292  FT922-9330	 00000000000000
292  FT923-2405	 00000000000000
292  FT923-2790	 00000000000000
292  FT923-289	 00001100000000
292  FT923-8922	 00000000000000
292  FT923-9849	 00000000000000
292  FT924-10895 00000000000000
292  FT924-1107	 00000000000000
292  FT924-11314 00010011000000
292  FT924-1207	 00000000000000
292  FT924-1255	 00000000000000
292  FT924-12824 00010100110000
292  FT924-14680 00000000000000
292  FT924-15607 00000000000000
292  FT924-2474	 00000000000000
292  FT924-4469	 00000000000000
292  FT931-1043	 00000000000000
292  FT931-10935 00000000000000
292  FT931-14988 00000000000000
292  FT931-15653 00000000000000
292  FT931-258	 00000000000000
292  FT931-2783	 00000000000000
292  FT931-3537	 00000000000000
292  FT931-7362	 00000000000000
292  FT931-9432	 00000000000000
292  FT932-1015	 00000000000000
292  FT932-11049 00000000000000
292  FT932-12315 00000000000000
292  FT932-12747 00000000000000
292  FT932-12865 00000000000000
292  FT932-12991 00000000000000
292  FT932-3822	 00000000000000
292  FT932-4859	 00000000000000
292  FT932-4864	 00000000000000
292  FT932-5425	 00000000000000
292  FT932-5668	 00000000000000
292  FT932-6510	 00000000000000
292  FT932-7223	 00000000000000
292  FT933-10651 00000000000000
292  FT933-14061 00000000000000
292  FT933-1422	 00000000000000
292  FT933-14927 00000000000000
292  FT933-15869 00000000000000
292  FT933-2471	 00000000000000
292  FT933-249	 00000000000000
292  FT933-6509	 00000000000000
292  FT933-9986	 00010000000000
292  FT934-10045 00000000000000
292  FT934-10072 00000000000000
292  FT934-10231 00000000000000
292  FT934-1035	 00000000000000
292  FT934-10921 00000000000000
292  FT934-11014 00000000000000
292  FT934-11906 00000000000000
292  FT934-12730 00000000001100
292  FT934-12800 00000000000000
292  FT934-13781 00000000000000
292  FT934-2265	 00010000001010
292  FT934-2293	 00000000000000	
292  FT934-2514	 00000000000000
292  FT934-4297	 00000000000000
292  FT934-6695	 00000000000000
292  FT934-7624	 00000000000000
292  FT934-8104	 00000000000000
292  FT934-8863	 00000000000000
292  FT941-11779 00000000000000
292  FT941-12323 00000000000000
292  FT941-12334 00000000000000
292  FT941-12373 00000000000000
292  FT941-12426 00000000000000
292  FT941-16163 00000000000000
292  FT941-5213	 00000000000000
292  FT941-5683	 00000000000000
292  FT941-8918	 00000000000000
292  FT942-10229 00000000000000
292  FT942-13791 00000000000000
292  FT942-15092 00000000000000
292  FT942-3981	 00000000000000
292  FT942-4141	 00000000000000
292  FT942-5248	 00000000000000
292  FT942-5651	 00000000000000
292  FT942-5904	 00000000000000
292  FT943-14063 00000000000000
292  FT943-14531 00000000000000
292  FT943-15578 00000000000001
292  FT943-1684	 00000000100000
292  FT943-1865	 00000000000000
292  FT943-2556	 00000000000000
292  FT943-274	 00000000000000
292  FT943-3011	 00000000000000
292  FT943-3430	 00000000001000
292  FT943-5084	 00000000000000
292  FT943-8923	 00000000000000
292  FT944-10159 00000000000000
292  FT944-15084 00000000000000
292  FT944-2202	 00000000000000
292  FT944-6246	 00000000000000
292  FT944-642	 00000000000000
292  FT944-6725	 00000000000000
292  FT944-8202	 00000000000000
292  FT944-9133	 00000000000000
292  FT944-9190	 00000000000000
292  FT944-9193	 00000000000000

- - - - - - - - - - - - - - - - - 
Aspect glosses from NIST assessors

Topic   
|	Aspect #
|	|	Descriptive Phrase

258 	1	Prince Philips Electronic Mailbox
258 	2	Earth to Space Shuttle Atlantis Comm's
258 	3	NASA
258 	4	Financial Times
258 	5	Headquarters of the European Community (EC)
258 	6	European Organization for Cancer Research/Treatment
258 	7	Pentagon
258 	8	CERN Nuclear Research Lab in Geneva
258 	9	Bank, SG Warburg
258 	10	British Telecommunications
258 	11	NATO
258 	12	Hospital Computer System
258 	13	General Electric (U.S.) Sensitive Material

274 	1	governmental requirements/incentives
274 	2	manufacturers' decision to produce/sell
274 	3	commercial trials
274 	4	improved battery pack
274 	5	quick recharge facility
274 	6	lightweight materials
274 	7	low-drag aerodynamics/tires
274 	8	industrial collaboration
274 	9	hybrid-elect-petrol
274 	10	govt.-industrial collaboration
274 	11	fuel-cell technology

292 	1	Cuts in Capital Spending
292 	2	Making Poor Pay For Medicines
292 	3	Free Medical Treatment
292 	4	Pensions (Monthly or Weekly Income For Poor)
292 	5	Annual Income Support Entitlement (Unemployment Comp)
292 	6	Social Security (For Poor)
292 	7	Community-Related Work
292 	8	Entitlement (Monies for Collateral-Loan)
292 	9	Child Benefit
292 	10	No Benefit (For Those With Sufficient Capital)
292 	11	Unemployment -  Received 2/3 of Former Salary For 32 Mo.
292 	12	Self-Financing / Contribution-Driven
292 	13	Percentage of GNP
292 	14	Family Credit

-------------------------------------------------------------------------------
7. Search data submitted for evaluation

site    search          searcher 
			     system 
				system-sequence 
				  topic 
				      time (secs) 
						 doc-seq 
						   doc

city 	a11 		mcf  zp 1 258 998 	 1 FT941-6108
city 	a11 		mcf  zp 1 258 998 	 2 FT944-15438
city 	a11 		mcf  zp 1 258 998 	 3 FT944-7040
city 	a11 		mcf  zp 1 258 998 	 4 FT911-3918
city 	a11 		mcf  zp 1 258 998 	 5 FT934-12464
city 	a11 		mcf  zp 1 258 998 	 6 FT923-449
city 	a11 		mcf  zp 1 258 998 	 7 FT922-2528
city 	a11 		mcf  zp 1 258 998 	 8 FT924-5109
city 	a12 		mcf  ru 2 274 1182	 1 FT923-11292
city 	a13 		mcf  ok 3 292 994 	 1 FT944-6246
city 	a13 		mcf  ok 3 292 994 	 2 FT923-8922
city 	a13 		mcf  ok 3 292 994 	 3 FT931-2783
city 	a13 		mcf  ok 3 292 994 	 4 FT933-14061
city 	a13 		mcf  ok 3 292 994 	 5 FT931-7362
city 	a13 		mcf  ok 3 292 994 	 6 FT932-12747
city 	a13 		mcf  ok 3 292 994 	 7 FT922-5936
city 	a13 		mcf  ok 3 292 994 	 8 FT931-14988
city 	a13 		mcf  ok 3 292 994 	 9 FT941-12373
city 	a13 		mcf  ok 3 292 994 	10 FT923-2790
city 	a13 		mcf  ok 3 292 994 	11 FT933-2471
city 	a13 		mcf  ok 3 292 994 	12 FT934-13781
city 	a13 		mcf  ok 3 292 994 	13 FT943-1865
city 	a13 		mcf  ok 3 292 994 	14 FT922-9224
city 	a13 		mcf  ok 3 292 994 	15 FT941-16163
city 	a13 		mcf  ok 3 292 994 	16 FT933-14927
city 	a13 		mcf  ok 3 292 994 	17 FT943-14063
city 	a21 		lea  ru 1 258 1118       9 FT944-16521
city 	a21 		lea  ru 1 258 1118	10 FT931-7553
city 	a22 		lea  ok 2 274 1136 	 1 FT943-8478
city 	a22 		lea  ok 2 274 1136 	 2 FT931-12869
city 	a22 		lea  ok 2 274 1136 	 3 FT941-17640
city 	a22 		lea  ok 2 274 1136 	 4 FT931-13738
city 	a22 		lea  ok 2 274 1136 	 5 FT944-16551
city 	a22 		lea  ok 2 274 1136 	 6 FT944-3280
city 	a22 		lea  ok 2 274 1136 	 7 FT934-14844
city 	a23 		lea  zp 3 292 1167 	 1 FT934-12800
city 	a23 		lea  zp 3 292 1167 	 2 FT934-10921
city 	a23 		lea  zp 3 292 1167 	 3 FT934-8104
city 	a23 		lea  zp 3 292 1167 	 4 FT934-4297
city 	a23 		lea  zp 3 292 1167 	 6 FT942-5651
city 	a23 		lea  zp 3 292 1167  	 5 FT922-12153
city 	a31 		aji  ok 1 258 1172	 6 FT922-2528
city 	a31 		aji  ok 1 258 1172 	 1 FT944-7649
city 	a31 		aji  ok 1 258 1172 	 2 FT933-8055
city 	a31 		aji  ok 1 258 1172 	 3 FT932-7897
city 	a31 		aji  ok 1 258 1172 	 4 FT931-7553
city 	a31 		aji  ok 1 258 1172 	 5 FT911-3918
city 	a31 		aji  ok 1 258 1172 	 7 FT944-7040
city 	a31 		aji  ok 1 258 1172 	 8 FT942-578
city 	a31 		aji  ok 1 258 1172 	 9 FT931-2005
city 	a31 		aji  ok 1 258 1172 	10 FT943-1215
city 	a31 		aji  ok 1 258 1172 	11 FT922-2176
city 	a31 		aji  ok 1 258 1172 	12 FT932-14128
city 	a31 		aji  ok 1 258 1172 	13 FT944-16521
city 	a32 		aji  zp 2 274 1204 	 1 FT921-1556
city 	a32 		aji  zp 2 274 1204 	 2 FT931-14778
city 	a32 		aji  zp 2 274 1204 	 3 FT931-14618
city 	a32 		aji  zp 2 274 1204 	 4 FT944-17283
city 	a32 		aji  zp 2 274 1204 	 5 FT941-16334
city 	a32 		aji  zp 2 274 1204 	 6 FT942-12711
city 	a32 		aji  zp 2 274 1204 	 7 FT924-2212
city 	a33 		aji  ru 3 292 1212 	 1 FT941-12426
city 	a33 		aji  ru 3 292 1212 	 2 FT942-5651
city 	a33 		aji  ru 3 292 1212 	 3 FT922-12153
city 	a33 		aji  ru 3 292 1212 	 4 FT924-11314
city 	a33 		aji  ru 3 292 1212 	 5 FT944-6725
city 	a33 		aji  ru 3 292 1212 	 6 FT932-3822
city 	a33 		aji  ru 3 292 1212 	13 FT931-1043
city 	b11 		bra  ok 1 258 1197 	 1 FT944-7314
city 	b11 		bra  ok 1 258 1197 	 2 FT943-1215
city 	b11 		bra  ok 1 258 1197 	 3 FT944-1726
city 	b11 		bra  ok 1 258 1197 	 4 FT922-2528
city 	b11 		bra  ok 1 258 1197 	 5 FT942-5241
city 	b11 		bra  ok 1 258 1197 	 6 FT932-2922
city 	b11 		bra  ok 1 258 1197 	 7 FT933-13340
city 	b11 		bra  ok 1 258 1197 	 8 FT933-10240
city 	b12 		bra  ru 2 274 1203 	26 FT942-9950
city 	b12 		bra  ru 2 274 1203 	29 FT942-15407
city 	b12 		bra  ru 2 274 1203 	41 FT944-18230
city 	b12 		bra  ru 2 274 1203 	42 FT943-11934
city 	b12 		bra  ru 2 274 1203 	45 FT921-1556
city 	b12 		bra  ru 2 274 1203 	53 FT943-8478
city 	b12 		bra  ru 2 274 1203 	54 FT944-16970
city 	b12 		bra  ru 2 274 1203 	55 FT943-11627
city 	b12 		bra  ru 2 274 1203 	56 FT944-3280
city 	b12 		bra  ru 2 274 1203 	57 FT944-4751
city 	b12 		bra  ru 2 274 1203 	58 FT941-17640
city 	b12 		bra  ru 2 274 1203 	59 FT924-2358
city 	b12 		bra  ru 2 274 1203 	60 FT923-11292
city 	b13 		bra  zp 3 292 1245 	 1 FT943-1684
city 	b13 		bra  zp 3 292 1245 	 2 FT942-5248
city 	b13 		bra  zp 3 292 1245 	 3 FT934-8104
city 	b13 		bra  zp 3 292 1245 	 4 FT934-10921
city 	b13 		bra  zp 3 292 1245 	 5 FT934-7624
city 	b13 		bra  zp 3 292 1245 	 6 FT944-9193
city 	b13 		bra  zp 3 292 1245 	 7 FT942-10229
city 	b13 		bra  zp 3 292 1245 	 8 FT934-2293
city 	b13 		bra  zp 3 292 1245 	 9 FT922-5307
city 	b13 		bra  zp 3 292 1245 	10 FT942-5651
city 	b13 		bra  zp 3 292 1245 	11 FT922-12153
city 	b13 		bra  zp 3 292 1245 	12 FT924-11314
city 	b13 		bra  zp 3 292 1245 	13 FT923-289
city 	b13 		bra  zp 3 292 1245 	14 FT931-1043
city 	b13 		bra  zp 3 292 1245 	15 FT922-4425
city 	b13 		bra  zp 3 292 1245 	16 FT944-10159
city 	b21 		att  zp 1 258 1203 	 1 FT931-2806
city 	b21 		att  zp 1 258 1203 	 2 FT932-7643
city 	b21 		att  zp 1 258 1203 	 3 FT934-12464
city 	b21 		att  zp 1 258 1203 	 4 FT944-7040
city 	b21 		att  zp 1 258 1203 	 5 FT923-449
city 	b21 		att  zp 1 258 1203 	 6 FT944-16521
city 	b21 		att  zp 1 258 1203 	 7 FT921-11682
city 	b21 		att  zp 1 258 1203 	 8 FT932-7897
city 	b22 		att  ok 2 274 1186 	 1 FT923-10730
city 	b22 		att  ok 2 274 1186 	 2 FT941-7609
city 	b22 		att  ok 2 274 1186 	 3 FT924-5561
city 	b22 		att  ok 2 274 1186 	 4 FT943-8478
city 	b22 		att  ok 2 274 1186 	 5 FT934-9597
city 	b22 		att  ok 2 274 1186 	 6 FT943-3737
city 	b22 		att  ok 2 274 1186 	 7 FT931-13738
city 	b22 		att  ok 2 274 1186 	 8 FT931-14618
city 	b22 		att  ok 2 274 1186 	 9 FT931-14778
city 	b22 		att  ok 2 274 1186 	10 FT944-18230
city 	b22 		att  ok 2 274 1186 	11 FT941-17640
city 	b22 		att  ok 2 274 1186 	12 FT924-2358
city 	b22 		att  ok 2 274 1186 	13 FT921-15508
city 	b22 		att  ok 2 274 1186 	14 FT944-3280
city 	b22 		att  ok 2 274 1186 	15 FT931-12869
city 	b22 		att  ok 2 274 1186 	16 FT921-5193
city 	b22 		att  ok 2 274 1186 	17 FT923-11292
city 	b22 		att  ok 2 274 1186 	18 FT944-4751
city 	b22 		att  ok 2 274 1186 	19 FT944-16970
city 	b22 		att  ok 2 274 1186 	20 FT934-14844
city 	b23 		att  ru 3 292 1197 	 4 FT932-12865
city 	b23 		att  ru 3 292 1197 	 5 FT911-1691
city 	b23 		att  ru 3 292 1197 	 9 FT911-1070
city 	b23 		att  ru 3 292 1197 	10 FT924-11314
city 	b23 		att  ru 3 292 1197 	12 FT943-14531
city 	b31 		col  ru 1 258 1096 	 1 FT944-7040
city 	b31 		col  ru 1 258 1096 	 2 FT922-13695
city 	b31 		col  ru 1 258 1096 	12 FT944-7649
city 	b31 		col  ru 1 258 1096 	15 FT942-578
city 	b31 		col  ru 1 258 1096 	16 FT932-14128
city 	b32 		col  zp 2 274 1133 	 1 FT943-8478
city 	b32 		col  zp 2 274 1133 	 2 FT944-3280
city 	b32 		col  zp 2 274 1133 	 3 FT934-4790
city 	b32 		col  zp 2 274 1133 	 4 FT941-17640
city 	b33 		col  ok 3 292 1141 	 1 FT942-3981
city 	b33 		col  ok 3 292 1141 	 2 FT942-4141
city 	b33 		col  ok 3 292 1141 	 3 FT942-5651
city 	b33 		col  ok 3 292 1141 	 4 FT931-258
city 	b33 		col  ok 3 292 1141 	 5 FT922-1568
city 	b33 		col  ok 3 292 1141 	 6 FT942-13791
city 	b33 		col  ok 3 292 1141 	 7 FT922-9180
city 	b33 		col  ok 3 292 1141 	 8 FT933-1422
city 	b33 		col  ok 3 292 1141 	 9 FT924-11314
city 	b33 		col  ok 3 292 1141 	10 FT922-770
city 	b33 		col  ok 3 292 1141 	11 FT931-1043
city 	b33 		col  ok 3 292 1141 	12 FT922-7329
city 	b33 		col  ok 3 292 1141 	13 FT932-12315
nist 	ns2-ok-258 	ns2  ok 2 258 1200 	 1 FT932-7897
nist 	ns2-ok-258 	ns2  ok 2 258 1200 	 2 FT931-2806
nist 	ns2-ru-274 	ns2  ru 3 274 1290 	 1 FT931-12869
nist 	ns2-ru-274 	ns2  ru 3 274 1290 	 2 FT944-17513
nist 	ns2-ru-274 	ns2  ru 3 274 1290 	 3 FT944-3280
nist 	ns2-ru-274 	ns2  ru 3 274 1290 	 4 FT941-17640
nist 	ns2-ru-274 	ns2  ru 3 274 1290 	 5 FT941-16609
nist 	ns2-ru-274 	ns2  ru 3 274 1290 	 6 FT944-4751
nist 	ns2-zp-292 	ns2  zp 1 292 1200 	 1 FT934-10921
nist 	ns2-zp-292 	ns2  zp 1 292 1200 	 2 FT932-7223
nist 	ns2-zp-292 	ns2  zp 1 292 1200 	 3 FT932-4859
nist 	ns3-ok-274 	ns3  ok 3 274 1200 	 1 FT943-8478
nist 	ns3-ok-274 	ns3  ok 3 274 1200 	 2 FT944-3280
nist 	ns3-ok-274 	ns3  ok 3 274 1200 	 3 FT924-2212
nist 	ns3-ok-274 	ns3  ok 3 274 1200 	 4 FT944-18127
nist 	ns3-ok-274 	ns3  ok 3 274 1200 	 5 FT944-18230
nist 	ns3-ok-274 	ns3  ok 3 274 1200 	 6 FT921-15508
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	 1 FT943-5084
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	 2 FT932-4864
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	 3 FT944-9190
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	 4 FT922-2754
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	 5 FT924-14680
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	 6 FT942-5248
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	 7 FT924-11314
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	 8 FT934-8104
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	 9 FT944-8202
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	10 FT923-2405
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	11 FT924-10895
nist 	ns3-ru-292 	ns3  ru 1 292 1200 	12 FT944-6725
nist 	ns3-zp-258 	ns3  zp 2 258 660 	 1 FT934-15659
nist 	ns4-ok-274 	ns4  ok 3 274 1200 	 1 FT941-17640
nist 	ns4-ok-274 	ns4  ok 3 274 1200 	 2 FT931-12869
nist 	ns4-ok-274 	ns4  ok 3 274 1200 	 3 FT943-11934
nist 	ns4-ok-274 	ns4  ok 3 274 1200 	 4 FT942-9950
nist 	ns4-ru-258 	ns4  ru 2 258 1200 	 1 FT943-5307
nist 	ns4-zp-292 	ns4  zp 1 292 1200 	 2 FT943-8923
nist 	ns4-zp-292 	ns4  zp 1 292 1200 	 3 FT942-5248
nist 	ns4-zp-292 	ns4  zp 1 292 1200 	 4 FT922-1215
nist 	ns4-zp-292 	ns4  zp 1 292 1200 	 5 FT924-4469
nist 	ns5-ok-258 	ns5  ok 2 258 1200 	 1 FT931-3482
nist 	ns5-ok-258 	ns5  ok 2 258 1200 	 2 FT931-2005
nist 	ns5-ok-258 	ns5  ok 2 258 1200 	 3 FT944-7040
nist 	ns5-ru-292 	ns5  ru 1 292 1200 	 1 FT942-5904
nist 	ns5-ru-292 	ns5  ru 1 292 1200 	 2 FT933-15869
nist 	ns5-ru-292 	ns5  ru 1 292 1200 	 3 FT924-2474
nist 	ns5-zp-274 	ns5  zp 3 274 1200 	 1 FT943-8478
nist 	ns5-zp-274 	ns5  zp 3 274 1200 	 2 FT921-2636
nist 	ns6-ok-292 	ns6  ok 1 292 1200 	 1 0
nist 	ns6-ru-274 	ns6  ru 3 274 1265 	 1 FT943-8478
nist 	ns6-ru-274 	ns6  ru 3 274 1265 	 2 FT931-13738
nist 	ns6-zp-258 	ns6  zp 2 258 1200 	 1 FT944-15438
nist 	ns6-zp-258 	ns6  zp 2 258 1200 	 2 FT944-1726
nist 	ns6-zp-258 	ns6  zp 2 258 1200 	 3 FT923-2796
nist 	ns8-ok-292 	ns8  ok 1 292 1210 	 1 FT934-13781
nist 	ns8-ok-292 	ns8  ok 1 292 1210 	 2 FT924-11314
nist 	ns8-ok-292 	ns8  ok 1 292 1210 	 3 FT924-1255
nist 	ns8-ok-292 	ns8  ok 1 292 1210 	 4 FT941-11779
nist 	ns8-ok-292 	ns8  ok 1 292 1210 	 5 FT924-1207
nist 	ns8-ok-292 	ns8  ok 1 292 1210 	 6 FT941-12426
nist 	ns8-ru-258 	ns8  ru 2 258 1205 	 1 FT932-14128
nist 	ns8-ru-258 	ns8  ru 2 258 1205 	 2 FT944-7040
nist 	ns8-ru-258 	ns8  ru 2 258 1205 	 3 FT934-12464
nist 	ns8-ru-258 	ns8  ru 2 258 1205 	 4 FT931-2005
nist 	ns8-ru-258 	ns8  ru 2 258 1205 	 5 FT924-8561
nist 	ns8-ru-258 	ns8  ru 2 258 1205 	 6 FT944-7314
nist 	ns8-ru-258 	ns8  ru 2 258 1205 	 7 FT944-16521
nist 	ns8-ru-258 	ns8  ru 2 258 1205 	 8 FT931-2806
nist 	ns8-ru-258 	ns8  ru 2 258 1205 	 9 FT931-7090
nist 	ns8-zp-274 	ns8  zp 3 274 1187 	 1 FT941-17640
nist 	ns8-zp-274 	ns8  zp 3 274 1187 	 2 FT923-11292
nist 	ns8-zp-274 	ns8  zp 3 274 1187 	 3 FT944-18127
nist 	ns8-zp-274 	ns8  zp 3 274 1187 	 4 FT931-12869
nist 	ns8-zp-274 	ns8  zp 3 274 1187 	 5 FT944-18230
nist 	ns8-zp-274 	ns8  zp 3 274 1187 	 6 FT924-2358
nist 	ns8-zp-274 	ns8  zp 3 274 1187 	 7 FT921-15508
rutgers s001.t258 	s001 zp 3 258 1235 	 1 FT942-578
rutgers s001.t258 	s001 zp 3 258 1235 	 2 FT931-2005
rutgers s001.t258 	s001 zp 3 258 1235 	 3 FT944-1726
rutgers s001.t258 	s001 zp 3 258 1235 	 4 FT941-6108
rutgers s001.t258 	s001 zp 3 258 1235 	 5 FT911-3918
rutgers s001.t258 	s001 zp 3 258 1235 	 6 FT934-12464
rutgers s001.t258 	s001 zp 3 258 1235 	 7 FT944-7314
rutgers s001.t258 	s001 zp 3 258 1235 	 8 FT923-449
rutgers s001.t258 	s001 zp 3 258 1235 	 9 FT944-7040
rutgers s001.t258 	s001 zp 3 258 1235 	10 FT941-11158
rutgers s001.t274 	s001 ok 1 274 1237 	 1 FT943-11755
rutgers s001.t274 	s001 ok 1 274 1237 	 2 FT943-985
rutgers s001.t274 	s001 ok 1 274 1237 	 3 FT943-8478
rutgers s001.t274 	s001 ok 1 274 1237 	 4 FT943-3737
rutgers s001.t274 	s001 ok 1 274 1237 	 5 FT921-2636
rutgers s001.t274 	s001 ok 1 274 1237 	 6 FT933-11925
rutgers s001.t274 	s001 ok 1 274 1237 	 7 FT934-4995
rutgers s001.t274 	s001 ok 1 274 1237 	 8 FT934-2530
rutgers s001.t274 	s001 ok 1 274 1237 	 9 FT924-2358
rutgers s001.t274 	s001 ok 1 274 1237 	10 FT941-17640
rutgers s001.t292 	s001 ru 2 292 1262 	 1 FT943-1684
rutgers s001.t292 	s001 ru 2 292 1262 	 2 FT941-11779
rutgers s001.t292 	s001 ru 2 292 1262 	 3 FT942-10229
rutgers s001.t292 	s001 ru 2 292 1262 	 4 FT934-8104
rutgers s001.t292 	s001 ru 2 292 1262 	 5 FT934-6695
rutgers s001.t292 	s001 ru 2 292 1262 	 6 FT944-10159
rutgers s001.t292 	s001 ru 2 292 1262 	 7 FT931-9432
rutgers s001.t292 	s001 ru 2 292 1262 	 8 FT934-11906
rutgers s002.t258 	s002 ru 3 258 1290 	 1 FT923-8456
rutgers s002.t258 	s002 ru 3 258 1290 	 2 FT944-9459
rutgers s002.t258 	s002 ru 3 258 1290 	 3 FT942-7826
rutgers s002.t258 	s002 ru 3 258 1290 	 4 FT922-468
rutgers s002.t258 	s002 ru 3 258 1290 	 5 FT923-14087
rutgers s002.t258 	s002 ru 3 258 1290 	 6 FT911-3918
rutgers s002.t258 	s002 ru 3 258 1290 	 7 FT931-1338
rutgers s002.t274 	s002 zp 1 274 1337 	 1 FT921-15508
rutgers s002.t274 	s002 zp 1 274 1337 	 2 FT943-8478
rutgers s002.t274 	s002 zp 1 274 1337 	 3 FT931-14618
rutgers s002.t274 	s002 zp 1 274 1337 	 4 FT932-11975
rutgers s002.t274 	s002 zp 1 274 1337 	 5 FT941-16609
rutgers s002.t292 	s002 ok 2 292 1000 	 1 FT922-9330
rutgers s002.t292 	s002 ok 2 292 1000 	 2 FT934-8863
rutgers s002.t292 	s002 ok 2 292 1000 	 3 FT943-2556
rutgers s002.t292 	s002 ok 2 292 1000 	 4 FT921-10164
rutgers s002.t292 	s002 ok 2 292 1000 	 5 FT923-9849
rutgers s002.t292 	s002 ok 2 292 1000 	 6 FT922-12153
rutgers s002.t292 	s002 ok 2 292 1000 	 7 FT943-3011
rutgers s002.t292 	s002 ok 2 292 1000 	 8 FT932-5425
rutgers s002.t292 	s002 ok 2 292 1000 	 9 FT931-3537
rutgers s002.t292 	s002 ok 2 292 1000 	10 FT941-5683
rutgers s002.t292 	s002 ok 2 292 1000 	11 FT943-8923
rutgers s003.t258 	s003 ok 3 258 1077 	 1 FT931-2005
rutgers s003.t258 	s003 ok 3 258 1077 	 2 FT944-1726
rutgers s003.t258 	s003 ok 3 258 1077 	 3 FT934-12464
rutgers s003.t258 	s003 ok 3 258 1077 	 4 FT924-8561
rutgers s003.t258 	s003 ok 3 258 1077 	 5 FT911-3918
rutgers s003.t258 	s003 ok 3 258 1077 	 6 FT941-6108
rutgers s003.t258 	s003 ok 3 258 1077 	 7 FT924-13926
rutgers s003.t258 	s003 ok 3 258 1077 	 8 FT933-10240
rutgers s003.t258 	s003 ok 3 258 1077 	 9 FT934-5580
rutgers s003.t258 	s003 ok 3 258 1077 	10 FT944-9024
rutgers s003.t274 	s003 ru 1 274 1360 	 1 FT943-14882
rutgers s003.t274 	s003 ru 1 274 1360 	 2 FT921-5494
rutgers s003.t274 	s003 ru 1 274 1360 	 3 FT924-6107
rutgers s003.t274 	s003 ru 1 274 1360 	 4 FT921-2636
rutgers s003.t274 	s003 ru 1 274 1360 	 5 FT943-8478
rutgers s003.t274 	s003 ru 1 274 1360 	 6 FT943-16085
rutgers s003.t292 	s003 zp 2 292 1171 	 1 FT934-10921
rutgers s003.t292 	s003 zp 2 292 1171 	 2 FT934-4297
rutgers s003.t292 	s003 zp 2 292 1171 	 3 FT941-12323
rutgers s003.t292 	s003 zp 2 292 1171 	 4 FT934-10072
rutgers s003.t292 	s003 zp 2 292 1171 	 5 FT942-5248
rutgers s003.t292 	s003 zp 2 292 1171 	 6 FT943-3430
rutgers s003.t292 	s003 zp 2 292 1171 	 7 FT934-11014
rutgers s003.t292 	s003 zp 2 292 1171 	 8 FT934-7624
rutgers s003.t292 	s003 zp 2 292 1171 	 9 FT932-11049
rutgers s003.t292 	s003 zp 2 292 1171 	10 FT924-15607
rutgers s003.t292 	s003 zp 2 292 1171 	11 FT924-1107
rutgers s003.t292 	s003 zp 2 292 1171 	12 FT934-10231
rutgers s003.t292 	s003 zp 2 292 1171 	13 FT934-12800
rutgers s003.t292 	s003 zp 2 292 1171 	14 FT943-15578
rutgers s004.t258 	s004 zp 1 258 1212 	 1 FT944-15438
rutgers s004.t274 	s004 ru 2 274 1363 	 1 FT921-5494
rutgers s004.t292 	s004 ok 3 292 1904 	 1 FT941-12334
rutgers s004.t292 	s004 ok 3 292 1904 	 2 FT933-9986
rutgers s004.t292 	s004 ok 3 292 1904 	 3 FT934-10045
rutgers s004.t292 	s004 ok 3 292 1904 	 4 FT932-1015
rutgers s004.t292 	s004 ok 3 292 1904 	 5 FT931-15653
rutgers s004.t292 	s004 ok 3 292 1904 	 6 FT933-6509
rutgers s004.t292 	s004 ok 3 292 1904 	 7 FT942-5248
rutgers s004.t292 	s004 ok 3 292 1904 	 8 FT944-2202
rutgers s004.t292 	s004 ok 3 292 1904 	 9 FT942-15092
rutgers s004.t292 	s004 ok 3 292 1904 	10 FT931-10935
rutgers s004.t292 	s004 ok 3 292 1904 	11 FT934-13781
rutgers s004.t292 	s004 ok 3 292 1904 	12 FT933-10651
rutgers s004.t292 	s004 ok 3 292 1904 	13 FT932-12991
rutgers s005.t258 	s005 ru 1 258 1610 	 1 FT944-7314
rutgers s005.t258 	s005 ru 1 258 1610 	 2 FT931-7553
rutgers s005.t258 	s005 ru 1 258 1610 	 3 FT944-7649
rutgers s005.t258 	s005 ru 1 258 1610 	 4 FT922-13695
rutgers s005.t258 	s005 ru 1 258 1610 	 5 FT932-7897
rutgers s005.t258 	s005 ru 1 258 1610 	 6 FT932-7643
rutgers s005.t258 	s005 ru 1 258 1610 	 7 FT931-2806
rutgers s005.t258 	s005 ru 1 258 1610 	 8 FT923-2796
rutgers s005.t258 	s005 ru 1 258 1610 	 9 FT931-1831
rutgers s005.t258 	s005 ru 1 258 1610 	10 FT944-15438
rutgers s005.t258 	s005 ru 1 258 1610 	11 FT931-7297
rutgers s005.t258 	s005 ru 1 258 1610 	12 FT931-7090
rutgers s005.t258 	s005 ru 1 258 1610 	13 FT932-7637
rutgers s005.t274 	s005 ok 2 274 1320 	 1 FT944-18127
rutgers s005.t274 	s005 ok 2 274 1320 	 2 FT923-14293
rutgers s005.t274 	s005 ok 2 274 1320 	 3 FT943-11934
rutgers s005.t274 	s005 ok 2 274 1320 	 4 FT923-11292
rutgers s005.t274 	s005 ok 2 274 1320 	 5 FT924-7255
rutgers s005.t274 	s005 ok 2 274 1320 	 6 FT941-17640
rutgers s005.t274 	s005 ok 2 274 1320 	 7 FT944-18230
rutgers s005.t274 	s005 ok 2 274 1320 	 8 FT921-15508
rutgers s005.t274 	s005 ok 2 274 1320 	 9 FT944-3280
rutgers s005.t274 	s005 ok 2 274 1320 	10 FT941-9725
rutgers s005.t292 	s005 zp 3 292 1178 	 1 FT944-642
rutgers s005.t292 	s005 zp 3 292 1178 	 2 FT941-5213
rutgers s005.t292 	s005 zp 3 292 1178 	 3 FT934-12800
rutgers s005.t292 	s005 zp 3 292 1178 	 4 FT931-1043
rutgers s005.t292 	s005 zp 3 292 1178 	 5 FT941-8918
rutgers s005.t292 	s005 zp 3 292 1178 	 6 FT922-4425
rutgers s005.t292 	s005 zp 3 292 1178 	 7 FT932-12865
rutgers s006.t258 	s006 ok 1 258 1229 	 1 FT931-7553
rutgers s006.t258 	s006 ok 1 258 1229 	 2 FT931-11384
rutgers s006.t258 	s006 ok 1 258 1229 	 3 FT931-7090
rutgers s006.t258 	s006 ok 1 258 1229 	 4 FT942-578
rutgers s006.t258 	s006 ok 1 258 1229 	 5 FT944-7649
rutgers s006.t258 	s006 ok 1 258 1229 	 6 FT932-14128
rutgers s006.t258 	s006 ok 1 258 1229 	 7 FT931-2937
rutgers s006.t274 	s006 zp 2 274 1169 	 1 FT921-1556
rutgers s006.t274 	s006 zp 2 274 1169 	 2 FT944-3280
rutgers s006.t274 	s006 zp 2 274 1169 	 3 FT941-17640
rutgers s006.t274 	s006 zp 2 274 1169 	 4 FT931-12869
rutgers s006.t274 	s006 zp 2 274 1169 	 5 FT933-12570
rutgers s006.t274 	s006 zp 2 274 1169 	 6 FT921-15508
rutgers s006.t292 	s006 ru 3 292 700 	 1 FT942-5248
rutgers s006.t292 	s006 ru 3 292 700 	 2 FT932-1015
rutgers s006.t292 	s006 ru 3 292 700 	 3 FT924-12824
rutgers s007.t258 	s007 ru 2 258 1014	 1 FT944-7040
rutgers s007.t258 	s007 ru 2 258 1014	10 FT931-7553
rutgers s007.t258 	s007 ru 2 258 1014 	 2 FT944-15438
rutgers s007.t258 	s007 ru 2 258 1014 	 3 FT911-3918
rutgers s007.t258 	s007 ru 2 258 1014 	 4 FT943-1215
rutgers s007.t258 	s007 ru 2 258 1014 	 5 FT921-11682
rutgers s007.t258 	s007 ru 2 258 1014 	 6 FT923-2796
rutgers s007.t258 	s007 ru 2 258 1014 	 7 FT944-7314
rutgers s007.t258 	s007 ru 2 258 1014 	 8 FT931-2806
rutgers s007.t258 	s007 ru 2 258 1014 	 9 FT944-11064
rutgers s007.t258 	s007 ru 2 258 1014 	11 FT933-11258
rutgers s007.t258 	s007 ru 2 258 1014 	12 FT931-7090
rutgers s007.t258 	s007 ru 2 258 1014 	13 FT931-1831
rutgers s007.t258 	s007 ru 2 258 1014 	14 FT931-2937
rutgers s007.t258 	s007 ru 2 258 1014 	15 FT931-2938
rutgers s007.t258 	s007 ru 2 258 1014 	16 FT944-16521
rutgers s007.t258 	s007 ru 2 258 1014 	17 FT922-2486
rutgers s007.t258 	s007 ru 2 258 1014 	18 FT933-13759
rutgers s007.t258 	s007 ru 2 258 1014 	19 FT923-10088
rutgers s007.t258 	s007 ru 2 258 1014 	20 FT923-449
rutgers s007.t258 	s007 ru 2 258 1014 	21 FT943-9656
rutgers s007.t258 	s007 ru 2 258 1014 	22 FT934-6622
rutgers s007.t258 	s007 ru 2 258 1014 	23 FT933-9071
rutgers s007.t258 	s007 ru 2 258 1014 	24 FT944-15440
rutgers s007.t258 	s007 ru 2 258 1014 	25 FT932-7637
rutgers s007.t258 	s007 ru 2 258 1014 	26 FT941-4379
rutgers s007.t258 	s007 ru 2 258 1014 	27 FT921-7240
rutgers s007.t258 	s007 ru 2 258 1014 	28 FT934-13338
rutgers s007.t258 	s007 ru 2 258 1014 	29 FT923-11228
rutgers s007.t258 	s007 ru 2 258 1014 	30 FT932-9585
rutgers s007.t274 	s007 zp 3 274 1094 	 1 FT944-3280
rutgers s007.t274 	s007 zp 3 274 1094 	 2 FT943-8478
rutgers s007.t274 	s007 zp 3 274 1094 	 3 FT944-3280
rutgers s007.t274 	s007 zp 3 274 1094 	 4 FT943-8478
rutgers s007.t274 	s007 zp 3 274 1094 	 5 FT943-3737
rutgers s007.t274 	s007 zp 3 274 1094 	 6 FT921-2636
rutgers s007.t274 	s007 zp 3 274 1094 	 7 FT942-15408
rutgers s007.t274 	s007 zp 3 274 1094 	 8 FT923-13374
rutgers s007.t274 	s007 zp 3 274 1094 	 9 FT943-11205
rutgers s007.t274 	s007 zp 3 274 1094 	10 FT921-15760
rutgers s007.t292 	s007 ok 1 292 1284 	 1 FT942-5248
rutgers s007.t292 	s007 ok 1 292 1284 	 2 FT934-2265
rutgers s007.t292 	s007 ok 1 292 1284 	 3 FT941-11779
rutgers s007.t292 	s007 ok 1 292 1284 	 4 FT944-15084
rutgers s007.t292 	s007 ok 1 292 1284 	 5 FT933-249
rutgers s007.t292 	s007 ok 1 292 1284 	 6 FT932-7223
rutgers s007.t292 	s007 ok 1 292 1284 	 7 FT934-2514
rutgers s007.t292 	s007 ok 1 292 1284 	 8 FT934-1035
rutgers s007.t292 	s007 ok 1 292 1284 	 9 FT944-9133
rutgers s007.t292 	s007 ok 1 292 1284 	10 FT934-12730
rutgers s007.t292 	s007 ok 1 292 1284 	11 FT932-6510
rutgers s007.t292 	s007 ok 1 292 1284 	12 FT942-10229
rutgers s008.t258 	s008 ok 2 258 1418 	 1 FT942-2223
rutgers s008.t258 	s008 ok 2 258 1418 	 2 FT932-3590
rutgers s008.t258 	s008 ok 2 258 1418 	 3 FT921-10278
rutgers s008.t258 	s008 ok 2 258 1418 	 4 FT942-578
rutgers s008.t258 	s008 ok 2 258 1418 	 5 FT931-7090
rutgers s008.t258 	s008 ok 2 258 1418 	 6 FT942-7890
rutgers s008.t258 	s008 ok 2 258 1418 	 7 FT922-2528
rutgers s008.t258 	s008 ok 2 258 1418 	 8 FT943-12981
rutgers s008.t258 	s008 ok 2 258 1418 	 9 FT934-12464
rutgers s008.t258 	s008 ok 2 258 1418 	10 FT922-5072
rutgers s008.t258 	s008 ok 2 258 1418 	11 FT944-7314
rutgers s008.t258 	s008 ok 2 258 1418 	12 FT933-10240
rutgers s008.t258 	s008 ok 2 258 1418 	13 FT931-2005
rutgers s008.t258 	s008 ok 2 258 1418 	14 FT932-7897
rutgers s008.t274 	s008 ru 3 274 1402 	 1 FT944-18230
rutgers s008.t274 	s008 ru 3 274 1402 	 2 FT941-16609
rutgers s008.t274 	s008 ru 3 274 1402 	 3 FT944-3280
rutgers s008.t274 	s008 ru 3 274 1402 	 4 FT941-17640
rutgers s008.t274 	s008 ru 3 274 1402 	 5 FT924-2358
rutgers s008.t274 	s008 ru 3 274 1402 	 6 FT923-14293
rutgers s008.t274 	s008 ru 3 274 1402 	 7 FT923-10730
rutgers s008.t274 	s008 ru 3 274 1402 	 8 FT944-18127
rutgers s008.t274 	s008 ru 3 274 1402 	 9 FT921-1556
rutgers s008.t274 	s008 ru 3 274 1402 	10 FT942-15407
rutgers s008.t274 	s008 ru 3 274 1402 	11 FT923-11292
rutgers s008.t274 	s008 ru 3 274 1402 	12 FT931-12869
rutgers s008.t274 	s008 ru 3 274 1402 	13 FT944-4751
rutgers s008.t274 	s008 ru 3 274 1402 	14 FT921-15508
rutgers s008.t274 	s008 ru 3 274 1402 	15 FT943-11934
rutgers s008.t292 	s008 zp 1 292 1296 	 1 FT944-6725
rutgers s008.t292 	s008 zp 1 292 1296 	 2 FT932-5668
rutgers s008.t292 	s008 zp 1 292 1296 	 3 FT934-10231
rutgers s008.t292 	s008 zp 1 292 1296 	 4 FT932-11049
rutgers s008.t292 	s008 zp 1 292 1296 	 5 FT934-10921
rutgers s008.t292 	s008 zp 1 292 1296 	 6 FT922-4425
rutgers s009.t258 	s009 zp 2 258 992 	 1 FT931-2005
rutgers s009.t258 	s009 zp 2 258 992 	 2 FT931-7553
rutgers s009.t258 	s009 zp 2 258 992 	 3 FT924-8561
rutgers s009.t274 	s009 ok 3 274 1263 	 1 FT931-12869
rutgers s009.t274 	s009 ok 3 274 1263 	 2 FT944-3280
rutgers s009.t274 	s009 ok 3 274 1263 	 3 FT923-11292
rutgers s009.t274 	s009 ok 3 274 1263 	 4 FT943-11934
rutgers s009.t274 	s009 ok 3 274 1263 	 5 FT942-15408
rutgers s009.t274 	s009 ok 3 274 1263 	 6 FT944-16970
rutgers s009.t274 	s009 ok 3 274 1263 	 7 FT941-13218
rutgers s009.t274 	s009 ok 3 274 1263 	 8 FT943-11627
rutgers s009.t274 	s009 ok 3 274 1263 	 9 FT942-12506
rutgers s009.t274 	s009 ok 3 274 1263 	10 FT944-18127
rutgers s009.t274 	s009 ok 3 274 1263 	11 FT941-17640
rutgers s009.t292 	s009 ru 1 292 1398 	 1 FT931-1043
rutgers s009.t292 	s009 ru 1 292 1398 	 2 FT941-12426
rutgers s009.t292 	s009 ru 1 292 1398 	 3 FT942-5651
rutgers s009.t292 	s009 ru 1 292 1398 	 4 FT932-3822
rutgers s009.t292 	s009 ru 1 292 1398 	 5 FT943-274

-----------------------------------------------
8. Aspect Analysis by the NIST Assessors

- - - - - - - - - - - - - - - - - - 
Topics as seen by the NIST assessors:

============
Number: 258i 
Topic: Computer Security

Description: 
Identify instances of illegal entry into sensitive computer 
networks by non-authorized personnel.
  
Narrative:  
Illegal entry into sensitive computer networks is a serious 
and potentially menacing problem.  Both 'hackers' and 
foreign agents have been known to acquire unauthorized entry 
into various networks. Items relevant to this subject would 
include but not be limited to instances of illegally 
entering networks containing information of a sensitive 
nature to specific countries, such as defense or technology 
information, international banking, etc. Items of a personal 
nature (e.g. credit card fraud, changing of college test 
scores) should not be considered relevant. 

Aspects: each different case of illegal entry into a computer network.

============
Number: 274i 
Topic: Electric Automobiles 

Description:  
What are the latest developments in the production of electric 
automobiles? 

Narrative: 
The economic feasibility of electric automobiles appears to 
be limited by a number of factors, including the limited range of 
operation between recharges of batteries.  What progress has been made 
in addressing these factors?

Answers: each different recent development in the field of electric automobiles

============
Number: 292i 
Topic: Worldwide Welfare

Description: 
Identify social programs for poor people in countries other 
than the U.S. 

Narrative: 
To be relevant a document would identify a welfare program 
in a foreign country and explain how it works to aid 
citizens who have little or no income.  It would include 
those who can't work because of a disability and people who 
have the extra burden of small children.  The document 
should indicate how these people are supported or not 
supported.  A relevant document should identify the source 
of the monies used to support such welfare programs. 
</top>

Answers: each different social program in each different country.

- - - - - - - - - - - - - - - 
Instructions to the assessors:

You will be presented with a topic, which is the description of an 
information need. Stated or implied in the topic is a question with 
multiple aspects or answers. Here is an example topic:

	<title> Common Treatments for High Blood Pressure in the U.S. 
	        and Western Europe

	<desc> Description: 
		Identify the various common alternatives available for the 
		treatment of hypertension in the U.S. and western Europe.
 
	<narr> Narrative: 
		A relevant document will discuss one or more treatments 
		commonly used for the treatment of high blood pressure in 
		the U.S. or western Europe. Treatments might include drug 
		therapy, life-style, etc. One or more specific treatments 
		must be identified for a document to be relevant. Documents 
		containing only generalities are not relevant. Neither are 
		documents about treatments no longer in common use in the 
		U.S. or western Europe.

	<aspects> Aspects:
		EACH DIFFERENT treatment of the sort described above 

You will also be given a set of documents - articles from the Financial
Times of London 1991-1994. 

Your task will be to read through the documents looking for relevant ones 
which contain one or more answers to the question posed by the topic. You 
are to gradually compile a list of all the different answers contained 
in relevant documents and identify which answer appears in which document.

On the following pages are materials for an example you should work through 
for practice. The example includes:

	- instructions

	- an example topic

	- 6 documents (to save time these are summarized in a sentence or 2)

	- a form on which to record the aspects

	- a form on which to record which aspects occur in which documents

You can ask questions at any time.

Instructions:  

	Your task is to make a list of all the different aspects of the 
	topic - answers to the question posed by the topic - which are 
	contained in relevant documents. You should do the following:

	- read the topic carefully noting what sort of aspects you
	  are to identify and what documents are considered relevant

        - then read each of the documents from the pool for the topic and 
	  gradually:

	   - create a list of the aspects found somewhere in the documents

           - record a word or short phrase, describing each aspect found,
	     on the "Aspects of Topic" form. 

           - determine which documents contain which aspects 

           - bracket each aspect in the text of the document in which it 
             was found

	   - on the "Document-Aspect-Matrix" form, for each document, circle 
	     the number of each aspect/answer the document covers/contains

	  Feel free to read the documents as many times as needed and to 
	  revise your list of answers as well as the list of which answers 
	  are provided by which documents.

Topic: Sample-Topic

	<title> Common Treatments for High Blood Pressure in the U.S. 
	        and Western Europe

	<desc> Description: 
		Identify the various common alternatives available for the 
		treatment of hypertension in the U.S. and western Europe.
 
	<narr> Narrative: 
		A relevant document will discuss one or more treatments 
		commonly used for the treatment of high blood pressure in 
		the U.S. or western Europe. Treatments might include drug 
		therapy, life-style, etc. One or more specific treatments 
		must be identified for a document to be relevant. Documents 
		containing only generalities are not relevant. Neither are 
		documents about treatments no longer in common use in the 
		U.S. or western Europe.

	<aspects> Aspects:
		EACH DIFFERENT treatment of the sort described above 

Sample pseudo-document (summaries):

Doc-1:	Talks about the increasing popularity of the 
	mediterranean diet in the U.S., its recommendation 
	by some physicians for the treatment of mild 
	hypertension. It mentions the common use in Spain of
	a tea made from the snakeroot plant to lower blood 
	pressure.

Doc-2:  Talks about how the ancient Roman physicians 
	used leeches to lower blood pressure. 

Doc-3:  Talks about how high blood pressure can lead 
	to health problems and about how more people 
	should get their pressure checked.

Doc-4:  Talks about use of diuretics and beta-blockers in Japan
	as cheap treatments for elevated blood pressure levels. 
	It states biofeedback techniques have been found to 
	be useful in managing hypertension in studies
	in California and is being used by physicians
	with increasing numbers of patients.

Doc-5:  Talks about the high stress levels among workers 
	at blood collection agencies where there is increasing 
	pressure on workers to do more on smaller budgets.

Doc-6:  Talks about Merck's sales of a drug Vasotec in the U.S.,
	used to lower blood pressure. It mentions biofeedback as 
	an available and popular treatment for high blood 
	pressure in the U.K's government hospitals.