From:	SMTP%"SERBAN@D0BNL2"  6-MAR-1998 10:01:53.68
To:	DIEHL
CC:	
Subj:	DAM review

Date: Fri, 06 Mar 1998 10:01:49 -0600
From: SERBAN@D0BNL2
Subject: DAM review
To: diehl@dafm03.fnal.gov
Message-id: <01IUCBQ9O636000093@FNAL.FNAL.GOV>
Organization: Fermi National Accelerator Laboratory
X-VMS-To: DIEHL
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

                                                           FNALD0,  6-MAR-1998

    In the next mail message I will be sending you the ps file with my
transparencies. Below is a written set of comments I sent to Hobbs:

    I want  to make  some  general comments  on DAM  that I  could not make
during the meeting as it was not my mandate. First let me say that what Lee
presented has the ingredients we need and that SAM as means of handling the
raw data for the reconstruction farm is probably the best we can do without
inventing an enormously complex  system. My main bone of contention is with
Heidi & Co.  proposal of  exclusive physical  streams. We  need to identify
very clearly what and who benefits from such proposal.

    First of all there are offline a number of critical phases for which
we must find optimal solutions:

Phase 1: Aligning and calibrating  the detector. Calibration means not just
correcting gains but  also determining sampling  fractions for calorimeter,
overall energy  scale and mmomentum  measurements in  the magenetive field.
During this phase a parallel activity is to debug and improve algorithms in
the reconstruction program.

Phase 2: Steady state processing of  data. This is not just processing data
through  the  reconstruction  farm. It  may require  some  preprocessing to
calculate time dependent quantities needed by reconstruction. An example of
such activity in run I was a special stream to the Express line to find hot
cells.   Although  phase 2  may  start  before  phase I is   satisfactorily
completed it  cannot start seriously  before much  effort has been spent on
phase I.  Thus it is  critical if we  want to be  competitive  that phase I
proceed as rapidly as possible.

Phase 3: Analysis. This can and  should begin while phase 2 is underway but
will not reach maturity until a significant amount of data has gone through
phase 2.

    Phase 1 definitely needs special  streams which will need to be defined
before we start running and  splitting should take place immediately, ie at
the data  logger stage.  We need to  make the  best data for  various tasks
available in the most convenient  form. This data will be accessed over and
over again until people are satisfied that they understand the detector and
can  provide  reliable  information  for  phase 2 to  begin.  These special
calibration & alignment  streams have nothing in  common with Heidi & Co.'s
proposal. We should try  to keep enough of that  data on disk as long as it
is needed.

    The  proposed  physical  streams are  trying to  address  only phase 3.
Splitting   streams  early at  the raw  data  stage  inclusively  as Marcel
proposed makes phase 2 more  complicated, the farm should be processing all
streams as  they come and pay  no attention  to the split.  The split is an
annoyance but not necessarily a big headache. It does however make an event
data base much  more complicated  because with each  event we must now keep
track of all the streams it is in.  And this goes all the way down the line
to STA's (which  may or may not  exist) , EDU50's and  EDU5's. So splitting
does nothing good for phase 2.

    So the  main  issue is  how to  handle  phase  3.  Splitting  into into
inclusive streams does alleviate the problem of having to cycle through all
the data  if you  want to only  do  something with a  portion  of the data.
Inevitably,   though, this  can only  work if  somebody  is  willing to set
priorities and  makes a decision of  who should go  ahead first. The lowest
guy on the  totem pole ends  waiting  probably longer than  if one where to
cycle through all the data willy  nilly. An example of how everybody may in
fact lose is the processing of  D0FIX, run Ic regular data and run Ic W-jet
trigger data.  When D0FIX was  proposed it  was though to  be a 3-month job
that could be run  using all farm  resources while  there was a lull in the
running. D0FIX was not ready in time to take advantage of the lull and then
it was decided to split the  resources with run Ic data processing. But run
Ic regular data was  given higher priority so  D0FIX was delayed some more.
Then it was realized that processing  run Ic W-jet data was going to take a
hell of a  lot more  resources  than  originally  anticipated  so more farm
resources were given to  that task. The net  result is that 3 months turned
into a year. So permit me to raise my eyebrows when somebody claims that by
splitting  data  into  streams his  job may  be speeded  up  significantly.
Somebody's job  may but it  will not  necessarily be his.  We just turned a
physical problem into a political problem.
    
    Splitting into  exclusive streams a la Heidi  in my view is even worse,
at least from the point of view of  top and new phenomena. The top analysis
used: electrons, muons and all jets  trigger. If you look at the overlap of
the top triggers you  will find they overlap with  almost all the available
triggers. Thus  the top  analysers will have  to cycle  through all streams
excepting low  pT B's  (StreamB) and low  multiplicity jets  (StreamJ). The
last two  streams may be the  bulk of the  data. So if we  are are going to
split physics  data at all it should  be into 3  streams: Stream B, StreamJ
and  StreamE  (everything  else).  This split  may make  some  sense if the
subequent treatment of the data is  very different for each stream. You can
think of  each  stream as  being a  separate   experiment.  Splitting those
streams into finer  entities only accomplishes  chaos. You will always have
special   streams  for  special  tasks  but  those  are not  what  is under
discussion. We cannot treat the physics data as a set of special tasks.

    What we really want and cannot  have at analysis stage is random access
to all  events and all  event  information. So  what we need  to devise for
phase 3 is a scheme that best  approximates the ideal situation. What we do
in analysis is use EDU5  to decide what are the  interesting events, how to
look for  interesting events  and what  subset needs  reprocessing. For the
interesting  events (assuming they  are fairly small in  number) we want to
keep on disk all  the available  information and we  would be able to acces
each event  randomly. The  next question how  to get in  that form when the
data resides in  sequential form on another  media. The pick event approach
is all right  as long as the  sample is  small and getting  doesn't require
mounting all  tapes. The  difficult problem  is what to do  when we want to
reprocess a  significant amount of  data (but still  small compared to full
data  sample).  A  solution that  may  work and  may  reduce the  amount of
politics in  setting priorities is  the freight train  approach as long the
freight train consists in making  available on disk in random access form a
substantial  portion of the  data for some  well defined  time period. Then
selections of the  pick event  variety that would  involve staging too many
tapes (but the number of events still small enough for permanent storage on
disk) can run on the freight train.  Reprocessing jobs will also run on the
freight train. The reprocessing job  will have lists of events to reprocess
and pick them  up as they  become available.  The EDU5's  are used to setup
those lists. Running  sequentially through a  particular physical stream to
accomplish this job may seem more  efficient to you but only if you are the
lucky stiff that will not let others get their analysis done.

                                   Serban