MDC3 Summary

Thomas Ullrich for the Physics Working Groups

April 28, 2000

MDC3 was planned as a STAR-only exercise directed at physics analysis,
exercising physics analysis software, analysis infrastructure and CAS
tools and facilities. While the preparation for the effort started as
early as February, the actual MDC3 itself was limited to two weeks, Mar
26 - April 9, 2000. In the following we summarize briefly the results,
experiences, and lessons learned.

# Preparations

Other than the previous MDCs that focused primarily on data
reconstruction, MDC3 emphasized data mining and analysis of already
processed data. This required a massive effort beforehand in order to
produce the required DST datasets.

All participating Physics Working Groups (PWG) agreed to concentrate on
the study of simulated data that corresponds to year 1 settings, i.e.
central to semi-central Au+Au collisions at s = 200 GeV. Only detectors
available in the first year of data taking were included in the
simulation. After thorough discussion between the various groups the
following datasets were requested for production:

+------------+------------+--------------------------------+----------+
| **PWG**    | **G        | **Settings**                   | **\# of  |
|            | enerator** |                                | events** |
+------------+------------+--------------------------------+----------+
| Spectra    | HIJING     | Et trigger \> 5 GeV,           | 20,000   |
|            |            | Jet-quenching on/off           |          |
+------------+------------+--------------------------------+----------+
| HBT        | MEVSIM     | Various multiplicities,        | 50,000   |
|            |            | p-spectra and HBT parameters   |          |
+------------+------------+--------------------------------+----------+
| EbyE,      | MEVSIM     | Various settings (flow,        | 50,000   |
| Spectra,   |            | fluctuations, resonances). 50  |          |
| S          |            | cascades added per event.      |          |
| trangeness |            |                                |          |
+------------+------------+--------------------------------+----------+
| pColl      | STARlight, | Two gamma events, cosmics,     | 780,000  |
|            |            | beam-gas, inclusive g+A        |          |
|            | dtunuc,    |                                |          |
|            |            |                                |          |
|            | HemiCosm,  |                                |          |
|            |            |                                |          |
|            | FRITIOF,   |                                |          |
|            |            |                                |          |
|            | VENUS      |                                |          |
+------------+------------+--------------------------------+----------+

The fast TPC simulator (tfs) was used for all datasets. Note that the
events requested by the Peripheral Collisions Group (pColl) are
considerably smaller than the rest. They contain a few tracks per event
only. For a more detailed list see
http://www.star.bnl.gov/STAR/html/all_l/html/mdc3.html.

The production consisted of three steps: Event generation, GSTAR
simulations, and reconstruction. The event generation was partially
delayed because some modification of MEVSIM was necessary. All datasets,
with the exception of the HIJING events, were produced by the referring
working groups. Although GSTAR simulations are very CPU intensive, which
was especially true for the MEVSIM datasets due to the flat rapidity
distributions[^1], all requested datasets were run through GSTAR by the
middle of March.

The reconstruction of the simulated data was severely delayed because of
the lack of a stable reconstruction chain. Crashes in various packages,
the switch of compilers on SUN platforms, and a consistently poor
stability of the RCF reconstruction farm (CRS) inhibited the release of
a production version in time. This was finally achieved two weeks before
the startup of MDC3.

Once the production chain was in place STAR was able to produce between
5k-10k events per day. This rate, however, was not sustainable because
of various problems at RCF and here especially with HPSS. As a
consequence MDC3 began with roughly 60% of all requested DSTs available.
As it turned out later a (small) fraction of these events was not usable
due to file corruption. Besides this the following shortcomings were
detected in the course of MDC3:

-   The additional 50 /event requested by the Strangeness group were not
    mixed into the MEVSIM events. This was supposed to be done at the
    production stage. These events were therefore re-produced during
    MDC3 with highest priority.

-   The MEVSIM "flow" events contained no flow due to a problem in the
    generator (MEVSIM). This was fixed and the production was repeated
    during MDC3.

The main issue, however, was the lack of stability at RCF. The following
listing is an extract from a presentation by Yuri Fisyak given at the
RCF quarterly meeting shortly after MDC3.

-   During production February-March we lost two whole days each two
    weeks.

-   Multiple interruptions of connections and network glitches during a
    day.

-   Multiple problems with STAR disk arrays.

-   HPSS:

```{=html}
<!-- -->
```
-   A few tapes have been destroyed (only two files were lost)

-   Failure of one of the tape drives blocks all STAR tape drives and
    thus stops production. The same is true if the tape drive only fails
    to read a file.

```{=html}
<!-- -->
```
-   Real production demands 24h 7days a week support from RCF which is
    not available.

-   RCF is understaffed, major areas (Linux farm, disk arrays, HPSS) are
    controlled by only one person.

-   Many problems caused by network -- Gigabit link and firewalls.

# Attendance during MDC3 

An estimated number of \~50 STAR collaborators attended MDC3. Not all of
them were present at all times because of parallel meetings (NA49
collaboration meeting, first week) and the LBL DOE-review (second week).
Roughly 30-35 people attended the joint PWG meetings each Tuesday and
Thursday.

# BNL Infrastructure

During MDC3 the center-of-gravity of activities moved from building 510
& 118 to building 902. The meeting rooms (902 & 118) turned out to be
sufficiently large and no lack of desks or terminals was reported.

# Experiences with RCF

## Stability of Operations 

The poor stability of RCF during MDC3 preparations extended into MDC3
itself. This was especially true for the first week where the effective
availability of the Central Analysis Server (CAS) was very low (\< 30%).
The situation improved somewhat during the second week but stayed below
our expectations. In the following we give a brief overview of the
problems and interruptions during MDC3:

  --------- -------------------------------------------------------------
  Sun 26:   All STAR files on AFS were accidentally deleted on Friday
            while moving the afs server behind the firewall. All files
            were recovered from backup but the CAS nodes couldn't access
            them (cache corrupted). Many nodes had to be rebooted to
            clear the problem.

  Mon 27:   Network went down starting at 4:30pm. The failure was caused
            by RCF personal testing two network switches. Continuing
            problems with AFS. LSF server was moved for unknown reasons.

  Tue 28:   Crash of STAR's MDS server in the morning caused by disk
            problems. The server was rebooted and HPSS restarted.
            Continuing network problems.

  Wed 29:   First production runs since Friday started. First day with
            almost stable conditions despite small network interruptions.

  Thu 30:   NFS server (rnfs04) went down at 7:15am. Problems with MTI
            drives. Back at 9:50am. Production stopped because no NFS
            service available. No access to HPSS all day. RCF installs
            patch on NFS server. No access to home directories from noon
            to 1pm. Continuous network interruptions all evening and
            night.

  Fri 31:   Network and AFS went down at noon. RAID array used for AFS
            storage is experiencing hardware problems. RCF announces that
            RAID box controller is broken and that an expert from the
            vendor is on his way.

  Sat 1:    Down all day.

  Sun 2:    Partially recovered, many files still missing. Several CAS
            nodes without AFS. Lost day.

  Mon 3:    Back in operation. RCF announced that they are going to
            replace the racks that hold the STAR AFS disks (1h
            interruption).

  Tue 4:    Stable conditions.

  Wed 5:    Bad network with lots of interruptions all evening and night.

  Thu 6:    Batch queuing system (LSF) breaks. Some CAS nodes are not
            responding.

  Fri 7:    Last official MDC3 working day. STAR's CAS usage/load very
            low.

  Mon 10:   One day after end of MDC3: RCF announces that all planned
            network changes are completed.
  --------- -------------------------------------------------------------

During all of MDC3 RCF was unable to provide stable conditions for more
than one day. The unusual high rate of hardware failures is hard to
understand when compared to similar computing facilities at other
laboratories. If this situation persists it imposes a severe problem for
STAR as it inhibits data mining and analysis tasks to be performed on
large amounts of events.

## LSF 

For the first time STAR was extensively using the LSF batch system at
CAS which is installed on all 27 STAR-CAS nodes. In order to get users
up to speed a tutorial on LSF was given at the MDC3 startup meeting.
Because many users already had some experiences using LSF at PDSF/LBL
the "how-to" turned out to be of no concern.

With increasing load on the CAS farm it became clear that the current
configuration of LSF is not sufficient for our needs and the setup
requires some improvements. During the course of MDC3 we already started
to change the setup of LSF and added (*i*) a short queue for jobs
limited to 30 min CPU time and (*ii*) disabled logins on 10 nodes in
order to preserve them exclusively for LSF batch jobs. The main
complaints concerning LSF were:

-   LSF doesn't recognize nodes that are not fully operational. As a
    result all submitted jobs are preferentially sent to those nodes
    (since they show no load) where they either disappear or hang. This
    problem usually persists until the nodes get rebooted.

-   Batch jobs not preferentially submitted to batch only nodes.
    Interactive and non-interactive nodes have same priority.

-   Queuing policy: large amount of short jobs can completely block all
    jobs sent to the long queue.

Users report that all LSF trouble tickets sent to RCF were resolved
within time.

## CPU and Disk Resources

Data analysis was more harmed by CAS/RCF stability problems than by the
lack of processors. The current available disk space was adequate for
MDC3 purposes.

## File Corruption

In the course of MDC3 several unreadable files were discovered. Further
investigation revealed that this is not related to STAR I/O software but
to actual corrupted files on disk. The cause of this problem is (still)
unknown. Defective cables, faulty disk controllers, NFS problems, or OS
kernel bugs are possible candidates. Although the overall number of
corrupted files was small, the occurrence of those files hindered data
mining analysis to a large degree.

# Physics Analysis

Various Physics Working Groups documented their specific goals and plans
beforehand. These documents can be accessed from the MDC home page at:
http://www.star.bnl.gov/STAR/html/all_l/html/mdc3-main.

One of the set goals common to almost all of the Physics Working Groups
was to produce efficiency corrected spectra which involves the use of
rather new embedding and hit-hit and track-track association tools. This
was, at least in part, achieved during MDC3 and studies and testing will
clearly continue. Another more technical goal was the first-time use of
the Grand Challenge apparatus to retrieve and scan events based on
queries of disk resident tags. Although first tests run by a small group
of people (guinea pigs) looked promising, STAR-wide usage didn't take
place.

In the following we only give a brief overview of the activities,
accomplishments[,]{.underline} and findings of the participating PWGs.
More detailed information can be found on the WWW pages of the
individual groups (see http://www.star.bnl.gov/STAR/html/subs.html).

## Event-by-Event

-   Flow:

```{=html}
<!-- -->
```
-   1100 events processed (all available flow events)

-   event­plane resolution is \~50% (for 2% flow in)

-   q­variable analysis confirms 2% flow

-   v1 *-*distribution indicates finite *-*resolution

-   v1 amplitude indicates 1% directed flow

-   v2 amplitude indicates 1% elliptical flow

-   discrepancy with 2% input amplitudes is resolved

-   v1, v2 p-distributions are uniform as expected

```{=html}
<!-- -->
```
-   Global Variables

```{=html}
<!-- -->
```
-   1500 events processed

-   several scatter plots for total momentum and multiplicity generated
    for vanilla events

-   general trends are consistent with MEVSIM input and expectations for
    uncorrelated events

```{=html}
<!-- -->
```
-   Two- point analysis

```{=html}
<!-- -->
```
-   25, 000 events were processed, of which \~ 13,000 succeeded

-   m m ratio plots were generated for four pair types and three event
    types

-   vanilla events gave flat ratio distributions with split-track ridge
    on main diagonals for like- signed pairs

-   dynamical fluctuation events gave expected large-scale correlation
    structure, *except,* +/- pairs showed *no* correlation - traced to
    algorithm choice in MEVSIM - good blind test!

```{=html}
<!-- -->
```
-   Process times

```{=html}
<!-- -->
```
-   flow - 7 sec/ event - 2 x 2 CPU hours for 1k events

-   two- point - 25 sec/event - 6 CPU hours for 1k events

```{=html}
<!-- -->
```
-   Corrupted Data

```{=html}
<!-- -->
```
-   corrupted files caused job crashes

-   strategy adopted to batch process in block units - 1 or ten files at
    a time

-   batch output concatenated with *hadd* root utility.

-   up to 50% event loss in first pass through data

```{=html}
<!-- -->
```
-   *primtrk* problems: ^2^-cut not useful, refit has developed
    difficulties recently

-   Multiple operations directly from DST imply significant overhead --
    community *primtrk* miniDST will be very useful

-   Many LSF protocol issues to be resolved

-   Logfile volumes are excessive - problem of data compression - what
    info is significant in log files - precedent of autoQA parsing

-   Need to rerun salvageable failed jobs in batch, avoid dead nodes -
    protocol issues

## HBT

MDC3 was a successful exercise from the standpoint of the HBT PWG. Solid
progress was made and it was demonstrated that the HBT group is ready to
start generating physics results when real data arrives. For the first
time, events with multiple pair-wise correlations (quantum statistical,
Coulomb, and strong final state) were generated, run through the full
simulation/reconstruction chain, analyzed with the code intended for
'real' data, and corrected (e.g. for splitting and Coulomb effects).
Successful reconstruction of the HBT input parameters (e.g. radius) is a
nontrivial milestone.

-   As all other PWGs, for most of the first week HBT was stymied by the
    dismal performance of the RCF.

-   Production and analysis of HBT-DST's played a key role in MDC3.

-   The DSTs worked smoothly helping to insulate from some instabilities
    in STAR software.

-   DSTs can now be used as input to the standalone (no-root) StHbt code
    speeding up analyses which are not looking at pions.

-   Track splitting was a major issue for users looking at
    identical-particle interferometry. This was more or less expected,
    but MDC3 mapped out the magnitude and systematics of the problem,
    and gave the opportunity to develop and test tools to handle it.

-   Track splitting totally dominated the identical pion, kaon, and
    proton correlations. The splitting was shown to be independent of
    event multiplicity, and worse at high p.

-   Related to all this is the work on handling the splitting via the
    topology maps. This method worked extremely well.

-   The Coulomb correction tools work well.

-   It was shown that a sound HBT result can be obtained after 5 min of
    real data taking.

-   3-pion analysis was able to see the residual signal from the
    2-particle correlations in the "data". Lots of efforts on reducing
    CPU requirements for three- AND two-particle analyses. These
    developments should take 3-particle correlation functions in STAR
    from the realm of the ludicrous to the feasible. This work has
    already led to a 4-fold performance boost in the 2-particle
    analysis.

-   The "resonance subgroup", focussing on 's, 's, and K\*'s, have
    demonstrated the ability to reconstruct these resonances prior to
    MDC3, using Hijing events. For reasons still under very active
    investigation, something changed during MDC3, and signals that used
    to be very strong, are now barely visible. It does not help matters
    that the simulations were NOT run according to agreed-upon
    standards, wherein GSTAR would decay (and keep the parentage of)
    these resonances. So, this remains under investigation.

-   Although back-of-the-envelope estimates indicated that
    event-by-event pion correlation functions *might* be feasible, the
    E-by-E HBT analysis of MDC3 finally put this to the test. This
    analysis proved quite feasible, if the multiplicities and radii
    assumed are close to data. The eventwise extracted radius tracked
    well with the 'input' radius, with a dispersion of \~2 fm. After
    corrections for splitting and Coulomb effects, the eventwise
    correlations themselves looked quite reasonable.

More details are documented at the HBT web page:
http://www.star.bnl.gov/STAR/html/hbt_l/MDC3.

## Peripheral Collisions

The Peripheral Collisions Working Group got a late start since their
files weren't processed until the last few days of MDC3.

So far the group was able to show that they can analyze data and see
peaks for ^+-^, K^+^K^-^ (in half field) and f2(1270) ^+-^. The 's study
includes particle identification by dE/dx.

A number of fairly significant problems were found:

-   Very low efficiency. The problem appears to be in the
    ^2^-distributions for global tracks. The ^2^ don't make sense, and
    this greatly reduces the efficiency for vertexing (with a 2-track
    event, if you lose one track, there's no vertex). This needs to be
    fixed.

-   Unknown relation between the number of events processed to the
    number generated; events with failures don't get written out, and,
    if a job crashes, one doesn't know how many events were processed.
    This makes efficiency determination problematic. It's a particular
    problem because of the vertex problem above, which gives us a rather
    low efficiency.

-   There are still memory leaks. This significantly limits the number
    of events one can put into a file, to \~10,000 events.

-   Not able to reconstruct FTPC tracks because BFC calls a pre-vertexer
    before FTPC reconstruction. This pre-vertexer always fails for low
    multiplicity events, and when it fails, the FTPC code isn't called.

## Strangeness

The group made a lot of progress during the second week of MDC3. Having
all the experts located in one place meant there was a fast turn-around
in problem solving, bug fixing and exchange of ideas.

The group got off to something of a slow start not only because of the
well documented computer havoc of the first week but also because, due
to mix up, the enriched cascade events requested were sans cascades. The
group therefore concentrated initially on available events that did
contain strangeness=1. The STAR reconstruction team did regenerate the
cascade events very quickly once the \"mistake\" was realized and by the
time the computer system was back operational at the end of the first
week more than enough cascade enriched events were available. Several
thousand events were analyzed and DSTs produced.

The two big successes were getting the StrangeMuDstMaker up and running
and mostly de-bugged and the successful analysis of cascades and kink
reconstruction all the way through to having corrected distributions.

[DST Making]{.underline}

This was one of the major successes for the group from MDC3. Much work
was on improving the StrangeMuDstMaker.

1)  The DST is now written as a ROOT tree.

2)  Inclusion of the kink branch

3)  Speed up of access to members of branches. Now takes only a few
    seconds to plots variables for 1000 entries as opposed to hours.
    This means the DST is now operational for cut determination which is
    a major part of the analysis

[Kaon, , Kinks and]{.underline}

Using the StEvent created DSTs. Then using StMcEvent and
StAssociationMaker studies were made on the acceptance and efficiency of
reconstruction of all strange particles.

Points to note:

-   No major pit-falls were found in doing the analysis using these new
    codes. The group was able to perform quite detailed studies
    (especially in the case of the neutral kaons) to see exactly where
    all the kaons get cut in each stage of the reconstruction (namely
    acceptance, tracking, V0 reconstruction and off line cuts).

-   The efficiency of and kaon reconstruction has improved since MDC2.
    This is put down to improvement in the TPC tracking.

-   It was \"discovered\" that there were not only 50 extra s added to
    the events but 100. It seems there were 50 s per unit rapidity in
    the TPC acceptance. It was also \"discovered\" that there were
    anti-s but no s though.

-   There was, and still is, some confusion about what the input
    distributions were and how one extracts that information from the
    data. It was a 50-50 spilt amongst the experts over whether the g2t
    and particle table contains ALL generated particles or just those
    that passed through some active volume in GEANT. When the input
    distributions were plotted as a step towards an acceptance
    calculation they certainly weren\'t the flat distributions expected.

-   The strangeness group\'s adapted StSpectraMaker was used to produce
    acceptance and efficiency corrected plots for s and kinks. This was
    the first use of this tool and no showstoppers were found. The
    statistics used were certainly too low to produce any \"final\"
    results and there were some questions about how to do background
    subtraction to the spectra but all the tools are now in place and
    have been shown to work in principle.

-   There still needs to be some adaption of the spectra maker to be
    able to work with Strangeness DSTs not just StEvent.

The HBT and Strangeness group had reasonable success in merging their
DSTs production to allow kaon-kaon and -proton correlation studies. The
work was mostly slowed down in that the HBT group had to wait for the
Strangeness DST definitions to de finalized and then passed successfully
to the HBT makers. By the end of MDC3 though a steady production had
been achieved where both HBT and Strangeness DST (which have entirely
different forms) were being produced in one production loop. This type
of merging of groups with common goals will become more important as the
free-for-all that is currently the norm will become less and less
operational and some type of organized DSTs production will have to be
done.

[Final analysis]{.underline}

The Strangeness group is ready to do physics analysis as soon as the
data becomes available. All the tools are in place to look at the data,
study the efficiency and acceptance and perform the corrections.

## Spectra & High-pt

The spectra/high-pt group concentrated on developing tools for analyzing
data.

-   Single pion tracks were embedded into full Au+Au events. The vertex
    used for the pion track was found from a first-pass reconstruction
    of the Au+Au event.

```{=html}
<!-- -->
```
-   Tracks in the reconstructed output were examined to see if they
    could be associated with the embedded track. Counting those embedded
    tracks that are associated with a reconstructed track and those that
    are not, will produce the tracking efficiency.

```{=html}
<!-- -->
```
-   The output of the associator tool was changed to a ROOT tree so that
    a user can easily apply cuts that are specific to his/her analysis.
    For example, if a user requires the efficiency for tracks with nhit
    \> 20 and dca \< 1cm this can be achieved by placing cuts on info in
    the root tree. One does not have to go back to the associator.

```{=html}
<!-- -->
```
-   Efficiency numbers as a function of e.g. (y, p) have not yet been
    obtained.

```{=html}
<!-- -->
```
-   However, spectra have been generated using StSpectraMaker with
    generic efficiency numbers.

```{=html}
<!-- -->
```
-   MDC3 dE/dx was calibrated. However the calibration parameters are
    not yet in the database.

```{=html}
<!-- -->
```
-   L3 dE/dx was developed using straight-line approximation to the
    path-length.

```{=html}
<!-- -->
```
-   RICH PID software was released and exercised.

# Summary

With respect to the set goals, MDC3 was a successful exercise. It
revealed that despite difficult circumstances the Physics Working Groups
made solid progress and demonstrated that they are in principle ready to
start generating physics once real data gets recorded.

MDC3 brought together an amount of people working in parallel on data
analysis not seen in STAR at BNL so far. The STAR/BNL infrastructure
turned out to be sufficient for a group of people of 40-50.

MDC3 was overshadowed by extremely poor CAS/RCF stability. RCF in its
current state turned out not to be able to provide smooth and stable
conditions required for sustained data mining and analysis efforts.
Although this problem was, at least in part, already known, the exercise
of MDC3 confirmed unambiguously shortcomings at CAS, MDS, and during the
preparations at CRS. MDC3 will turn into an even bigger success if the
accumulated experiences can be used to improve the situation at RCF.

[^1]: CPU time for GEANT simulations roughly scales with the energy of
    the particles. With a flat rapidity distributions more energy flows
    into the forward directions.