PHOBOS Report on New CRS

11/13/23 3:12 PM

Marguerite Tonjes: belt@bnl.gov

Contents:

Section 1) Recent New CRS Testing Report

Section 2) Suspend/Freeze Discussion

Section 3) GUI problems

Section 4) Testing Still To Do

### Section 1) Recent New CRS Testing Report

We issued jobs as follows:

Date: \~number of jobs

Oct. 18: \~400

Oct. 19: \~880

Oct. 20: \~400

Oct. 21: \~200

Oct. 22: \~400

(Oct. 20 was resetting of error jobs and new jobs, Oct 21 and 22 was
reset jobs)

Initially, I issued jobs using --create_and_submit. That threatened to
take down the new CRS system, so I switched to using --create, and let
the automatic submission daemon handle the submission.

I learned to not pay attention to "INCONSISTENT" jobs. Those appeared
and went away before job completion, being self-corrected.

Problems encountered and solution (if applied):

-   On Oct. 19^th^, about 11pm-2am, a number of jobs failed during
    export due to the HPSS DST cache having filled.

    -   fixed: Tomasz added check: when exporting files, crs will check
        if hpss is full. If it is full it will sleep for one hour.

-   3 jobs failed to stage data

    -   HPSS experts looking into it

-   74 jobs failed due to a "Phobos Checksum" error, where we compare a
    checksum of the Raw file to one from our database, all got checksum
    of zero

    -   Under investigation

-   "archive" and "kill" did not save Phobos output files. Tomasz had
    asked the experiments for specifications. I gave some. Please to
    discuss the features.

    -   "archive" is now:

        -   Only jobs in DONE state can be archived.

        -   All files from archived jobs are tar-ed and moved to
            directory /direct/newcrs/phobreco/archived. This directory
            can be changed if you want another one. In the future I
            expect it to fill up rapidly, so I will add a script that
            allows for only 10k archived jobs. If there are more than
            10k, it will delete starting from the oldest.

        -   I haven't tested this, can do once the Phobos upgrade is
            finished. There's only a /direct/newcrs/phobreco/archieved
            directory right now with a production log, no tarred files.

        -   I don't expect to find our .log, .err, and .root files in
            the tar, only the new CRS .err, .out and other files one
            sees in "spy execdir".

    -   "kill" is supposed to:

        -   keep our Phobos .log and .err file along with new CRS .err
            and .out files

        -   be performable on jobs in any state

        -   Haven't tested this, not sure if the fix is in place. This
            is needed to test our "Phobos checksum" error.

-   We had "notify" in the job definition files set to
    phobreco@rcrsuser2, that was changed (a lot of email error messages
    came with rcrsuser2 being offline)

-   \~100 of our jobs failed due to wrong error permissions in the HPSS
    DST output directory, this was fixed, and the jobs are reset,
    waiting to run after the upgrade

-   We had some issues with job suspension/freezing, which led to
    "suspend/remove" being removed from the main panel as Tomasz said it
    used the Condor suspend which he does not trust. See Section 2)
    Suspend/Freeze Discussion below.

    -   Needs fixed -- see below

-   Web page instructions need to reflect current new CRS

-   Web page new CRS status didn't reflect correct job status

    -   Been fixed. Also includes list of running subjobs and what node
        they are running on, as requested: very useful, and was in old
        CRS

    -   Will this also show suspended subjobs and node? (suspended jobs
        still take up memory and *may* affect whether general users'
        jobs will run on Condor)

    -   check this after Phobos upgrade finished

### Section 2) Suspend/Freeze Discussion

We need a fast suspend/freeze. I list below the cases I can think of for
such a thing and the possible implementations that I can think of. In
some cases, this maybe should be a line command, as it would be faster
than selection and processing in a GUI. This is to incite discussion of
how to solve this problem.

1.  NFS problems

    a.  Suspend everything, parent as well as subjob on every machine -
        resume once NFS problem is fixed

    b.  Want to do this quickly to avoid job failure

    c.  Was possible with "suspend/resume" on main panel in old CRS

2.  HPSS problem or want HPSS resources for other things

    a.  Use HPSS flag on/off

    b.  Was not in old CRS - \> selected all jobs and did
        "suspend/resume"

3.  Want CPU to be able to be used elsewhere for some machines or a
    queue

    a.  Make machines unavailable for new jobs

    b.  Suspend jobs stored by machine or queue

    c.  In old CRS -\> had to do this on a job by job basis, using the
        lower part of the panel which shows what machine each job is
        running on

4.  Output NFS disk is full

    a.  Does HPSS flag off also mean NFS files not transferred?

    b.  Want to be able to suspend everything such that no job finishes
        and copies files to NFS disk

5.  Network problem symptom:

6.  Cannot communicate with Phobos database (important at start and end
    of subjob)

    a.  Suspend running jobs

    b.  Don't want any new jobs to start running (use HPSS flag for
        this)

7.  Cannot communicate with NFS machines (see NFS problems above)

8.  Cannot communicate with AFS

    a.  Usually this is not an issue for Phobos (in the past there were
        some commands that were on AFS, but I think they've all been
        fixed)

9.  Cannot communicate with Condor

    a.  What happens?

10. Cannot communicate with new CRS database

    a.  How does one suspend if that happens? Assuming the
        non-communication is for a long period of time, i.e. not just a
        few "inconsistent" jobs

Suspend/Freeze Conclusions:

-   Need to be able to keep jobs from automatically starting

-   in old CRS this was done by suspending the job submission script

-   in new CRS that will have to be done too, but since jobs can go from
    "created" to "started" with the daemon running, that may require
    being able to suspend the daemon or something

-   Need to be able to suspend running jobs

-   HPSS flag on/off will be useful in freezing jobs

Making Machines Available/Unavailable:

-   Have not tested this in the current implementation

-   There may be issues with making machine unavailable in the new CRS
    affecting jobs running in Condor

-   Need to have non-CRS Condor jobs running as well to test this fully

Reasons to do this:

1.  A single rcrs machine goes offline for disk problems

2.  CPU of those machines is to be used for non-CRS purposes: i.e. a
    shift in use priority for Physics or other reasons

### Section 3) GUI problems

1.  GUI appears as pictured below on my screen. Other screens cut off
    different amounts of the right-hand set of buttons. Resizing the
    width doesn't help.\
    ![](media/image1.png){width="4.80625in"
    height="3.5236111111111112in"}

2.  Show pftp system command brings up two windows, the front one blank,
    the other saying no pftp jobs (job in error state)

    a.  Show hpss button has same issue

3.  Machines panel takes too long to refresh

4.  If waiting for machines panel to refresh -- main panel does not work

    a.  Maybe make them separate?

### Section 4) Testing Still To Do

1.  In old CRS, Phobos created and submitted job in the same step.
    During data taking, this meant (presuming there was free CPU) that
    the jobs would run while the Raw data file was still on the HPSS
    cache. With the current "submit daemon" this may not be the case.

2.  Need to test 1000+ jobs

3.  Mock Data Challenge with all experiments processing and sinking
    "data" at the same time

    a.  make sure this doesn't overwhelm the local job storage
        directory:

    b.  rcrsfm02.rcf.bnl.gov:/home/newcrs 102G 5.7G 96G 6%
        /direct/newcrs

    c.  currently: 2747 jobs: 2037 done, 461 created, 248 error, 1
        submitted

        i.  largest directory: 20 MB (unusually large .err file: 100
            events only!)

        ii. average directory size (without 3 unusual large files): 1.5
            MB

        iii. median directory size: 1.7 MB

    d.  How many jobs will we have in the system (in any stage) at any
        one time?

4.  Test other failure modes

5.  Test submitting jobs to same few machines/queue with different
    priorities

    a.  What do the different priorities mean?

6.  Using Condor and new CRS on the same machines, see how they interact

    a.  is CRS configured to always over-ride Condor, or do we have to
        be careful about job priority to do that?

### Section 5) Non-Essential problems/errors

Some of these may be useful, that's up for discussion.

1.  Does setup_crs need to be executed in rcrsuser4:
    /home/phobreco/.cshrc or /phobos/u/phobreco/.cshrc for the new CRS
    commands to work in jobs and crs_panel to work on rcrsuser4? Or
    should it be a universal command for all rcas, rcrs, and rcrsuser
    machines?

2.  GUI: crs_panel --w80 and crs_panel --w150 show no difference

3.  GUI: resizing window, buttons & job name area do not scale,
    horizontal scroll bar does not work (our job names are short enough
    to fit in the job area)

4.  GUI: After sorting by status or timestamp, it would be nice if one
    could sort back to the original default sorting which doesn't appear
    to be name, status, time, or timestamp.

5.  GUI: Sort in reverse direction for status, name, time, timestamp,
    and default

6.  GUI: machine: sort by subjob name, sort by timestamp

7.  GUI: machine: show queue machine is in

8.  GUI: in spy execdir, doing a "cat" on a selected file is sometimes
    time consuming for a very long file, maybe "tail" as well as "cat"

    a.  In a few jobs in the past, we've had continual repeating error
        output, making a 40GB+ error file. Doing a "cat" on that file
        will probably take so long to process, it would seem as if the
        GUI is frozen

9.  Remove the requirement that a job must have an input file. There are
    several types of jobs which can run without needing CRS to prepare
    any input files

10. Text command to spy execdir of all executing jobs, or ones with
    certain wildcard parameters for the jobname

    a.  tail .log, .err

    b.  cat .log, .err

    c.  spy execdir files by choosing machine subjob is running on.