**GRAM Plans for GT 4.0\
(Work in Progress: April 25, 2004)**

**Abstract**

TBD.

**Table of Contents**

[1 Requirements 2](#requirements)

[1.1 Improve Performance 2](#performance-improvements)

[1.1.1 Performance Aspects to Improve
2](#performance-aspects-to-improve)

[1.1.2 Planned Improvements 3](#planned-improvements)

[1.2 Improve Codebase Maintainability
4](#improve-codebase-maintainability)

[1.3 Improve Service Diagnostics 5](#improve-service-diagnostics)

[1.4 Improve Fault Tolerance 5](#improve-fault-tolerance)

[1.1 Add New Features 5](#add-new-features)

[1.5 Remove Features 6](#remove-features)

[Design Approaches 6](#design-approaches)

[2 6](#design-approaches)

[2 User Interface Diagram 9](#user-interface-diagram)

[3 ManagedJob scheduler interface 10](#managedjob-scheduler-interface)

[4 Karl GRAM Security Diagram 11](#karl-gram-security-diagram)

[5 Todos 12](#todos)

#   {#section .Style-Heading-1-+-Before:--0\"-First-line:--0\" .unnumbered}

#  {#section-1 .Style-Heading-1-+-Before:--0\"-First-line:--0\" .unnumbered}

# Requirements {#requirements .Style-Heading-1-+-Before:--0\"-First-line:--0\"}

We identify requirements in six areas: improvements to performance,
improvements to code base maintainability, improvements to service
diagnostics, improvements to fault tolerance, new features, and features
to remove.

## Performance Improvements

The GRAM tier should add little overhead to job submissions made
directly to a local resource manager, and should not be the limiting
factor compared to the network and the local resource manager.

###  Performance Aspects to Improve

We aim to improve performance in three areas: latency, throughput, and
concurrency.

1.  **Latency.** We define latency from the point of view of the client,
    therefore we propose to measure latency based on GRAM use cases and
    the interface definition of the services, i.e., the latency of:

    a.  **Job submission:** Measured as the interval between a client
        submitting a job and the client receives response that job is in
        GRAM "stageIn" or "pending" state. We propose to remove the
        start() operation in order to improve overall throughput and
        reduce the latency of the overall job submission. See more
        details below.

> Current submission latencies are 0.3s for GT2 (in what environment?)
> and XXXs for GT3.2. Our target is XXX.

b.  **Subscription to job state change notification:** This is directly
    dependent on the latency of calls subscribing to resource properties
    in the Core GT framework, and therefore is not specific to GRAM.

c.  **Job state change notifications:** Measured as intervals between
    state changes such as "client receives response that job is pending"
    and "client receives notification that job is active." The state
    change notification latency is mostly due to the state change
    discovery in the GRAM tier, not in the delivery of the notification
    message to the client.

d.  **Job destruction:** This is directly dependent on the latency of
    service/resource\
    destruction calls in the Core GT framework.

> Current cost is XXX; target is XXX.

2.  **Throughput (in number jobs/sec)**. [ADD CARL's COMMENT]{.mark}
    (pov of client, etc) Throughput is related to latency, and improving
    both aspects of performance go hand-in-hand. We define throughput as
    the number of jobs that have achieved a specific state per second,
    which leads us to the following measurements:\
    - number of jobs in the "pending" state per second. This corresponds
    to the job being in the "queue" of the GRAM tier and\
    - \....etc....**\
    **- number of jobs reaching the "done" state per second

> Current GT3.2 throughput: [TO FILL]{.mark}\
> Required GT4 results: [TO FILL]{.mark} more
>
> Our targets are a burst submission rate of 100 job/sec and a sustained
> submission rate of 1 job/sec. Ideally, GRAM should enable as high a
> job submission rate as the local resource manager would by itself.

3.  **Concurrency (in number of jobs).** We define concurrency for GRAM
    as the maximum number of jobs that can simultaneously be managed by
    the GRAM tier of the job submission architecture.\
    [INSERT PICTURE]{.mark}

> Current GT2 concurrency:

-   Fork: max 980 jobs

-   PBS: max 219 jobs\
    Multiple concurrent execution of kill (Fork) or qstat (PBS) by RIPS
    account in great part for the poor concurrency results. See the test
    details at
    http://www-unix.globus.org/ogsa/tests/scalability/seperate/scal_gt2.html

> Current GT3.2 results:
>
> **[TO FILL]{.mark}**\
> Required results for GT4:

-   Number of jobs in "pending" state (i.e. in the queue) 10,000

-   Number of jobs in "active" state

1.  w/o stdout/err redirection 1,000

2.  with stdout/err redirection 100

> Ideally GRAM would manage as many simultaneous jobs as the\
> local resource manager would by itself.

### Planned Improvements

The following improvements are planned for GT4.0 to address the
requirements just listed.

1.  **Reduce the memory footprint of an individual job.** In GT3, each
    job is implemented as an Axis Web service, of significant memory
    footprint. This can lead to exhaustion of memory resources before a
    useful number of jobs is managed by GRAM. We plan to reduce this
    memory footprint so as to enhance concurrency.

2.  **Reduce the number of client-to-GRAM round-trips per job
    submission.** We have currently a total of six roundtrips (RTs), as
    follows.

    a.  1 RT to have the MMJFS create an MJS and return its GSH

    b.  Secure conversation: 2 RTs to set up SSL from client to MJS

    c.  Delegation: 2 RTs, one as a flag for the client to
        enable/disable delegation, one for the delegation itself
        (response not required, but one-way messages are not available
        in Web service stack yet).

    d.  1 RT to invoke start operation and actually submit the job.

> By making secure conversation and credential delegation optional, the
> corresponding round-trips can be eliminated when the features are not
> needed for a given job submission.
>
> In GT3 the create operation on the MJFS port type ensures creation of
> a job but a potentially unreliable messaging layer may not deliver the
> response back to the client. This is the reason behind the existence
> of the start() operation on the job, which actually submits the job to
> the scheduler. We want to remove the need for this extra
> request/response round-trip between the client and the GRAM service
> (so as to reduce latency of the overall job submission process). In
> order to do that, we must of course devise an alternate method of
> ensuring that jobs are not orphaned from client control. We describe
> such a mechanism below. \[[RIGHT]{.mark}?\]
>
> We therefore expect to reduce the number of total round-trips from 6
> (GT3.2) down to only 1 for job submissions without secure conversation
> and delegation.

3.  **Provide the administrator with fine-grained control to throttle
    the load caused by processing jobs.** GRAM adds to the load on the
    cluster's head node when processing jobs. We plan on providing
    configuration options to limit the load. Note that limiting load
    could also limit throughput.

4.  **Pay no performance penalty if a feature is not requested by the
    user.** . . Certain features of GRAM such as stdout/err streaming or
    credential delegation incur an overhead to job submissions even when
    they are not actually used by the job submitter. We want to make
    such features inactive unless explicitly required by a given job
    submission.

5.  **Reduce load on the head node.** To process a job, GRAM performs
    many commands/operations, many of which are executed on the
    clusters's head node. Under heavy job concurrency, the head node can
    become unresponsive. Where possible, we plan on executing commands
    on the cluster's compute nodes. We will also work to make the
    commands more efficient.

6.  **Provide a better performing implementation of the
    managed-job-globusrun client.\
    **The existing implementation of the managed-job-globusrun client is
    written in Java and therefore suffers from the weight of the Java
    Virtual Machine when it comes to executing the command-line client
    many times. As a consequence, client throughput (from one given
    host) is poor. We will address this problem by providing a C
    implementation of managed-job-globusrun.

7.  **Reduce delay between job state change and notification to managed
    job.** There is a noticeable delay between when a job state changes
    and when GRAM sends out notification of the state change. This
    increases latency of state change notifications significantly.

8.  **Factor security setup over many submissions instead of doing it
    for each one**. If a client plans to submit a burst of jobs to the
    same GRAM service, security overhead per submission should be
    reduced by setting up the security context once and for all
    submissions.

9.  **Improve performance of stdout/stderr streaming.\
    **

## Improve Codebase Maintainability

TBD.

## Improve Service Diagnostics

Provide the means for a client to determine what is and is not working
with the GRAM service.

Note here, perhaps, the Grid3 requirement: "**API for troubleshooting
and accounting** information. APIs are needed particularly for GRAM and
GridFTP, the job submission and file transfer systems, respectively.
These APIs should provide *direct* information without the necessity of
parsing log files."

## Improve Fault Tolerance

1.  **The server must behave better under overload conditions**, i.e.
    conditions when more requests are received by the GRAM job factory
    service that it can handle.

2.  **Improve handling of out-of-memory conditions.** Currently,
    OutOfMemory errors are thrown by the Java virtual machine of the MHE
    as it runs out of memory after one-too-many request for job
    creation.

> [INSERT threshold of burst of jobs which cause this.]{.mark}
>
> It seems the problem is caused by too many security context objects
> being kept in memory by Axis. Attempts to monitor the memory to
> intercept the problem before it cause a crash have been unsuccessful
> so far. More research needs to be done on this problem.

3.  **Improve recovery of managed job hosting environment.** Fail over
    in GT3 has been implemented as a recovery mechanism for User Hosting
    Environment, i.e., the JVM acting as a sandbox for user-based job
    management. However this model suffers from some stability issues
    and needs to be revised. The current thinking is to remove the
    user-based JVM model and use one JVM only (we are not talking about
    replication here) for all users.

## Add New Features

1.  **Add the ability of processes in a job to self organize**. A new
    feature of the Web services Globus toolkit is an optional
    application runtime library that permits the tasks (processes) of a
    parallel job to rendezvous and exchange simple messages for
    bootstrapping purposes. This will be Web service-enabled equivalent
    to the GT2 GRAM component 'myJob', but in a form initially
    restricted to suit most rendezvous scenarios of which we are aware,
    while minimizing the technological dependencies of the runtime
    library (to minimize impact on applications where this functionality
    is desired).

2.  **Add the ability for a job to communicate its application-specific
    status.** A requirement from the client point of view is to have the
    job expose not only the abstract GRAM job state but also its
    application-specific state, as defined by the application being
    mandated to run by the job description.

3.  **Make the history of all job states for a given managed job
    available to the client.**\
    A requirement from a client point of view is to be able to trace the
    last n GRAM states in which it were. Since the number of job states
    in the GRAM model is finite and small, and since the job state
    machine is linear and acyclic, we propose to have the job keep a
    full history of its previous states.

4.  **Make the exit code of a job available to clients.** A requirement
    from a client perspective is to be able to query the exit code
    returned by the application executed by the job.

5.  **Provide clients with the ability to target individual nodes in a
    cluster as allowed by the cluster admin.** Provide the ability for a
    client to specify cluster node types and processes per node. For
    example, 2 ia64Nodes with 4 proccesses + 4 ia64MicroMuralNodes with
    8 processes

## Remove Features

I.  **File staging redundancy management is out-of-scope.** GRAM will no
    longer maintain a GASS cache for users. Coordination of sharing
    files staged between jobs must be managed by the client. Testing of
    GT3.2 indicated that the GASS cache might be reducing throughput.
    Removing it is therefore expected to improve throughput.

II. **Usage of http URLs for file staging and streaming.** The http
    protocol is to be dropped for file staging and streaming. GT4 will
    include a GridFTP server that can be easily run by a user, similarly
    to the way the current GASS server program is run.
    managed-job-globusrun will be changed to stage files using the
    GridFTP protocol.

# Design Approaches

1.  **Move to WSRF (Requirement 1.a.1).** GT4 will follow the WSRF model
    for interfacing systems using services and resources. A revision of
    the Globus toolkit Core framework is being implemented so as to
    offer developers an easy way to implement WSRF services. Of course
    GRAM is going to be ported to WSRF using that framework, which means
    new expression of the service interfaces using WSRF-like WSDL.
    Besides, adopting the GT WS-Core WSRF programming model will make
    service implementations less memory-intensive: a managed job will
    not be a heavy Axis service anymore but a lightweight resource
    object used at invocation time by a stateless service. This will
    reduce the footprint of managed jobs and thus improve concurrency.

2.  **Make the number of threads operating on a run queue configurable
    by the administrator (Req. 1.a.2).** We would like to improve the
    liveness of the run queue implemented in the GRAM tier. We need more
    investigation as to what can be achieved, but it is expected that
    performance will be improved by adding more threads to the run
    queue. Since it is difficult to predict anything we envision the
    implementation as being based on a thread pool which resource usage
    (i.e. number of threads) is not hard-coded but on the contrary
    configurable.

3.  **Use RFT for file staging (Req. 1.a.5).** RFT stransfers files will
    occur through a GridFTP server. The server should be run on a host
    other than the head node.

4.  **Eliminate overhead of stdout/err file streaming when the feature
    is not requested (Req. 1.a.4).** File streaming for stdout/err of a
    job is always somehow activated in the implementation of the managed
    job even when no streaming is dictated by the job description. We
    want to get rid of that overhead. We are planning on removing file
    streaming as it is now and replacing it with GridFTP-based file
    streaming, performed actively by the client.

5.  **Remove start() operation and instead use a unique job identifier
    on create() (Req. 1.B.2).** Removing the start operation means
    making the job creation operation will have to submit the job to the
    scheduler, which leads to a potential reliability issue. In fact, if
    the response message gets lost while on its way to the client, the
    client will not obtain a handle to the job service/resource, and
    will loose every possibility of control. We therefore plan to make
    the create operation reliable by:

    a.  Having the client insert a new RSL element "MessageID" (a unique
        job identifier created by the client and checked by the service)
        into the RSL for each new job creation (the RSL is chosen as a
        placeholder so as not to make lower layers of the Web service
        stack proprietary and reduce interoperability).

    b.  Having the create operation check the ID against a list of
        previously started jobs. If the ID exists, return a fault to the
        client with the existing handle to the managed job. If the ID is
        new, create a new managed job and return the handle to the
        client.

    c.  Having the client retry the job creation call using the same ID
        if the response from the initial call gets lost.

6.  **All jobs of all users will run in a single hosting environment
    (Req 1.a, 1.b, 1.c)**

    a.  Exploit sudo when needed, else use scheduler submission programs

        i.  PBS will need sudo

        ii. Need to deal with users who don't have sudo installed

    b.  With this design approach, GRAM4 lives or dies by its throughput
        capabilities!

        i.  An individual user or subset of users cannot consume
            available resources**;** there is a scheduling problem
            implicit here; as the managed job service is throttled, who
            gets to run?

    c.  Delegation model will need to change; 3 scenarios:

        i.  Job needs no credentials

        ii. Job should run with GRIM credentials

        iii. Job should run with proxy cert delegated from requester

             1.  In case 3, requester should delegate its right without
                 knowing the actual user. Use GRIM as a trusted
                 intermediary:

                 a.  GRIM uses a proxycert received from the *requester*
                     and issues 2 different proxy certificates to the
                     *user* both based on the GRIM keypair generated for
                     that account:

                     i.  a *grim-proxycert* chains to the GRIM identity

                     ii. a *requester-grim-proxycert* chains to the
                         requester identity

                 b.  Requester generates proxycert based on the GRIM key
                     and passes it to a hosted service that can securely
                     (setuid?) deliver it to GRIM.

                 c.  User asks GRIM to refresh its short-lived
                     requester-grim-proxycert

        iv. Must still support client refresh of the proxy delegated to
            GRIM

7.  **Make Proxy delegation disabled by default (Req. 1.a.4)**

8.  **Optimize interface to schedulers (Req. 1.a.7).** The periodic
    execution by RIPS of the queue inspection command(s) of the back-end
    scheduler adds a significant overhead to the architecture. RIPS
    keeps tracks of previous job states and compare each polled results
    with the previous state of the queue so as to infer the new states
    of the jobs. This is causing serious scalability issues.

> We want to remove RIPS and replace it with a scheduler-specific push
> model. Ideally the scheduler (or a GRAM-specific plug-in to the
> scheduler) would notify the job manager, for instance through email
> notifications. Unfortunately not all schedulers offer notifications.
> For instance Platform LSF does not support any form of push model,
> however it seems feasible to write a command executed by LSF to
> monitor a LSF log file periodically. While still a polling type of
> monitoring, this is much better than comparing new and old queue
> information as RIPS does.

9.  **Optimize multiple job submissions by running under a single Secure
    Context (Req. 1.a.8).**

10. **File Streaming is no longer part of GRAM, but will now be provided
    by GridFTP (Req. 1.a.9)**

    a.  Remove possibility of URLs for stdout/err?

        i.  User includes local file(s) in RSL stdout

        ii. client does remote tail --f on local files(s).\
            Note: this will require a small addition to the GridFTP
            protocol, which will be implemented in the new GT4 GridFTP
            server.

11. **File Staging is no longer part of GRAM, but will now be provided
    by GridFTP (Req. 1.b)**

    a.  Will require a GridFTP server that can be easily run as a user,
        similar to the current globus-gass-server used by
        managed-job-globusrun.

    b.  Will Staging be done using RFT?

12. **Remove overhead of file streaming when disabled (Req 1.a.4)**

    a.  When stdout/err streaming not used in RSL, there should be no
        cost.

13. **Add a new resource property to hold the history of all past job
    states (Req 1.e.3)**

14. **Add a new resource property to hold the exit code of the user's
    job (Req. 1.e.4)**

15. **GRAM myJob (Req. 1.e.1).** Support MPICH-G2 using only web service
    client libraries (simpler implementation):

    a.  compose ServiceGroup and ServiceGroupRegistration into
        ManagedJob port type

    b.  service group members describe tasks/subjobs and identifiers/EPR
        to reach them. Member tasks consult group to find peers.

    c.  Enable further evolution to DUROC.

16. **Add RSL selection criteria for node types in a cluster (Req.
    1.e.5)**

17. **Add diagnostic tools to the GRAM service (Req 1.c)\
    **To help a client or sys admin diagnose where GRAM is failing, the
    service needs to respond to a set of "dry run" job submission. Given
    a job submissions, a client can have the service stop at different
    job submission milestones:

    a.  **After completing authentication and authorizations**

    b.  **After accessing the local user account**

    c.  **After file staging**

    d.  **Before submitting the job**

    e.  **After submitting and then canceling the job?**

# User Interface Diagram {#user-interface-diagram .Style-Heading-1-+-Before:--0\"-First-line:--0\"}

![](media/image1.png){width="5.990972222222222in"
height="4.4527777777777775in"}

> **GridDelegate** - a new command line program to delegate a remote
> credential via the delegation service.
>
> Globusd - Globus Hosting Environment

-   **SJW** (SimpleJobWorkflow service) - accepts a similar RSL to GT
    3.2 and performs the same functionality while using *only* other
    available services (RFT, TC, MJ). It has become a composite service
    (workflow service) using a set of finer grain services.

-   **RFT** (ReliableFileTransfer service) - performs third party
    transfers over GridFTP servers. Can be used for files staging and
    remote directory creation.

-   **TC** (TaskCoordination service) - provides a generic checkin and
    barrier service. For example, it can be used for process
    coordination and job coordination for an mpich-g2 job.

-   **MJ** (ManagedJob service) -- It will understand only a subset of
    the GT 3.2 RSL only those attributes specific to executing and
    monitoring a scheduler job. It will convert the RSL into a scheduler
    job description and submit the job under the users account (via
    sudo). It will monitor the job and send out job state change
    notifications.

# ManagedJob scheduler interface {#managedjob-scheduler-interface .Style-Heading-1-+-Before:--0\"-First-line:--0\"}

![](media/image2.png){width="5.992361111111111in"
height="4.179861111111111in"}

> schedd - scheduler Hosting Environment?
>
> **eventd --** The eventd will do a tail --f on the scheduler log file,
> parse out job state events and send them to the JobMonitor
>
> Globusd - Globus Hosting Environment
>
> **MJ --** The ManagedJob service will start a background thread to
> monitor receive the job state events from the eventd.
>
> **JobMon --** The JobMonitor is a background thread that will update a
> ManagedJobResources state and send a job state change notification to
> the client.
>
> Question: Why not just put the eventd functionality inside the
> JobMonitor?

# Karl GRAM Security Diagram {#karl-gram-security-diagram .Style-Heading-1-+-Before:--0\"-First-line:--0\"}

![](media/image3.png){width="5.99375in" height="4.638888888888889in"}

Please view it in color.

Here are the security \"zones\" (accounts):

client

globusd, where the GRAM WSRF service container runs

grimd, where the GRIM2 WSRF service container runs (\*)

schedd, where the platform specific scheduler and log crawler runs

user1, where the example user jobs run

(\*) I don\'t know if GRIM and the globusd will be separate zones?

GRIM2 to distinguish this fictional future thing from the GT3 GRIM

green arrows represent WSRF invocations.

green rounded boxes are WSRF services.

blue rounded box is unix-style daemon.

dog-eared boxes are files/data.

courier font words are programs/command invocations.

Here\'s the story we are trying to convey:

1\) user account gets credentials with help of GRIM2

a\) by pulling credentials via a setuid command (like in GT3?)

b\) by GRIM pushing credentials into the account (via sudo?)

2\) I\'ve assumed that all user creds are stored in the user account,
rather than having GRIM be a repository for client-derived credentials.
But I left an unknown \"cred3\" method in case there is some other story
brewing?

3\) GRAM job submission never does anything w/ delegation from client

4\) user jobs CAN access any credential available in user account

5\) GRAM job resource runs all job actions (staging, program) as user
via sudo

6\) eventd crawls over logfile in schedd account and pushes
notifications into RIPS2. RIPS2 is a simple demuxer, rather than the
complicated soft-state comparator that it is in GT3. RIPS2 knows the
local jobID\--\>GRAM job EPR mapping.

7\) RIPS2 is only used w/ eventd. w/ fork interface, simple child
process signals should route state change directly to job?

8\) GRIM2 owns mapfile and answers authz callouts for GRAM factory?

# Todos {#todos .Style-Heading-1-+-Before:--0\"-First-line:--0\"}

1)  **Iterate on doc design -\> You!**

2)  **Resolve open design issues -\> Stu, Bill, Steve, Karl, Sam,
    Jarek**

    a.  GridFTP mods needed in support of file streaming: bill, steve,
        karl

    b.  Duroc: Stu and Karl

    c.  Specify termination beor in WS-Core: Sam, Jarek

```{=html}
<!-- -->
```
i.  Soft-state: ged job expires after a specified duration

ii. Explicit destruction by client

iii. Automatic sweep of managed job when job terminates**?**

```{=html}
<!-- -->
```
3)  **Immediate implementation tasks -\> Stu, Joe, Peter, Alain**

    a.  Turn Design Approaches into campaigns (Stu)

    b.  Work out scheduler adapter interface (Joe)

        i.  Forked case

            1.  Get rid of polling

                a.  Does java have a way to get notified on process
                    completion

        ii. Scheduler adapters

            1.  monitoring

                a.  parse the log file

                b.  communication path btw log scanner and java code

                    i.  bootstrapping process

            2.  submissions

                a.  modified perl scripts

    c.  Work on GRAM benchmark tests based on Karl's design (Peter)

    d.  Prototype 3.2 MJS using sudo fetching a cred from MyProxy
        (Alain)

4)  **Check Sanity -\> You!\
    The following will not be included in GT4.0 unless someone makes a
    good case**

    a.  Patches aimed at fixing NFS errors

    b.  Dynamic service deployment

    c.  Advanced reservation

    d.  Job arrays

    e.  Ability for a job to communicate its application-specific status
        (i.e., Managed job publishing an EPR (or more) to the web
        service that is started by the app itself

5)  **Create Implementation Timeline -\> Stu and Alain**

6)  **Conduct Performance Study**

    a.  Review GRAM benchmark tests

    b.  Run tests on 3.0 and 3.2