# Presentation: 085522

## Globus Advanced Administration Tutorial

- Charles Bacon
- GRIDS Center

## Overview

- Client interfaces to the Globus Toolkit:
- Gatekeeper clients
- Resource Specification Language
- Metacomputing Directory Service clients
- GridFTP clients
- Server administration
- Cluster deployment and configuration
- Firewalls
- Running your own CA

## What do we already know?

- Elementary Packaging
- Default installation of the Globus Toolkit
- Requesting certificates
- Configuring the grid-mapfile
- Verification of a correct installation

## What are we trying to do?

- We would like to use remote resources, in a scheduler-independent fashion
- Prerequisites:
- The remote site has installed/configured the Globus toolkit
- We have obtained certificates
- Our certificate subject maps to an account on the remote machine (grid-mapfile)
- We ran grid-proxy-init

## Client Tools

- globus-job-run
- globus-job-submit
- globusrun
- All specify a host to connect to, and a command to run there
- So what’s the difference?

## globus-job-run

- The globus-job-run client is a sample GRAM client that integrates GASS services for executable staging and standard I/O redirection, using command-line arguments
- % globus-job-run pitcairn.mcs.anl.gov /bin/ls
- % globus-job-run pitcairn.mcs.anl.gov –s myprog
- % globus-job-run pitcairn.mcs.anl.gov \
-          –stdin –s in.txt –stdout –s out.txt –s myprog

## globus-job-run

- Runs in the foreground, delivers output directly
- Even if you redirect with –stdin –stdout
- In the last example, if myprog took 5 minutes to run, the command would have stayed in the foreground until the command completed
- Connects to default gatekeeper port 2119, and requests default jobmanager

## What is the jobmanager?

- $GLOBUS_LOCATION/libexec
- Using PBS as an example ...
- globus-script-pbs-submit
- globus-script-pbs-poll
- globus-script-pbs-rm
- globus-script-pbs-signal
- globus-script-pbs-queue
- List of configured jobmanagers in $GLOBUS_LOCATION/etc/grid-services

## Default jobmanager - fork

- Fork jobmanager implemented in C code
- Simply spawns a job on the local host, doesn’t go through a scheduling mechanism
- Uses UNIX interfaces to keep track of child status
- Configured by default by $GLOBUS_LOCATION/setup/globus/setup-globus-gram-jobmanager

## Adding Other Jobmanagers

- $GLOBUS_LOCATION/setup/globus/setup-globus-gram-jobmanager
- Takes a –type= parameter
- Name will be jobmanager-type
- E.g., jobmanager-pbs
- Currently, default jobmanager should be fork jobmanager if you want to use globus-job-get-output

## Contact Strings

- Used to reach nonstandard gatekeeper ports and request specific jobmanagers
- host[:port][/jobmanager][:subject]
- E.g., server.test.edu:8188/jobmanager-condor
- A verbose way to get standard behavior:
- globus-job-run server.test.edu:2119/jobmanager /bin/echo “Hello, world!”

## globus-job-submit

- What if I don’t want to run in the foreground?
- Output needs to be stored somewhere
- globus-job-submit server.test.edu /bin/echo “Hello, world!”
- Returns a job contact; an opaque string
- “https://server.test.edu:2173/21737/1011301465/”
- globus-job-get-output <job contact>
- Collects the data at a later date
- globus-job-status <job contact>
- ACTIVE, PENDING, SUSPENDED, DONE, FAILED
- globus-job-clean <job contact>

## GASS Cache

- Where did that output get stored?
- ~/.globus/.gass_cache
- What happens when I run globus-job-get-output?
- Our orignal jobmanager and job have completed and exited, but left some files behind
- Submits a new job which fetches the files left behind (using globus-gass-server) and streams the result back to the client

## Error code 73

- Could not open stdout/stderr
- This is a GASS error
- Error in connecting back to my host
- globus-hostname used to contact me.  If it is wrong, connection will fail
- Can override with $GLOBUS_HOSTNAME
- Or, error in creating/reading ~/.globus/.gass_cache
- Can happen immediately (with globus-job-run) or delayed (with globus-job-submit)

## globusrun

- Sits underneath the other submission tools
- Used to submit jobs to gatekeepers
- Uses Resource Specification Language (RSL) to encode options
- globusrun -r contact-string “RSL”
- globus-job-run host command
- globusrun –o –r host ‘&(executable=command)’

## Resource Specification Language

- Much of the power of GRAM is in the RSL
- Common language for specifying job requests
- A conjunction of (attribute=value) pairs
- GRAM understands a well defined set of attributes

## Example RSL

- &(count=1)
- 		(executable=/bin/echo)
- 		(args=“Hello, world!”)
- 		(environment=(MYFLAGS=“-l -a”))

## RSL Attributes For GRAM

- (executable=string)
- Program to run
- A file path (absolute or relative) or URL
- (directory=string)
- Directory in which to run (default is $HOME)
- (arguments=arg1 arg2 arg3...)
- List of string arguments to program
- (environment=(E1 v1)(E2 v2))
- List of environment variable name/value pairs

## RSL Attributes For GRAM

- (count=integer)
- Number of processes to run (default is 1)
- (hostCount=integer)
- On SMP multi-computers, number of nodes to distribute the “count” processes across
- (project=string)
- Project (account) against which to charge
- (queue=string)
- Queue into which to submit job

## RSL Attributes For GRAM

- (maxTime=integer)
- Maximum wall clock or cpu runtime (schedulers’s choice) in minutes
- (maxWallTime=integer)
- Maximum wall clock runtime in minutes
- (maxCpuTime=integer)
- Maximum CPU runtime in minutes

## RSL Attributes For GRAM

- (maxMemory=integer)
- Maximum amount of memory for each process in megabytes
- (minMemory=integer)
- Minimum amount of memory for each process in megabytes

## RSL Attributes For GRAM

- (jobType=value)
- Value is one of “mpi”, “single”, “multiple”, or “condor”
- mpi: Run the program using “mpirun -np <count>”
- single: Only run a single instance of the program, and let the program start the other count-1 processes.
- multiple: Start <count> instances of the program using the appropriate scheduler mechanism
- condor: Start a <count> Condor processes running in “standard universe”

## RSL Attributes for GRAM

- (save_state=yes)
- Causes the jobmanager to save job state/information to a persistent file on disk
- Recover from a jobmanager crash
- (two_phase=<int>)
- Implement a two-phase commit for job submission and completion
- Int=seconds to wait before job times out

## RSL Attributes for GRAM

- (restart=<old jm contact>)
- Start a new jobmanager but instead of submitting a new job, start watching over an existing job.
- (stdout_position=<int>)
- (stderr_position=<int>)
- specified as part of a job restart
- restart file streaming from this byte

## RSL Substitutions

- RSL supports simple variable substitutions
- Substitutions are declared using a list of pairs
- (rslSubstitution=(SUB1 val1)(SUB2 val2))
- A substitution is invoked with $(SUB)
- Processing order:
- Within scope, processed left-to-right,
- Outer scope processed before inner scope
- Variable definition can reference previously defined variables

## RSL Substitution Example

- This
- &(rslSubstitution=(URLBASE “ftp://host:1234”))
-   (rslSubstitution=(URLDIR $(URLBASE)/dir))
-   (executable=$(URLDIR)/myfile)
- is equivalent to this
- &(executable=ftp://host:1234/dir/myfile)

## GRAM DefinedRSL Substitutions

- GRAM defines a set of RSL substitutions before processing the job request
- Machine Information
- GLOBUS_HOST_MANUFACTURER
- GLOBUS_HOST_CPUTYPE
- GLOBUS_HOST_OSNAME
- GLOBUS_HOST_OSVERSION
- Defined in $GLOBUS_LOCATION/etc/globus-job-manager.conf

## GRAM DefinedRSL Substitutions

- Paths to Globus
- GLOBUS_LOCATION
- Miscellaneous
- HOME
- LOGNAME
- GLOBUS_ID

## GRAM Examples

- The globusrun client is a more involved prototype that allows complicated RSL expressions.
- % globusrun –r pitcairn.mcs.anl.gov –f myjob.rsl
- % globusrun –r pitcairn.mcs.anl.gov \
-       ‘&(executable=myprog)’
- globus-job-run and globus-job-submit are wrappers around globusrun

## DUROC Review

- Simultaneous allocation of a resource set
- Handled via optimistic co-allocation based on free nodes or queue prediction
- In the future, advance reservations will also be supported
- globusrun will co-allocate specific multi-requests
- Uses a Globus component called the Dynamically Updated Request OnlineCo-allocator (DUROC)

## A Co-allocation Multirequest

- +( & (resourceManagerContact=server1.test.edu)
-         (count=1)
-         (label="subjob A")
-         (executable= my_app1)
-    )
-    ( & (resourceManagerContact= server2.example.com)
-         (count=2)
-         (label="subjob B")
-         (executable=my_app2)
-    )

## RSL Attributes For DUROC

- (label=string)
- Identifier for this subjob
- (resourceManagerContact=string)(resourceManagerName=string)
- Resource manager to which to submit a subjob

## GRAM configuration files

- $GLOBUS_LOCATION/etc
- globus-gatekeeper.conf
- Location of certificates, grid-mapfile, logfile, port number, etc.
- globus-job-manager.conf
- Specifies some RSL substitutions, some gatekeeper information
- grid-services/
- Contains configured jobmanager names/types

## Using MDS

- With many resources at our disposal, we need a way to get information about them
- This information is published using the Metacomputing Directory Service
- Currently implemented over openldap

## grid-info-search

- grid-info-search “LDAP filter”
- # filter: (objectclass=*)
- # requesting: ALL
- # ept.mcs.anl.gov, local, grid
- dn: Mds-Host-hn=ept.mcs.anl.gov,Mds-Vo-name=local,o=grid
- objectClass: MdsComputer
- objectClass: MdsComputerTotal
- objectClass: MdsCpu
- ...

## grid-info-search (cont.)

- # eth0, networks, ept.mcs.anl.gov, local, grid
- dn: Mds-Device-name=eth0, Mds-Device-Group-name=networks, Mds-Host-hn=ept.mcs.
-  anl.gov,Mds-Vo-name=local,o=grid
- objectClass: MdsDevice
- objectClass: MdsNet
- Mds-Device-name: eth0
- Mds-Net-name: eth0
- Mds-Net-netaddr: 140.221.8.0/22
- Mds-Net-addr: 140.221.10.71
- Mds-validfrom: 20020116040124Z
- Mds-validto: 20020116041624Z
- Mds-keepto: 20020116041624Z

## MDS Configuration files

- $GLOBUS_LOCATION/etc/grid-info-*
- grid-info.conf
- Sets default behavior of grid-info-search
- grid-info-resource-ldif.conf
- Describes the local information providers for the GRIS
- You can add your own providers:
- 	http://www.globus.org/gt2/mds2.1/creating_new_providers.pdf

## MDS Configuration files

- grid-info-resource-register.conf
- Lists the GIIS servers to which this GRIS will register directly. The default is to register to the local GIIS on the host.
- grid-info-site-policy.conf
- Controls acceptance of incoming registrations.  Defines a set of acceptable remote MDS services.  Defaults to only accepting information from the local machine

## MDS Configuration files

- grid-info-slapd.conf
- The slapd.conf for openldap.  Includes access control (such as whether anonymous binding is allowed) and identifies schema files
- If anonymous access is not allowed, you must be in the grid-mapfile to query
- grid-info-resource.schema
- The schema for GRIS information

## GridFTP clients

- globus-url-copy
- Included in the Globus API bundle
- gsi-wuftpd client: gsincftp
- Included in a contributed bundle, available from the source bundle download page

## globus-url-copy

- globus-url-copy URL1 URL1
- Can be file://, http://, gsiftp://, ftp://, or https://
- Can be used for third-party transfer
- I.e., Clients on host A can initiate a transfer between hosts B and C.
- You will usually need to edit the ftpaccess file of both sites to achieve this

## gsincftp

- gsincftp URL
- Can be ftp:// or gsiftp://
- Opens a GSI authenticated/encrypted connection with the remote host.
- Make sure your shell is in /etc/shells
- If the remote host is running gsi-wuftpd with GSSAPI turned on, cannot authenticate using login/password

## GridFTP Configuration Files

- $GLOBUS_LOCATION/etc
- ftpaccess
- Access control
- ftpconversions
- Downloading .tar.gz version of a file, when only the original file exists
- Same as ordinary wu-ftpd server

## Deployment scenarios

- Single host
- Single gatekeeper
- Many hosts, not a cluster (or no scheduler)
- Many gatekeepers
- Many hosts, one cluster
- One gatekeeper

## Many hosts, no scheduler

- Create a .tar.gz of a sample Globus 2.0 installation
- Keep copies of each globus-postinstall.sh
- Untar, re-run postinstall scripts
- This will customize it to the local hosts
- Request certificates, run setup-gsi, setup-globus-gram-jobmanager
- Done!

## Cluster deployment scenarios

- Only one host needs to run the gatekeeper
- Gatekeeper must be accessible from the outside world
- Submission tools to scheduler must be available
- E.g., PBS qsub command

## Dealing with Firewalls

- Connections from client to server
- Pick whatever port you’d like, make sure to use a contact string
- Connections back to submitting host
- GLOBUS_TCP_PORT_RANGE
- When this variable is set, globus_io will ensure that new ports opened will fall in the port range specified
- MPICH-G2 jobs
- Compute nodes need to communicate with jobmanager host

## Dealing with Firewalls (cont.)

- Two firewalls:
- 	Host1  |FW1| -------- |FW2| Host2
- Host1
- Set GLOBUS_TCP_PORT_RANGE to open ports of Firewall 1.  Any listening port opened will be in a valid range for Host2 to connect to.
- Host2
- Set GLOBUS_TCP_PORT_RANGE to open ports of Firewall 2, so Host1 may connect.
- Can set in global environment variables, as well as in gatekeeper environment

## Running your own CA

- Can use openssl included with Globus, or can use a commercial package
- First, create a self-signed certificate
- Trust has to start somewhere!
- Choose a namespace for signing
- Don’t use “/O=Grid/O=Globus”!
- Modify Globus to use your CA automatically

## Creating a self-signed certificate

- openssl req -days 365 -config openssl.cnf
- 	-x509 -newkey rsa:1024
- 	-keyout private/cakey.pem -out cacert.pem
- Use the openssl.cnf from $GLOBUS_LOCATION/ssl
- Edit it to change the [CA_DEFAULT ] dir to a directory of your choosing
- This is where the rest of the CA work will take place

## Choosing a namespace

- /O Organization
- /OU Organizational Unit
- No real RFC about how to do this, but it should be traceable back to your organization
- Be sure to check signing_policy files that people send you to make sure they are not overlapping someone else’s namespace

## CA Setup

- Use $GLOBUS_LOCATION/setup/globus/setup-gsi to set the namespace in /etc/grid-security/grid-security.conf
- Edit SETUP_GSI_CA_EMAIL_ADDR by hand
- Then run $GLOBUS_LOCATION/setup/globus/grid-cert-request-config
- This makes grid-cert-request use your CA’s namespace to generate requests.

## CA Setup (cont.)

- Copy the CA certificate into /etc/grid-security/certificates
- Use `openssl x509 –in cacert.pem –noout –hash`.0 as the filename
- Add a signing_policy
- Copy the default signing policy, change the access_id to your certificate subject, and the cond_subjects to your namespace
- Don’t edit the second line, pos_rights

## Signing Certificates

- You will receive email with a certificate request.  Output the whole email to a file, req.pem
- Double-check for accuracy
- Is it in your namespace?
- Do you trust this user?
- From your CA directory, run
- 	openssl ca -config openssl.cnf –in req.pem
- 	-out cert.pem
- Return the file cert.pem to the user

## Continuing Maintenance

- So far, we’ve covered how to start up a Globus installation.  What continuing tasks remain?
- grid-mapfile updates for new users
- Add trust for other CAs
- /etc/grid-security/certificates
- Renew gatekeeper, LDAP certificates
- grid-cert-renew
- Users will also renew their certs
- May add a new scheduler in the future

## Methods of Automation

- Centralize the grid-mapfile, find a method of distribution
- Centralize the list of trusted CAs (and signing policies)
- Possibly integrate certificate acquisition with account creation if you run your own CA
- Can add to grid-mapfile at the same time