**CDF Offline Initiative**

**Closure Report**

Version 1.0

25 November 2008

Robert D. Kennedy, *et al.*

**[Table of Contents]{.underline}**

**Initiative Overview [2](#initiative-overview)**

Introduction [2](#introduction)

CD Charge to the Initiative [2](#cd-charge-to-the-initiative)

Project Team [2](#project-team)

Project Repository [3](#project-repository)

Project History [3](#__RefHeading___Toc214098042)

Reasons for Closing the Project [5](#reasons-for-closing-the-project)

**State of the Initiative Objectives
[6](#state-of-the-initiative-objectives)**

1\. CAF Adaptation to FermiGrid [6](#caf-adaptation-to-fermigrid)

1a. (WBS 4.5) Deploy New FermiGrid CAF Head Nodes
[6](#a.-wbs-4.5-deploy-new-fermigrid-caf-head-nodes)

1b. (WBS 4.5) Migrate GroupCAF Nodes into FermiGrid CAF
[7](#b.-wbs-4.5-migrate-groupcaf-nodes-into-fermigrid-caf)

1c. (WBS 4.6) Adopt GlideinWMS [7](#c.-wbs-4.6-adopt-glideinwms)

2\. Critical Node Upgrades [8](#critical-node-upgrades)

3\. Issue Tracking System [8](#issue-tracking-system)

4\. Monitoring Framework [9](#monitoring-framework)

4a. (WBS 2.1) Low-Level Monitoring Framework (Zabbix)
[9](#a.-wbs-2.1-low-level-monitoring-framework-zabbix)

4b. (WBS 4.3) User Monitoring Framework for Grid Infrastructure
[10](#b.-wbs-4.3-user-monitoring-framework-for-grid-infrastructure)

5\. CDF Offline Architecture [11](#cdf-offline-architecture)

6\. CDF Offline Operations [11](#cdf-offline-operations)

**Lessons Learned [13](#lessons-learned)**

Successes [13](#successes)

Opportunities for Future Improvements
[13](#opportunities-for-future-improvements)

Personnel Issues [13](#personnel-issues)

Other Comments [14](#other-comments)

**Next Steps [14](#next-steps)**

**Summary [15](#summary)**

# Initiative Overview

## Introduction {#introduction .unnumbered}

The CDF Offline Initiative is an umbrella project to achieve a broad set
of goals by providing management consultation, adding effort to handle
transition demands, and applying a modest level of project management
formalism to track and sustain progress. The Initiative helps plan,
prioritize, and coordinate work amongst the existing personnel and
groups, with an eye towards identifying and reducing the long-term
maintenance and support requirements of RUN2 experiments.

## **CD Charge to the Initiative** {#cd-charge-to-the-initiative .unnumbered}

The charge to the Initiative from the Computing Division can be
summarized by "do what is important in six months to get CDF Offline
Computing up to production grade". Based on consultation with the major
stakeholders, existing work plans, and consideration of the CDF CAF Task
Force Interim Report (v4.02) recommendations and associated status
document (2/27/2008), a number of highest priority objectives were
identified to support this charge. The Initiative Objectives are:

1.  CAF Adaptation to FermiGrid (added March 2008)

2.  Critical Node Upgrades

3.  Issues Tracking System

4.  Monitoring Framework

5.  CDF Offline Architecture

6.  CDF Offline Operations (added April 2008)

The Initiative started by capturing all of the known upcoming work of
CDF Offline in a high-level work list that itself was a deliverable to
help guide long-term planning. Realistic estimates of task durations and
staff availability indicated that, as expected, not all of the
envisioned work could be fit into the Initiative. Tasks were either
accepted into the work scope of the Initiative ("scheduled" tasks) or
not ("unscheduled" tasks). Major high priority tasks were identified and
scheduled in finer detail.

## Project Team {#project-team .unnumbered}

Director: Margaret Votava (CD ILC/DAQ Deputy Dept Head, now REX Dept
Head)

Manager: Robert D. Kennedy (CD OPMQA)

The Advising Committee consisted of the Project Director, Project
Manager, and:

-   Jerry Guglielmo (CD ILC/DAQ Dept Head, now LSC Quad Assoc. Leader)

-   Rick Snider (CD REX Dept Head, now REX Deputy Dept Head; CDF Collab
    and co-leader of CDF Offline)

-   Donatella Lucchesi (INFN -- Padua, CDF Collab and co-leader of CDF
    Offline)

> Later joined by:

-   Dennis Box (on assignment to Initiative and REX Dept)

-   Joe Boyd (on assignment to Initiative, in REX Dept)

The CDF Offline Computing liaison role was performed largely by Rick
Snider and Donatella Lucchesi as co-leaders of CDF Offline Computing
during this Initiative.

## Project Repository {#project-repository .unnumbered}

Project Management and some subject matter documentation are maintained
in the Google Group [\"CDF Offline
Initiative\"](http://groups.google.com/group/cdf-offline-initiative?hl=en).
This is organized into several distinct areas, the most useful of which
may be the
[\"Pages\"](http://groups.google.com/group/cdf-offline-initiative/web?hl=en).
Some examples include:

-   [Initiative
    Schedules](http://groups.google.com/group/cdf-offline-initiative/web/cdf-offline-initiative-schedules-2?hl=en):
    MS Project plans and related reports tracking the Initiative.

-   [Work
    Lists](http://groups.google.com/group/cdf-offline-initiative/web/wbs-2?hl=en):
    All tasks in CDF Offline that might be done. Led to the first WBS.

-   [JIRA
    Configuration](http://groups.google.com/group/cdf-offline-initiative/web/jira?hl=en):
    Notes on configuring JIRA for CDF Offline.

The [Fermilab JIRA Issue Tracker](https://fermilab.onjira.com/) contains
operations and development issue tracking information dating back to its
deployment in April (beta)/May (production) 2008.

We chose to use standing CDF Offline meetings as-is to track progress in
the Initiative since one of the initial concerns was the high workload
of the CDF people involved in supporting Offline. The CDF meetings used,
the CDF Offline Operations and the CDF Offline Development meetings, are
[documented in
WebTalks](http://www-cdf.fnal.gov/cgi-bin/WebTalks/searchArchives.pl?done=done&meeting=offline&person=&title=&match=&Action=Search+Archives).

[]{#__RefHeading___Toc214098042 .anchor}***Project History***

### Offline Issues Evaluation: January 2008 -- February 2008 {#offline-issues-evaluation-january-2008-february-2008 .unnumbered}

-   January 2008: Jerry Guglielmo, Margaret Votava interview CDF Offline
    team.

-   February 2008: Rob Kennedy joins Initiative team, drafts CDF Offline
    work lists.

-   February 2008: Initiative project concept developed: 6 month
    Umbrella Project

-   February 2008: CDF CAF Task Force Interim Report with status
    annotations

### Initiative Planning, Product Evaluation: February 2008 -- May 2008 {#initiative-planning-product-evaluation-february-2008-may-2008 .unnumbered}

-   February -- May 2008: Furloughs and "forced vacations" complicate
    staffing.

-   February 2008: Jira and Zabbix product evaluations

-   March 10, 2008: First full "CDF Offline Work List" v0.7 released for
    comment.

-   March 19, 2008: Executive presentation of Initiative plan to CD
    leadership.

-   April 9, 2008: Initiative Introduction presented at CDF Offline
    Operations Mtg.

-   April 10, 2008: First resource-loaded schedule draft v0.5.0, with
    Work List v1.0 embedded. Scheduled tasks are in-scope, unscheduled
    tasks are out. Execution begins.

-   April 18, 2008: Joint CDF-CD Executive Mtg with overview of
    Initiative.

-   April 28, 2008: Baseline Planning Meeting (using plan v0.8.3)

-   May 10, 2008: Baseline Project Plan v1.0.0 project released.

### Execution: April 2008 -- October 2008 {#execution-april-2008-october-2008 .unnumbered}

-   April 2008: Jira evaluation and initial integration into Offline
    support processes.

-   April 2008: Offline Project draft re-org released. Some future roles
    unclear.

-   May 2008: Shadow CAF setup. High Priority KCA Upgrade work all
    month. Staff issues and repeated little problems in head node task
    chain.

-   Mid-June 2008: FermiGrid Head Node progress deemed unacceptable.
    Task force formed to re-organize the work, one group to create a
    tagged "CAF" software release and the other to undertake FermiGrid
    CAF scale tests. CDF effort redirected from monitoring objectives to
    the task force work. Production and Ntuple coordinators begin basic
    operations tests on FermiGrid CAF.

-   July 2008: Offline Project re-organization fully implemented.

-   July 2008: Jerry Guglielmo leaves Initiative. Low-Level Monitoring
    work ends.

-   Early August 2008: Dennis Box and Joe Boyd join Initiative.

-   September 2008: CafCondor Config v2.0: First formal tagged release,
    able to be wiped and re-installed with high reproducibility. Final
    testing by production users.

-   October 2008: New CAF head node with tagged release in production.
    Begin shifting nodes and production users to upgraded FermiGrid CAF.
    (a.k.a. CdfGrid)

### Close-Out: October 2008 -- November 2008 {#close-out-october-2008-november-2008 .unnumbered}

-   October 15, 2008: Enough objectives judged by Initiative Advising
    Committee to be accomplished or on a smooth path to completion.
    Close-out is a transition of responsibility to new Line Management
    and existing Offline Management with the opportunity to take stock
    of the Initiative experience.

-   October 31, 2008: Executive Meeting of Initiative Advisory Committee
    with CD and CDF Heads on Close-Out. Well received overall.

-   November 7, 2008: Drop-dead date for task completion to be
    documented by Initiative.

> Status at Completion:

-   19 of 32 Major Milestones completed or will be completed by
    drop-dead date.

-   8 of 32 Major Milestones are in progress, but unlikely to finish by
    drop-dead date.

    -   Miles 10,17: GlideinWMS integration and deployment

    -   Mile 22: GroupCAF to FermiGrid CAF Migration

    -   Miles 3, 7, 15: CDF Offline Architecture documents

    -   Miles 23, 24 (SL4 Migration): Reduced in priority, after FNAL
        support for SL3 was extended, removing migration urgency. Work
        is continuing.

-   5 of 32 Major Milestones are "Closed, Incomplete":

    -   Miles 5, 20 (Low-Level Monitoring -- Zabbix): Closed to free up
        resources for higher priority work. May have saved effort
        globally to do so since this overlapped with work that was done
        on a later time-scale by FEF.

    -   Miles 9, 21 (User Monitoring Framework): Replanned to free up
        resources for higher priority work. Alternative proof of concept
        delivered Nov 2008.

    -   Mile 19: Multiple schedd hosts for FermiGrid CAF: Slipped to
        mid-2009 until work in progress on FermiGrid CAF is completed.
        The need for this work at that time should be re-evaluated
        before it is undertaken though.

## Reasons for Closing the Project {#reasons-for-closing-the-project .unnumbered}

The CDF Offline Initiative was defined to be a time-limited project to
achieve what was possible in six months, effectively beginning in April
2007. In mid-October, it became apparent that three of the four high
priority initiative objectives were accomplished or on a low-risk path
to completion in the near future, though the fourth (user monitoring)
was not completed as originally intended. Since the active Initiative
leader became the REX department head, it became apparent that the
Initiative could transfer its remaining charge smoothly to the existing
line management and experiment project management structures for
completion of the remaining work. On 31 October 2008, we presented the
Initiative accomplishments and a close-out plan to the CDF spokespeople
and Computing Division Head at an executive meeting. The Initiative will
just track the "GroupCAF to FermiGrid CAF Migration" and "GlideinWMS
Migration" task chains to completion, which are expected to completed by
mid-December 2008.

# State of the Initiative Objectives {#state-of-the-initiative-objectives .unnumbered}

## 1. CAF Adaptation to FermiGrid {#caf-adaptation-to-fermigrid .unnumbered}

This objective consisted of three main components necessary to achieve
the envisioned future CAF system based on FermiGrid technology and
having sufficient capacity to absorb existing CAFs on-site and meet
long-term production and analysis demands.

### 1a. (WBS 4.5) Deploy New FermiGrid CAF Head Nodes {#a.-wbs-4.5-deploy-new-fermigrid-caf-head-nodes .unnumbered}

Goals:

-   Deploy new, more performant hardware in the critical head node role
    to replace aging hardware.

-   Create and use, for the first time in a while, stable tagged
    releases of CAF service software capable of reliable, reproducible
    installation.

-   Demonstrate the new FermiGrid CAF system can manage the anticipated
    demand in the immediate future of 5k WN slots.

-   Demonstrate the new FermiGrid CAF system can be used reliably by
    current production users of GroupCAF (Production, Ntupling,
    Calibrations, etc).

State at Closing: DONE

Deliverables:

-   CafCondorConfig v2.0 released and v2.1 is about to be released at
    this writing.

-   Scale tests successfully completed on the ShadowCAF, albeit at a
    reduced 3k to 4k WN slot level emulation.

-   Fcdfhead10 in production as head node of new CdfGrid CAF instance

Outstanding Risks (including Operations, Support):

-   The node head11 failed to operate reliably after several repair
    attempts, a potentially serious blow to the original 2-head node
    configuration. It was determined, however, that all services could
    be run on head10 and doing so would in fact reduce the number of
    SPOF by one node. The integration system head nodes are prepared to
    be used as a replacement if head10 should fail badly too.

-   If head11 is repaired, since it is no longer needed in its
    originally envisioned role, then it might be deployed to further
    reduce operational risk by acting as a back-up host to non-singleton
    services in the FermiGrid/GlideinWMS CAF system.

-   The ShadowCAF (a.k.a. sleeper pools) approach to testing the new CAF
    system at scale failed to operate at sufficiently large scale due to
    a Condor bug, reportedly fixed in Condor 7.1.4 (a development
    version made available in late November, after the Initiative
    closure). The pace of the WN migration will be sufficiently slow to
    permit the greater vigilance required in the transition to
    full-scale production to catch and resolve any unexpected at-scale
    limitation. After this bug is fixed, the ShadowCAF scale-testing
    will be revisited to prepare for GlideinWMS testing (see 1c, below).

### 1b. (WBS 4.5) Migrate GroupCAF Nodes into FermiGrid CAF {#b.-wbs-4.5-migrate-groupcaf-nodes-into-fermigrid-caf .unnumbered}

Goals:

-   Migrate all worker nodes (WNs) and users from the GroupCAF to the
    new FermiGrid CAF in order to support a CAF system built from fewer
    experiment-specific components.

State at Closing: IN PROGRESS, MIDWAY TO COMPLETION (early Nov 2008)

We expect smooth execution of this task chain from November to
mid-December 2008. The plan is well-documented and considered to have
low technical risk.

Deliverables:

-   [WN migration plan for CdfGrid, completion by December 08,
    2008](http://www-cdf.fnal.gov/internal/WebTalks/Archive/0811/081105_offline_operations/02_081105_offline_operations_Stephan_1_lammel_talk.pdf)
    (see p.3)

Outstanding Risks (including Operations, Support):

-   While this task is open, the CAF operations team will have to
    support both the old and the new CAFs. This is likely to increase
    stress on the team for a time with more distinct services having to
    be supported.

### 1c. (WBS 4.6) Adopt GlideinWMS {#c.-wbs-4.6-adopt-glideinwms .unnumbered}

Goals:

-   Adapt to using the GlideinWMS model for job workflow management.

-   Demonstrate the new FermiGrid CAF system using GlideinWMS can manage
    the anticipated demand in the future of 10k WN slots.

-   Migrate all WNs and users from the non-GlideinWMS FermiGrid CAF to
    the GlideinWMS FermiGrid CAF in order to support greater demand in
    the future.

State at Closing: IN PROGRESS, AT EARLY STAGE (early Nov 2008)

Recovering the past proof-of-concept implementation, planning
integration of change required to the code base with modern tagged
releases, and preparing an integration testbed.

Deliverables:

-   CdfGrid migrated to use GlideinWMS

-   GlideinWMS monitoring available to operations and users.

Outstanding Risks (including Operations, Support):

-   GlideinWMS adoption carries with it some risks:

    -   Does not operational simplify the system

        -   Increase from 2 to 3 condor pools

        -   Increase number of production head nodes from 1 to 3

    -   First time installs need an expert now. Knowledge transfer in
        progress.

    -   May miss goal of being in production for winter conference use
        since starting behind: the head10 replacement took much longer
        than planned

-   While this task is open, the CAF operations team will have to
    support both the old and the new CAFs. This is likely to increase
    stress on the team for a time with more distinct services having to
    be supported.

-   Long-term support for the GlideinWMS component will be addressed in
    a briefing being organized by Eileen Berman.

## 2. Critical Node Upgrades {#critical-node-upgrades .unnumbered}

Goals:

-   Replace unreliable out-of-warranty servers with new more-performant
    servers, thus improving reliability and reducing operations effort.

Areas of Work:

> 2a. (WBS 6.1) Critical Node Upgrades: ICAF Nodes
>
> 2b. (WBS 7.5) Critical Node Upgrades: dCache File Servers
>
> 2c. (WBS 3.4) Critical Node Upgrades: Code Server

State at Closing: DONE.

Deliverables:

-   The new nodes are in service, and the replaced nodes out of service.

Outstanding Risks (including Operations, Support):

-   The CDF Offline Infrastructure group should have a plan to upgrade
    the few remaining nodes on the original [posted
    list](http://groups.google.com/group/cdf-offline-initiative/web/cdf-nodes-that-need-retiring?hl=en).
    The FY09/FY10 procurement plans have not been checked for this by
    the Initiative.

## 3. Issue Tracking System {#issue-tracking-system .unnumbered}

Goals:

-   Move issue reporting from an ad hoc set of e-mail lists to an
    organized issue tracking system integrated into existing CDF Offline
    support systems.

-   The issue tracking system should enable the following capabilities
    \[LINK: Margaret's presentation June 19, 2008 CDF Week\]:

    -   Gain metrics to understand how to best utilize limited
        development time

    -   Analyze trends

    -   Load balance open issues across available staff

    -   Set priorities of open issues

    -   Support a well-defined workflow for issue resolution

-   Deploy across CDF Offline. Insure it is used exclusively in place of
    old system.

Areas of Work:

> 3a. (WBS 2.2) Issue Tracking System (JIRA)

State at Closing: DONE.

Deliverables:

-   [Production JIRA system hosted by
    Atlassian](http://fermilab.onjira.com/)

-   CDF Offline is represented by a JIRA Category, and broken down into
    its historical sub-projects with each sub-project mapping to a
    single JIRA Project. There are ten Projects at this writing, ranging
    from operations-oriented to development-oriented, such as:

    -   [CDF CAFS](https://fermilab.onjira.com/browse/CDFCAFS)

    -   [CDF Code
        Management](https://fermilab.onjira.com/browse/CDFCODE)

    -   [CDF Data Handling](https://fermilab.onjira.com/browse/CDFDH)

    -   [CDF Offline
        Development](https://fermilab.onjira.com/browse/CDFOFFDEV)

Outstanding Risks (including Operations, Support):

-   The Atlassian external hosting solution has its positives as well as
    its negatives:

    -   Positive: External hosting allowed rapid, inexpensive
        large-scale deployment that enabled the Initiative to see JIRA
        in use by CDF Offline for months instead of taking months in
        procurements and deployments.

    -   Positive: The included long-term support has freed up effort in
        our organization to address domain-specific issues like the CAF
        Migration, since we avoided having to add staff to support JIRA.

    -   Negative: A recent Jira "email interface feature" downtime
        revealed that the host's pre-deployment testing may be less than
        complete.

    -   Negative: The service contract limits the number of users
        permitted to a rather coarse-grained scale, requiring
        potentially a large expense to increase user limits at some
        point in time.

    -   Neutral: Two features become complicated by external hosting and
        were not achieved: an e-mail plug-in feature and LDAP strong
        authentication integration. The latter would have been delayed
        in any case due to CD Central Services schedules. Nevertheless,
        the JIRA system has been very successful without these features,
        so they are not critical requirements.

-   As with any external hosting solution, we have to plan how to get a
    copy of our data if the hosting business were to close. Since the
    JIRA software itself is freely available, we should technically be
    able to get a snapshot of the underlying database on short notice
    and be able to reload it into a local database with a local JIRA
    service running against it.

## 4. Monitoring Framework {#monitoring-framework .unnumbered}

This objective consisted of two distinct and independent lines of work:
low-level monitoring in the role of NGOP, and the much higher-level user
monitoring intended to complement the job management systems in OSG and
LCG.

### 4a. (WBS 2.1) Low-Level Monitoring Framework (Zabbix) {#a.-wbs-2.1-low-level-monitoring-framework-zabbix .unnumbered}

Goals:

-   The low-level monitoring framework is to play a similar role as
    NGOP, but

    -   Be scalable to 10k's of monitored components

    -   Lack some limitations of NGOP revealed in the past few years,
        such as an isolated change requires a time-consuming restart of
        the entire service

State at Closing: CLOSED, INCOMPLETE AS ORIGINALLY DEFINED.

Explanation:

Work was begun on this task chain, and a CDF-independent proof of
concept developed. Resources were removed from this task chain and
re-assigned to the higher-priority Head Node Upgrade task chain. After a
short time, it became clear this had to be a permanent re-assignment and
this task chain was closed.

Outstanding Risks (including Operations, Support):

-   Lacking a low-level monitoring service that can scale to the number
    of monitoring points anticipated in the future CDF CAF system will
    make the system unmanageable as hardware begins to age and fail at a
    non-trivial rate.

-   Much of the planned infrastructure was developed a little later by
    the FEF Department anyway. Thus, redirecting effort away from this
    objective may actually have avoided some redundancy in the long run.

### 4b. (WBS 4.3) User Monitoring Framework for Grid Infrastructure {#b.-wbs-4.3-user-monitoring-framework-for-grid-infrastructure .unnumbered}

Goals:

-   Create a database-backed monitoring framework for user jobs that can
    be used with OSG Condor-based systems as well as LCG systems.

State at Closing: CLOSED, INCOMPLETE AS ORIGINALLY DEFINED.

Deliverables:

-   [March 5, 2008 CAF Monitoring Presentation by Hans
    Wenzel](http://home.fnal.gov/~wenzel/cdf/cdf.pdf)

-   [October 2008 Draft of CAF Monitoring CDF
    Note](http://home.fnal.gov/~wenzel/cdf/cdf/Monitoring_cdfnote.ps)

-   [November 4, 2008 CAF Monitoring Presentation by Hans
    Wenzel](http://www-cdf.fnal.gov/internal/WebTalks/Archive/0811/081104_offline_development/07___other_1_3_development2.pdf)

Explanation:

Work was begun on this task chain, and two main underlying components
(Postgres, Zabbix) were deployed on a development machine. The
installation and testing of Quill++ (RDB-backed Condor status
information system) stalled however due to a CDF-specific authentication
problem. Then, resources were removed from this task chain and
re-assigned to the higher-priority Head Node Upgrade task chain. After a
short time, when it became clear this re-assignment was to be long-term,
a new less ambitious plan was drawn up to demonstrate the concepts of
the framework using other customer systems already deployed. This too
ran into problems, and alternative components were evaluated. A working
model was demonstrated in the last week of the Initiative.

Outstanding Risks (including Operations, Support):

-   Lacking an RDB-backed user monitoring framework means CDF will
    continue to use its "logfile scraping"-like approach.

-   Lacking crisp requirements from CDF for the user monitoring, it is
    unclear whether the delivered proof-of-concept required a small or
    large amount of effort to become a production-quality solution.

-   The time required to investigate problems will continue to impact
    operations personnel while the low-level and user-level monitoring
    and associated tools are developed and deployed into production.

## 5. CDF Offline Architecture {#cdf-offline-architecture .unnumbered}

Goals:

-   Document the high-level architecture of the CDF Offline system in
    order to clarify roles and responsibilities among the services and
    tools.

-   Document the strategies applied to the major areas of the CDF
    Offline system.

-   Document the organization structure and responsibilities in CDF
    Offline.

Areas of Work:

> 5a. (WBS 1.1) Strategy Sheets
>
> 5b. (WBS 1.2) Offline Services Design Document
>
> 5c. (WBS 1.3) Project Organization Chart

State at Closing: IN PROGRESS (early Nov 2008)

Deliverables:

-   [Grid and CAF-Grid Instances strategy
    sheets](https://cd-docdb.fnal.gov:440/cgi-bin/ShowDocument?docid=2688)

-   Strategy sheets for Disk Space, Offline Infrastructure, and Data
    Handling are in the plan but not yet delivered.

-   Initial draft of services system diagrams posted, to be done after
    CAF migration.

-   Project organization chart done, though lacks list of
    responsibilities per box/role

Outstanding Risks (including Operations, Support):

-   With short-term goals capturing most of the attention in the
    Initiative, addressing these documents has been a challenge since
    they are seen as mostly delivering value in the long-term.

-   Strategy sheets: risks of not completing or using for CDF Offline
    planning

-   System Diagrams: risks of not completing

-   Project organization chart: risks of unclear responsibilities, same
    name in many boxes.

## 6. CDF Offline Operations {#cdf-offline-operations .unnumbered}

Goals:

-   Support the CDF Offline Project Leaders in the management of
    operations.

-   Help in the planning and tracking small projects necessary to
    maintain operations.

The following are some examples of the topics covered under this
objective.

Topic 6a. (WBS 3.5) SL4 Migration

-   Goal: Upgrade CDF Offline to use SL4

-   State at Closing: SLOWED DUE TO LOWER URGENCY (early Nov 2008)

-   Description: This task chain was executed by the CDF Offline
    Infrastructure group. After support in offline development for SL4
    was achieved, the task chain dropped in priority considerably as
    FNAL CD extended SL3 support through mid-2010. CDF Offline chooses
    to continue slowly ahead with SL4 deployment and then force
    migration only when mid-2010 approaches or when the effort to
    support SL3 and SL4 together becomes too much, whichever comes
    first.

-   Risks: What metric triggers this decision? How is SL5 transition
    impacted?

Topic 6b. (WBS 4.4) FNAL KCA Upgrade

-   Goal: Upgrade CDF Services globally to use the upgraded FNAL KCA

-   State at Closing: DONE

-   Description: This task chain was led by Donatella Lucchesi and
    executed within the CDF Offline Project context. The Initiative
    consulted during the planning stage and helped track the effort as
    the major challenges were uncovered.

-   Risks: None identified.

Topic 6c. (WBS 5.7) Operations Shifts, Incident Management, and Effort
Reduction

-   Goal: Reduce operations load, reduce stress of support personnel,
    clear process and responsibilities, no user requests "lost"

-   State at Closing: IMPROVED

-   Description: Operations internal meeting, JIRA, workflow, shift
    rotation, etc.

-   Risks:

    -   The estimated reduction in staff effort to operation CDF CAF
        systems from 4.0 to 3.5 FTEs is less than expected, though the
        systems are still in transition to the long-term solution. The
        available tools can be used to study the issues leading to the
        greatest operations load and prioritize their resolution.

    -   Lacking monitoring tools will lead to greater operations work
        per incident resolution and less pro-active work overall than
        would be possible.

Topic 6d. Management Consulting

-   Goal: Advise on improving effectiveness of Offline Project where
    process and personnel were involved.

-   State at Closing: IMPROVED

-   Description:

    -   Process introduced for development to integration to production
        in "development"

    -   Process for workflow, ticket prioritization, etc. in operations

    -   Brought in a more experienced developer and a more experienced
        operator to provide leadership and guide-by-example within
        Offline.

    -   Helped improve match of personnel to roles in re-organized
        Offline Project.

-   Risks: Need to plan for transition of Initiative added staff out of
    the Offline project at some point in the future. For example: Dennis
    Box.

# Lessons Learned {#lessons-learned .unnumbered}

## Successes {#successes .unnumbered}

-   Issue Tracking with Jira was a big step forward. It was leveraged
    well by being delivered early in the Initiative to have the most
    impact on operations.

-   Restored the use of tagged releases in the CDF CAF system.

-   The CAF head nodes were replaced with a new node, running a tagged
    release.

-   The critical node upgrades did happen, after a long hiatus. Fewer
    critical node deaths containing analysis data led to fewer
    unexpected fires for operations to treat and reduced threat of lost
    productivity for scientists.

-   Strategy documents are being written, with first priority given to
    documents guiding work in progress.

-   Restored the devàintàprd development discipline to the CDF CAF
    system.

-   Payoffs

    -   No more fear of reinstalls (many, many have happened over the
        last several weeks)

    -   No more fear of shutdowns (systems brought back same day as
        outage)

    -   Operations personnel are less stressed (but not stress free
        yet!)

    -   Better user job efficiency -- some long standing, nagging
        problems fixed

    -   CAF effort is functional and support people are productive.

    -   Customers have lower anxiety level about issue resolution

## Opportunities for Future Improvements {#opportunities-for-future-improvements .unnumbered}

-   Once a re-organization is drafted and announced, it needs to be
    finalized quickly to avoid staff becoming confused about
    responsibilities, having to make choices between current and
    uncertain future responsibilities. This led to reduced productivity
    and reduced ownership of objectives by personnel. In some cases,
    people were disappointed when their roles were clarified to be
    something other than what they had first heard or had imagined due
    to the lack of information.

-   If a person with many roles and responsibilities leaves the project
    or organization, then each role should be clearly defined as well as
    to whom each role is transferred with acknowledgement from the
    receiving person.

-   When one or more people are in many boxes of an organizational
    chart:

    -   There need to be clear responsibilities for each group to link
        to priorities.

    -   Managers need to be clear about task priorities to avoid
        confusion.

-   More status requests by managers or progress reports from the
    developer might have reduced the surprise level when a User
    Monitoring prototype was delivered.

-   The User Monitoring project should have first generated a
    requirements document with experiment sign-off. The deliverables
    could be judged against a static reference point rather than against
    possibly shifting priorities of the day.

## Personnel Issues {#personnel-issues .unnumbered}

-   Personality clashes and/or strained personnel relations have a much
    larger effect on a project than is visible to the people immediately
    involved. One overly harsh e-mail can generate hours of
    fence-mending by project managers, line managers, and even the
    experiment spokespeople/division head (much of which happens behind
    the scenes) as well as the people involved.

-   There were instances of clearly reduced productivity as some
    personnel refused to talk to others on certain topics for whatever
    reason in a knowledge transfer task.

-   Repeatedly, people interpreted e-mail in the worst possible light
    and did not take the time to ask for clarification before rallying
    allies and responding. At times, even relatively innocent e-mail set
    off unproductive clashes where no undue criticism was intended,
    sapping effort and breaking down team productivity.

-   It took a long time to figure out the right combinations of people
    that could work together effectively. How fragile is this?

## Other Comments {#other-comments .unnumbered}

-   Furloughs had an especially harsh impact on operations personnel as
    rotations were hard to schedule/balance/maintain and there were
    nearly always fewer people than needed (hired for) who were
    available for peak periods\... which then impacted their development
    fraction as well.

-   Architectural Diagrams: not immediately valuable to operations staff
    intimately familiar with system, but helps managers and other groups
    re-acquaint themselves with the service design quickly for more
    meaningful discussion.

# Next Steps {#next-steps .unnumbered}

While the Lessons Learned section covers a broad set of topics,
long-term and short-term, the next steps that we believe may have the
greatest impact are:

-   Complete the WN migration to CdfGrid, now in progress.

-   Complete GlideinWMS adoption in progress, and clarify its support.

-   Address the User Monitoring objective and assemble a consistent
    monitoring suite exploiting the GlideinWMS monitoring as well.

-   Reduce operations load with more automation and "Low-Level
    Monitoring" via Zabbix as now supported by the FEF department.

-   Leverage JIRA as much as possible to prioritize which areas of
    operations to automate and/or monitor first.

# Summary {#summary .unnumbered}

Despite being pushed significantly behind schedule by technical and
personnel issues, the CDF Offline Initiative accomplished its higher
priority goals and laid a foundation for future work. The JIRA issue
tracking system has been deployed and adopted by CDF collaborators for
CDF Offline operations, replacing the use of e-mail lists for incident
management. This transition also led to [online ticket
metrics](https://fermilab.onjira.com/browse/CDFOFFDEV?report=com.atlassian.jira.plugin.system.project:roadmap-panel)
tools, [a CAF feature release
roadmap](https://fermilab.onjira.com/browse/CDFOFFDEV?report=com.atlassian.jira.plugin.system.project:roadmap-panel)
support for managed development, as well as reduced stress for
operations personnel and reduced anxiety for users concerned about their
issues being reporting to unattended e-mail lists. The CDF CAF portion
of the Offline project has been re-organized to use a mature development
cycle process and produce tagged releases in CVS that can be reliably
and quickly installed. This has led to rapid and managed progress
towards a production quality GlideinWMS-based CAF system to be deployed
in the near future, in parallel with GroupCAF worker nodes undergoing
overdue OS upgrades and being migrated to the new CdfGrid CAF. Critical
servers have been upgraded with new hardware, helping to insure more
stable Offline operations in the near future. While the overall
operations effort has not yet been reduced significantly, we are
confident it will begin to decrease more noticeably after the planned
CAF transitions are completed. The monitoring-related objectives were
not achieved due in part to effort starvation and in part due to unclear
requirements. These objectives should be pursued once effort is
available on concentrate on this area of work. While not all that was
originally envisioned was accomplished, CDF Offline has progressed
significantly towards its goal of more stable and less effort-intensive
operations as RUN2 winds down.