United States Department of Agriculture (USDA) eGovernment Program

***Support for Development of***

***USDA Electronic Government Prototype***

***Operations Plan***

**Version 3.0**

**June 3, 2003**

![](media/image1.jpeg){width="1.75in" height="1.2666666666666666in"}

Content

[1. Introduction 3](#introduction)

[1.1. Overview of document sections 3](#overview-of-document-sections)

[1.2. Prototype Background 3](#prototype-background)

[2. Physical Architecture Overview 4](#physical-architecture-overview)

[2.1. Development Architecture 4](#development-architecture)

[**2.1.1.** **Environment** 4](#environment)

[**2.1.2.** **Hardware and Software** 5](#hardware-and-software)

[**2.1.3.** **Technical Architecture diagram**
5](#technical-architecture-diagram)

[**2.1.4.** **Backup** 5](#backup)

[**2.1.5.** **Security Procedures** 5](#security-procedures)

[**2.1.6.** **Roles and Responsibilities**
6](#roles-and-responsibilities)

[2.2. Production Architecture 6](#production-architecture)

[**2.2.1.** **Environment** 6](#environment-1)

[**2.2.2.** **Hardware and Software** 6](#hardware-and-software-1)

[**2.2.3.** **Technical Architecture diagram**
7](#technical-architecture-diagram-1)

[**2.2.4.** **Backup** 7](#backup-1)

[**2.2.5.** **Disaster Recovery** 8](#disaster-recovery)

[**2.2.6.** **Failover** 8](#failover)

[**2.2.7.** **Load Balancing** 8](#load-balancing)

[**2.2.8.** **Security Plan** 8](#security-plan)

[**2.2.9.** **Roles and Responsibilities**
9](#roles-and-responsibilities-1)

[3. Folder Structure 9](#folder-structure)

[4. Filename standard 11](#filename-standard)

[5. Unit Test 11](#unit-test)

[6. Adding Business Scenarios 12](#adding-business-scenarios)

[7. Process to modify existing scenarios
13](#process-to-modify-existing-scenarios)

[8. Appendix A -- Contact List 15](#appendix-a-contact-list)

[9. Appendix B -- NITC Service Description
16](#appendix-b-nitc-service-description)

#  {#section .unnumbered}

# Introduction

## Overview of document sections

The purpose of this document is to provide instructions to permanent
owners of the prototype on how to maintain the content and code. This
document contains the specifications of the development and production
environments, roles and responsibilities for business owners and IT
hosts, and instructions to create or augment additional scenarios. This
document is intended to provide the overall process documentation to
assist a system administrator on how to perform the various tasks
associated with operations and maintenance of the prototype.

## Prototype Background

The prototype will visually illustrate how eGovernment can be
implemented at USDA to change the way USDA do business. The main goals
are to:

-   Showcase the vision set forth in the eGovernment strategic plan and
    its respective initiatives;

-   Articulate how net-centric technology will be used to enhance
    program and service delivery and enhance internal operations;

-   Foster discussion about changing business processes and the positive
    impacts of those changes;

-   Help program managers understand how technology can enhance their
    business;

The prototype serves as a marketing and communication tool. Employees
see the Department making progress and begin to develop a new mindset
about the current operational paradigm. A consistent message can be
delivered to business partners about how USDA plans to change. Executive
leadership can use as a consistent message to constituents about the
strategic direction of the Department.

# Physical Architecture Overview

The technical architecture represents essentially pieces of the system
such as the Web Server, Application Server and Database. The logical
architecture also communicates how these technology fragments relate to
one another. The physical dimensions of the architecture such as
quantity, hardware, software and network are not represented in this
section, but are covered in various sections of the operations plan
document. The physical architecture provides a representation of those
elements while the logical architecture focuses on the makeup of the
components and how they interact with one another.

The Prototype Architecture layout is as follows:

![](media/image2.png){width="0.5in"
height="0.5007874015748032in"}![](media/image3.jpeg){width="0.7811023622047244in"
height="0.47834645669291337in"}![](media/image3.jpeg){width="0.7818897637795276in"
height="0.47834645669291337in"}

The Technical Architecture for the USDA eGovernment Prototype requires a
server consisting of a web server and database server. The IIS will act
as a web server and the SQL Server will be the database system. The
users will have access to the Prototype server through the USDA Intranet
only.

## Development Architecture

### **Environment**

The development environment for the prototype is hosted locally in the
same location as the eGovernment team. It is currently in Washington DC
USDA building located in eDeployment team office in NAL library.

The purpose of this environment is to provide an area where developers
can develop and unofficially test the application. In this environment,
the developers will develop and test the individual components such as
HTML, Active Server Pages and etc. The development environment is
accessible only by the USDA Intranet only. No special bandwidth
considerations are needed and the development environment is accessed by
only the USDA eGovernment team within USDA Intranet. To access prototype
related files including source codes, the prototype team must map the
network drive as: \\\\199.129.42.209\\wwwroot\\USDA_Prototype\\

### **Hardware and Software**

The development environment consists of a server that has the following
specifications:

  ---------------- -------------------------------------------------------
  **DASD:**        18 GB hard drives

  **Memory**       256 MB of SDRAM

  **CPU**          Quantity of 1, Pentium III, 1.0 Ghz
  ---------------- -------------------------------------------------------

The development environment is installed with Windows 2000 Server as
operating system.

The development server has 3 functions. It will act as web server,
database server and printer server for eGovernment. The IIS 5.0 is
installed for the web server and SQL Server 7 for the database server.
There is no need for any licensing agreement for these software since
the USDA already has the enterprise licensing agreement. Also, the
development server is located locally, remote management software such
as VPN or PC Anywhere will not be required.

### **Technical Architecture diagram**

![](media/image2.png){width="0.5in"
height="0.5in"}![](media/image4.wmf){width="0.25275590551181104in"
height="0.6456692913385826in"}![](media/image5.wmf){width="0.25275590551181104in"
height="0.6456692913385826in"}![](media/image3.jpeg){width="0.7811023622047244in"
height="0.47834645669291337in"}![](media/image3.jpeg){width="0.7818897637795276in"
height="0.47834645669291337in"}

The development environment is access by USDA Intranet only. The static
IP address for the prototype is 199.129.42.209. The development
environment is configured to act as web server using IIS, database
server installed with SQL Server 7 and printer server for eGovernment
team.

### **Backup**

All source code will be developed within the development environment,
but there's no scheduled backup process. It is developers'
responsibility to copy developed source code from the development
environment to another location for backup purpose.

### **Security Procedures**

The only security measure done on development environment is to restrict
permission to the root folder. Only people who can access the prototype
directory within the development environment are the prototype team
members. The root folder of the prototype is
[\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\](../../../../../../../../wwwroot/USDA_Prototype/).

### **Roles and Responsibilities**

The prototype team is responsible for the development environment.

The responsibilities of the prototype team includes, but not limited to:

-   Maintenance and services for the project;

-   Support for enhancements, optimization, installations, and upgrades;
    and

-   Backups.

## Production Architecture

### **Environment**

The production environment for the prototype is hosted by NITC. The
primary NITC hosting facility is located in Kansas City, MO. NITC is a
fee for service organizations that caters to both USDA and external
clients. The facility provides some core services such as storage,
physical security (biometrics), redundant power supply, backup power
generators and network connectivity. NITC is currently install and
support over 500 COTS (commercial off-the-shelf) products and manage
approximately 14.1 terabytes of DASD collectively at its centers.

**(Steps to connect to NITC POC Server using Terminal Service)**

### **Hardware and Software**

The hardware and software will be provided by NITC. The production
environment will consist of a server that has the following
specifications:

+---------------+------------------------------------------------------+
| **Model**     | IBM x342 Server; System Model: eserver xSeries 342   |
+---------------+------------------------------------------------------+
| **Memory**    | 1 GB of SDRAM                                        |
+---------------+------------------------------------------------------+
| **CPU**       | Quantity of 2, Pentium III, 1.0 Ghz                  |
+---------------+------------------------------------------------------+
| **Disk        | C: 7G Used 2.52G Free 4.47G (This is for system      |
| Space**       | files only)                                          |
|               |                                                      |
|               | D: 9.94G Used 1.11G Free 8.82G                       |
+---------------+------------------------------------------------------+

The development environment is installed with Windows 2000 Server as
operating system.

The production environment has 2 functions. It will act as web server,
and database server. The IIS 5.0 is installed for the web server and SQL
Server 2000 for the database server. For these applications, USDA has
the enterprise licensing agreement. The production server is not located
locally; therefore, terminal service will be used to connect to the
production server.

### **Technical Architecture diagram**

![](media/image2.png){width="0.49094488188976376in"
height="0.49015748031496065in"}![](media/image4.wmf){width="0.24803149606299213in"
height="0.6338582677165354in"}![](media/image5.wmf){width="0.24803149606299213in"
height="0.6338582677165354in"}![](media/image4.wmf){width="0.24645669291338582in"
height="0.6338582677165354in"}![](media/image5.wmf){width="0.24645669291338582in"
height="0.6338582677165354in"}![](media/image4.wmf){width="0.24803149606299213in"
height="0.6338582677165354in"}![](media/image5.wmf){width="0.24803149606299213in"
height="0.6338582677165354in"}![](media/image4.wmf){width="0.24803149606299213in"
height="0.6338582677165354in"}![](media/image5.wmf){width="0.24803149606299213in"
height="0.6338582677165354in"}

The users will have access to the Prototype production server through
the USDA Intranet only. The developers will have the access to Prototype
production server using the existing USDA network structure along with
Terminal Service.

The production environment is access by USDA Intranet only. The static
IP address for the prototype is XXX.XXX.XXX.XXX. The production
environment is configured to act as web server using IIS 5, and database
server installed with SQL Server 2000.

### **Backup**

Storage management is on of the value added services performed by the
NITC. The administration of all DASD volumes and datasets is continuous
task at the NITC. The Center currently maintains approximately 14.1
terabytes of online space containing over one million datasets. This
administration work is performed through the use of specific DASD pools,
specialized software dataset migration, backup and recovery strategies,
and utilization reports.

Backup activities include the following:

-   Incremental backups. Backups of new or updated DASD datasets are
    taken nightly. Backup tapes are kept on-site in the tape library.

-   Weekly/monthly full volume backups. Backups of all DASD volumes are
    accomplished on a weekly and/or monthly basis. (Retention for
    monthly backups is longer). Full volume backups are used at the NITC
    and are kept off-site, but can be recalled whenever needed.

-   ABARS backups. IBM's Aggregate Backup and Recovery Support (ABARS)
    software was selected by the NITC for disaster backup and recovery.
    The NITC's customers assume responsibility for determining the
    applications saved and the frequency of these backups.

Administration of tape datasets is also provided by the NITC. The Center
uses IBM's Removable Media Manager (RMM) software to handle tape
management functions under S/390. RMM provides automated functions such
as tape initialization, extraction, and expiration processing;
management of tape movement and retention throughout the life cycle;
recording of dataset and volume information; and management reporting.
In addition to the approximately one half million tape cartridges stored
in the tape library, the NITC also maintains and operates two Storage
Tek Automatic Cartridge Systems ("silos") which house an additional
11,000 cartridge tapes.

### **Disaster Recovery**

NITC concurrently restores software and data including operation system
software, utilities, database software, and critical data, and the
backbone communications network within 72 hours. When this task is
complete, the prototype team will be given access to the hot site
computer system to recover selected applications that have been
previously backed up and taken to the off-site location for storage.
NITC personnel are available through the duration of the recovery to
assist customers in restoration of systems and data.

### **Failover**

NITC will notify the Prototype team of failure or anticipated system
failure, initiate procedures required to resume normal operations, and
keep the team apprised of progress towards resuming normal operations.
Response to a trouble report will be initiated immediately during normal
business hours, and within two hours outside of normal business hours.

### **Load Balancing**

There is no need for load balancing since the prototype is very small
website with small user group.

### **Security Plan**

NITC will provide constant network intrusion detection of unauthorized
access to systems. The prototype team will be notified immediately of
any unauthorized access. Vulnerabilities in systems will be detected by
scanning servers with Internet Scanner Software (ISS) on a monthly
basis.

### **Roles and Responsibilities**

In the production environment, both NITC and the eGovernment Prototype
team must work together to make sure the production environment run
smoothly. There are separate roles and responsibilities for NITC and the
eGovernment prototype team.

Roles and responsibilities for NITC are the following:

-   Procure hardware, software, maintenance and services for the
    project;

-   Facility Security;

-   24/7 server availability;

-   USDA WAN connectivity;

-   Internet intrusion detection/vulnerability assessment;

-   Notification of downtime/ system failure;

-   System outage logs;

-   Support for enhancements, optimization, installations, maintenance,
    and upgrades; and

```{=html}
<!-- -->
```
-   Backups.

The eGovernment Prototype team will be responsible for developing and
maintaining the content of the prototype in the production environment.

# Folder Structure

All source code will be developed in the development environment. The
coding of each ASP and HTML file should be developed in its own
sub-directory on the Web server under
"\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\\..." There are
sub-folders under USDA_Prototype directory that are created for each
scenario. When adding a new scenario, new sub-folder must be created
within USDA_Prototype folder. The name of the sub-folder should be
representative of the new scenario.

Folder structures for source code in development environment:

[\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\eGrants\\](../../../../../../../../wwwroot/USDA_Prototype/eGrants/)

[\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\EIS\\](../../../../../../../../wwwroot/USDA_Prototype/EIS/)

[\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\eLearning\\](../../../../../../../../wwwroot/USDA_Prototype/eLearning/)

[\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\FoodSafety\\](../../../../../../../../wwwroot/USDA_Prototype/FoodSafety/)

[\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\OTA\\](../../../../../../../../wwwroot/USDA_Prototype/OTA/)

[\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\eExtention\\](../../../../../../../../wwwroot/USDA_Prototype/eExtention/)

[\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\Footer\\](../../../../../../../../wwwroot/USDA_Prototype/Footer/)

[\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\Personalize\\](../../../../../../../../wwwroot/USDA_Prototype/Personalize/)

Other folders related to prototype business scenarios:

Main folder:

\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\Prototype Business
Scenarios

Sub folders under main folder:

> \\1 - Deliverable Templates\\
>
> \\2 - Scope Documents\\
>
> \\3 - Site Maps\\
>
> \\4 -- Storyboards\\
>
> \\5 - UI Design\\
>
> \\6 - CAR Diagrams\\
>
> \\7 - Demo Scripts\\
>
> \\Status Meetings\\

Two other folders in development environment are the technical
architecture folder and TestScript folder. The technical architecture
folder ([\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\Technical
Architecture\\](../../../../../../../..//199.129.42.209/wwwroot/USDA_Prototype/Technical%20Architecture/))
contains all technical documents and the TestScript folder
([\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\TestScript\\](../../../../../../../../wwwroot/USDA_Prototype/TestScript/))
contains test scripts and database to report any issues arisen from
testing.

The production environment will have the same folder structure as
development environment, but will only have folder containing source
codes.

Folder structures for source code in production environment:

[\\\\xxx.xxx.xxx.xxx\\wwwroot\\USDA_Prototype\\eGrants\\](../../../../../../../../wwwroot/USDA_Prototype/eGrants/)

[\\\\xxx.xxx.xxx.xxx\\wwwroot\\USDA_Prototype\\EIS\\](../../../../../../../../wwwroot/USDA_Prototype/EIS/)

[\\\\xxx.xxx.xxx.xxx\\wwwroot\\USDA_Prototype\\eLearning\\](../../../../../../../../wwwroot/USDA_Prototype/eLearning/)

[\\\\xxx.xxx.xxx.xxx\\wwwroot\\USDA_Prototype\\FoodSafety\\](../../../../../../../../wwwroot/USDA_Prototype/FoodSafety/)

[\\\\xxx.xxx.xxx.xxx\\wwwroot\\USDA_Prototype\\OTA\\](../../../../../../../../wwwroot/USDA_Prototype/OTA/)

[\\\\xxx.xxx.xxx.xxx\\wwwroot\\USDA_Prototype\\eExtention\\](../../../../../../../../wwwroot/USDA_Prototype/eExtention/)

[\\\\xxx.xxx.xxx.xxx\\wwwroot\\USDA_Prototype\\Footer\\](../../../../../../../../wwwroot/USDA_Prototype/Footer/)

[\\\\xxx.xxx.xxx.xxx\\wwwroot\\USDA_Prototype\\Personalize\\](../../../../../../../../wwwroot/USDA_Prototype/Personalize/)

# Filename standard

It is very important to use naming convention for filenames to avoid any
confusion as to the intent of the files. Filename format should start
with scenario name and include few words describing the content and with
asp extension.

Example:

USDA_eGrant_home.asp (Homepage for eGrant scenario).

# Unit Test

Unit test must be complete in the development environment. Once the
latest HTML and ASP files are promoted to the Web server, the
development team can commence the appropriate tests. As the issues are
identified, the developer should perform the appropriate modifications.
After the developers have completed the modifications, it should be unit
tested again. Once the development passes 3 rounds of unit test, it is
ready to be move into system test.

# Adding Business Scenarios

Adding a new scenario is a painless process, however, it requires
attention to detail.

**Step 1:** Determine which new scenario to add.

**Step 2:** Create business scenario scope document.

**Step 3:** Create initial high-level site map for scenario.

**Step 4:** Update overall site map.

**Step 5:** Create paper-based storyboard.

**Step 6:** Develop content for each screen in site map.

**Step 7:** Finalize content for each screen in site map.

**Step 8:** Create demo script.

**Step 9:** Develop ASP/HTML according to Style Guide. Following
consideration must be considered during coding:

1.  Ensure the new scenario is consistent with the existing site design
    and style guide.

2.  Create or copy a template adhering to existing guidelines and style
    sheets.

3.  Give the template an ASP extension. This is done to remain
    consistent with the prototypes architect used through out the site.
    Be sure to write code so that the content is broken up into modules.

4.  Identify these modules with HTML comment tags. This is to done to
    enhance content management and it makes it easier for someone to
    open the file and make modifications.

5.  Unit test by the developer.

**Step 9:** Develop test scripts.

**Step 10:** Unit test.

**Step 11:** System Test.

**Step 12:** Send out message to system owner.

**Step 13:** Migrate to production.

# Process to modify existing scenarios

Content Management refers to the combination of people, processes, and
technology that enables an organization to manage content through all
phases of production. The content component of a web site should be
treated like all other capability components with a formalized
development and maintenance process, well defined roles to own those
processes, and sufficient resources to accomplish the desired goals.

For the prototype, the volume of published content will be small and
there will be no need for vendor content management tools. Although,
since there will be no content management tools, it is still important
to manage the relationship between the editable and consumable forms of
content. The capability includes the storing, versioning and work flow
management of content. Below are the instructions to follow for
Prototype Content Management or modifying existing scenarios or modules.

**Step 1:** Locate the proper directory. This can be found in the URL of
the page that needs modification. The directory and URL structures are
mirrors of each other.

Example:

For eGrant scenario for development environment:

[\\\\199.129.42.209\\wwwroot\\USDA_Prototype\\eGrants\\](../../../../../../../../wwwroot/USDA_Prototype/eGrants/)

The file can be located in the "main_directory" or under the "eGrants"
for an example.

Follow the below steps to modify contents:

**Step 2:** Locate the file in development environment. Copy the file
and rename it with XXX_MMDDYYYY.asp. Put the copied file into backup
directory within development environment.

**Step 3:** Open the original file and make HTML changes. Then save.
When deleting html, keep in mind that there may be content gaps in the
page in which code you are modifying. To avoid this it may be necessary
to move html code around to make up for code gaps.

**Step 4:** Test the changes in the development environment to make sure
nothing was broken or misspelled. Once it is determined that the changes
made did not cause functionality errors, it is ready to push the page to
the production server.

**Step 5:** Locate the file in production environment that needs
modification. Copy the file and rename it with XXX_MMDDYYYY.asp. Put the
copied file into backup directory within production environment.

**Step 6:** Copy the file from development environment to production
environment and override existing file.

**Step 7:** Test the changes in the production environment.

# Appendix A -- Contact List

The purpose of this contact list is to identify for NITC and the
Prototype team to contact on project issue. The person should be
contacted in the order they appear on the list, until contact is
established.

eGov Contacts:

  ----------------------------------- -----------------------------------
  **Name**                            **Work Phone \#**

  Chris Niedermayer, eGov Executive   202-690-2118

  Adam Siegel, eGov team              202-720-6144

  Yeong C. Kim, eGov team             202-720-6144
  ----------------------------------- -----------------------------------

NITC Project Contacts:

  ----------------------------------- -----------------------------------
  **Name**                            **Work Phone \#**

  Dave Ireland                        816-926-2331
  ----------------------------------- -----------------------------------

NITC Technical Contacts:

  ----------------------------------- -----------------------------------
  **Name**                            **Work Phone \#**

  Crystal Wallace, System             816-823-1193
  Administrator                       

  Michael Craig, Backup Sys           816-823-4944
  Administrator                       
  ----------------------------------- -----------------------------------

# Appendix B -- NITC Service Description

NITC Gold Level Service Description

1.  [System Administration]{.underline}. Provide on-site technical
    expertise to maintain project hardware, systems software, and
    Independent Software Vendor (ISV) software. This includes
    enhancements, optimization, installations, maintenance, and upgrades
    to existing hardware and system software components, as agreed upon
    by both parties. Provide security at the operating system level with
    User ID/Password verification.

2.  [Daily and weekly backups to tape]{.underline}. Data management
    procedures for the project will include daily and weekly backups.
    Multiple sets of backup tapes will be maintained and cycled through
    an on-site tape library as well as an off-site storage facility. The
    off-site storage will be provided through NITC's contracted
    provider.

3.  [Procurement]{.underline}. Procure hardware, software, maintenance
    and services for the project. After that time NITC will continue to
    procure maintenance for the hardware and operating system software
    at the agreement of both parties. The actual cost for the
    maintenance will be charged back to the customer.

4.  [Facility security]{.underline}. Secure access to the equipment. The
    computer room is located in a facility secured by a guard station
    located inside of the building. This station contains surveillance
    screens allowing monitoring of the surrounding parking lots and
    entrances to the computer room. Entrance to and egress from the
    operations areas can be made only through a buffer zone with access
    restricted by a biometric entry system.

5.  [Computer room raised floor space]{.underline}. Host the project
    server(s) on computer room raised floor space in the NITC facility.
    This environment is supported by dual electrical leads into the
    computer room, an uninterrupted power supply (UPS), redundant power
    (supplied by diesel generators), climate control, and dual-source
    fire protection. The workstation includes a desktop workspace and
    storage space for manuals and tapes.

6.  [Server availability 24 hours a day, 7 days a week]{.underline}.
    Assure the project platform will be available 24 hours per day, 365
    days per year, except for scheduled downtimes. Scheduled downtime
    notifications will be made through Customer Memos and regular
    Customer Meetings. NITC will work with the customer to provide,
    monitor and control ADP resources and ensure system and network
    availability and efficient system performance.

7.  [USDA WAN connectivity]{.underline}. Maintain and operate the
    project telecommunications infrastructure for connectivity to the
    Internet via the USDA backbone. 100 Megabit per second access to the
    server(s) will be provided via a TCP/IP network. Coordinate with the
    customer for any telecommunications infrastructure changes that
    would affect the operations of the project platform.

8.  [Internet intrusion detection/vulnerability assessment]{.underline}.
    The NITC will provide constant network intrusion detection of
    unauthorized access to systems. The customer will be notified
    immediately of any unauthorized access. Vulnerabilities in systems
    will be detected by scanning servers with Internet Scanner Software
    (ISS) on a monthly basis.

9.  [System Network Control Center]{.underline}. NITC System Network
    Control Center (SNCC) will accept calls from the customer
    first-level Point of Contact (POC) Contact List. If necessary, the
    individual from NITC's SNCC will contact the on-call NITC technical
    specialist. NITC technical specialist will initiate calls to project
    equipment and/or software vendors for support, track problems using
    the NITC problem and change management process, and coordinate with
    any involved parties.

10. [Notification of anticipated downtime]{.underline}. Provide a
    minimum of a 24-hour notice for any anticipated, but unscheduled
    downtime (e.g. if NITC must replace a piece of equipment that is
    giving a warning sign of failing) to the customer Point of Contact
    (POC). These downtimes generally will not exceed two hours and will
    be for the purpose of performing infrastructure, hardware, system
    software, or application software maintenance, or to perform data
    backups. All scheduled downtimes will be negotiated with the
    customer and will be planned to avoid downtime affecting peak usage
    time periods.

11. [Notification of system failure.]{.underline} Notify the appropriate
    customer POC (See B for Contact List) of failure or anticipated
    system failure, initiate procedures required to resume normal
    operations, and keep the customer POC apprised of progress towards
    resuming normal operations. Response to a trouble report will be
    initiated immediately during normal business hours, and within two
    hours outside of normal business hours.

12. [System outage logs]{.underline}. Maintain a log of all NITC system
    and network outages.

13. [Billing]{.underline}. Bill the customer on a monthly basis.