# 21. COTS Hardware Maintenance

Hardware for the ECS Project consists of commercial-off-the-shelf (COTS)
hardware procured especially for the ECS Project and a minimal amount of
Government furnished equipment (GFE). As of August 31, 1997, the
Government had provided only Liebert power distribution units (a COTS
hardware product) for use and support by the ECS Contractor. Therefore,
M&O support (maintenance) tasking for only COTS hardware is discussed in
this section. The discussion is based on: 194-602-OP1-001, "Property
Management Plan for the ECS Project"; 423-41-02, "Functional and
Performance Requirements Specifications"; 501-CD-001, "ECS Performance
Assurance Implementation Plan"; 601-CD-001-004, "Maintenance and
Operations Management Plan"; 613-CD-002-001, "Version 2.0 COTS
Maintenance Plan for the ECS Project"; 616-CD-002-001, "Version 2.0
Integrated Support Plan for the ECS Project"; 532-CD-002-001,
"Environmental Control Plan for the ECS Project."

## 21.1 COTS Hardware Support - General

COTS hardware support consists of preventive maintenance and corrective
maintenance accomplished in compliance with procedures and guidance
contained in this section. Preventive maintenance and corrective
maintenance may be accomplished by the warranty support provider, COTS
hardware support provider contracted for post-warranty support, or the
local maintenance coordinator (LMC) using local DAAC resources. COTS
hardware support is available from the COTS hardware support provider
during the contracted principal period of maintenance (PPM) or on a
time-and-materials (T&M) basis, in accordance with the terms of the
appropriate support contract. The LMC ensures that requirements of this
section are complied with by all COTS hardware support providers and
that information entered in the electronic trouble ticket is accurate.

The integrated logistics support (ILS) maintenance coordinator is a
staff position in the ILS office, which is under the ILS manager in the
maintenance and operations (M&O) managerÕs area of responsibility. The
ILS maintenance coordinator is available during normal work hours to
provide assistance to the LMC in obtaining COTS hardware support if
normal c-in efforts do not provide a satisfactory vendor effort and to
receive information from the LMC. The ILS maintenance coordinator may be
reached via the Internet, telephone, or FAX with the Internet being the
preferred method. The Internet address is ilsmaint@eos.hitc.com; the
telephone number is 1-800-ECS-DATA, select option #1 then dial 4180 or
5180. The FAX number is 1-301-925-0438.

### 21.1.1 Corrective Maintenance

Corrective maintenance is the unscheduled repair of equipment and
includes fault detection, diagnosis, isolation, and resolution through
line replaceable units (LRU) replacement. COTS hardware corrective
maintenance will be documented using procedures in SectionÊ8.1, Trouble
Ticket System Procedures; Section 9, Configuration Management; the
safety requirements of Section 21.1.4, COTS Hardware Support Safety, and
the guidance found in 609-CD-001-001 and expanded in document
609-DR-002-001 (Draft). The 609 document provides Project guidance and
direction for documenting COTS hardware problems, corrective actions,
and hardware support management actions. The 609 document may be found
on the Internet at http://edhs1.gsfc.nasa.gov/. Once at the web site,
type in the number 609 in the Quick Search block to access all versions
of the 609 document.

### 21.1.2 Preventive Maintenance 

EMASS and Storage TechnologyÕs automated tape library robots are
currently the only hardware requiring scheduled preventive maintenance.
Preventive maintenance is typically performed by the original equipment
manufacturer (OEM) during a corrective maintenance procedure. Otherwise
it will be scheduled by the LMC in conjunction with the responsible
maintenance organization and the using organization to minimize
operational impact. Preventive maintenance will be documented by
preparing a trouble ticket and a MWO using procedures in this documentÕs
Section 8.2, Using the Trouble Ticket System Tool and the guidance found
in the 609 document.

### 21.1.3 Configuration Management

Configuration Management (CM) requirements are addressed in Section 9 of
this document. Ensuring compliance with Section 9 procedures during COTS
hardware support by a COTS hardware support provider or local site
support personnel is the responsibility of the LMC. The process to be
used in documenting CM issues is contained in the 609 document.

### 21.1.4 COTS Hardware Support Safety

COTS hardware support will be accomplished using safety procedures to
protect personnel and equipment from harm. Guidance for establishment of
safety practices, standards, and procedures is found in Section 6 of the
ECS Performance Assurance Implementation Plan (PAIP), 501-CD-001-004.
The LMC will ensure that these safety procedures, as well as applicable
local safety requirements, are known and observed during COTS hardware
support by local site support personnel or COTS hardware support
providers.

COTS hardware safety practices include electrostatic discharge (ESD)
protection; safe dissipation of static electricity; work place common
grounding requirements; and parts handling and protection while in
storage, outside the manufacturerÕs protective packaging, and being
readied for installation or removal. When not being worked on or when
outside protected areas, electronic parts and assemblies are to be
covered by an ESD protective covering or packaging. During installation
or removal of electronic parts or LRUs, a common ground will be
established between the technician, worktable/area, the part/LRU, and
the equipment it is to be installed in/removed from. The ESD program
will be locally developed by the LMC using the Environmental Control
Plan, 532-CD-002-001 for guidance.

It is the responsibility of the LMC to ensure compliance with these
safety procedures by the COTS hardware support provider or local site
support personnel.

## 21.2 COTS Hardware Support - Contract Information 

The ECS procurement organization is located at the ECS development
facility (EDF) and is responsible for ordering initial warranty support
and post-warranty support where appropriate. Questions or comments
concerning COTS hardware support are to be directed to the ILS
maintenance coordinator using procedures contained in Section 21.1, COTS
Hardware Support - General.

### 21.2.1 Management of COTS Hardware Support Contracts

COTS hardware is typically procured with a twelve-month extended on-site
warranty from the OEM. Some resellers provide two or three years of
return-to-depot warranty support instead of on-site support. At the
conclusion of the warranty period a support contract for one or more
years is purchased. In arriving at a support decision for a COTS
hardware item, the ECS ILS office considers operational availability
(Ao) and mean down time (MDT) requirements, operational commitments, and
costs. Once maintenance terms are determined, the ILS maintenance
coordinator directs the ECS procurement office to procure follow-on
(post-warranty) COTS hardware maintenance support. Based on support
considerations, OEM support is typically procured. However, cost and
support considerations may result in support being provided by a third
party provider. The support providerÕs identity and maintenance terms
are entered into the ILS web pageÕs hardware support spreadsheet (see
Table 21.2-1) and the maintenance fields in the procurement database
(see Table 21.2-2) by the ILS maintenance coordinator. To access the ILS
web page open the EDHS home page (http://edhs1.gsfc.nasa.gov/) then
select the Maintenance & Operations button. When that page opens, scroll
to the bottom and click on the COTS Hardware - Software link. You are
now in the ILS COTS ECS Hardware - Software Maintenance web site.
Continue to click the next page links at the bottom of each page (except
for the first page, itÕs at the top) until you reach that part of the
ILS web site desired.

Table 21.2-1. ILS Web Page Hardware Support Fields

+--------------+-----------------+-----------------+-------------------+
| > **1**      | **2**           | > **3**         | > **4**           |
+--------------+-----------------+-----------------+-------------------+
| [Authorized  | [ManufacturerÕs | Support         | [Type             |
| Caller]      | Na              | [ProviderÕs     | Equip             |
| {.underline} | me]{.underline} | Na              | ment]{.underline} |
|              |                 | me]{.underline} |                   |
+--------------+-----------------+-----------------+-------------------+
| > **5**      | > **6**         | > **7**         | > **8**           |
+--------------+-----------------+-----------------+-------------------+
| Information  | Contract        | Support         | Support Contract  |
| Needed to    | [Maintenance    | VendorÕs [Name  | Terms (type spt,  |
| Obtain       | Ter             | and Contact     | [response time &  |
| [Vendor      | ms]{.underline} | #               | PPM)]{.underline} |
| Support]     |                 | Õs]{.underline} |                   |
| {.underline} |                 |                 |                   |
+--------------+-----------------+-----------------+-------------------+

### 21.2.2 Contract Maintenance Terms

Warranty maintenance is provided by the OEM/vendor during the first
year. Typically post-warranty support is provided by the same OEM/vendor
that provided the warranty support. The LMC can determine equipment
types, their maintenance support provider, and other information
relating to their support by referring to the COTS hardware spreadsheet
in the ILS web page.

### 21.2.3 COTS Hardware Database

Information related to COTS hardware support contracts is maintained in
a procurement database. Table 21.2-3, Typical Procurement Database
Fields, shows fields normally in such a database.

Table 21.2-2. Typical Procurement Database Fields

  ----------------- -----------------------------------------------------
  Field Name        Information

  PO number         Purchase order number

  Vendor            Name of the firm that sold the COTS product

  Vendor Code       Abbreviated vendor name

  ECS Equip Name    ECS name given to a major hardware item

  Status            Status as to received, installed, consumable item,
                    out for maintenance, or archived

  Parent ID Number  EIN assigned to a major hardware item by the property
                    administrator

  Part ID Number    EIN applied to a subordinate hardware item by the
                    property administrator

  Part Number       OEMÕs part number

  Serial Number     OEMÕs assigned serial number

  Cost of Unit      Purchase order cost

  ProductÕs Name    Name of the item as stated on vendor purchase order

  Model or Version  Model or version of the item

  OEMÕs Initials    Initials of the OEM or software developer

  OEMÕs Name        Full name of the OEM or software developer

  Date Hardware     Date item received by ECS (generally at the EDF)
  Received          

  Received by       Name of person receiving the COTS product from the
                    vendor

  ReceiverÕs Phone  Phone number of person that received the products

  Hardware          Date item was first installed ( date the warranty
  Installation Date period begins).

  Date Support Ends ECS COTS hardware or software warranty or support
                    expiration date

  User of the       Name of the person assigned responsibility for the
  Hardware          ECS COTS product

  UserÕs Phone      Phone number of person assigned responsibility for
                    ECS COTS product

  Location          Site where the product is located (E. G., GSFC, SMC,
                    EOC, etc.)

  Building          Building in which the product is located

  Room              Specific room in which the product is located

  Maintenance       The contract number or maintenance vendorÕs access
  Contract          authorization information

  Maintenance Code  Initials of the maintenance support provider

  Maintenance       The name of the firm providing maintenance support
  Vendor            

  Maintenance       Phone number for the maint vendorÕs technical support
  ProviderÕs Phone  desk or help desk

  Hardware or       Identification of the product as hardware or software
  Software          

  Comment           For any remarks to be added that are not included in
                    other fields
  ----------------- -----------------------------------------------------

The ECS ILS office uses the procurement database information to manage
COTS hardware support contracts. During Release B, the LMC can obtain
extracts of maintenance contract information via the Internet on the ILS
web page. Information fields in the ILS web page are updated
periodically by the ILS maintenance coordinator.

The LMC can obtain specific COTS hardware information from the
procurement database by requesting the ILS maintenance coordinator to
run a special query. The LMC should be specific in identifying the
information wanted. If the information requested is too voluminous to
provide to the LMC telephonically, it will be E-mailed to the requester.
The request for hardware support information should be made using
procedures in Section 21.1. The time it takes for the ILS maintenance
coordinator to provide a reply will vary with the criticality and
complexity of the request.

Generally, COTS hardware support providers require an access code and/or
the serial number of the host equipment item to verify that the item is
under support contract (e.g., support requested for a computer monitor
or keyboard problem, the parent workstation or server serial number
would be provided as the access code). The ILS maintenance coordinator
determines what information is needed by the maintenance providers to
verify support is authorized. For some equipment, names of authorized
contact persons are required by the COTS hardware support provider. The
ILS maintenance coordinator arranges, within the limits imposed by the
COTS hardware maintenance provider and the needs of individual DAACs,
for specified personnel to become authorized contact persons. The ILS
maintenance coordinator will include the authorized contact person list
in the appropriate field in the ILS web page COTS hardware support
spreadsheet. Recommended changes to the names of authorized contacts are
to be provided to the ILS maintenance coordinator by the LMC for
updating the ILS web site COTS hardware support spreadsheet. The LMC is
to identify changes to the authorized contact list as a permanent or
temporary change and, if temporary, the inclusive dates of the change. A
temporary change may occur when the authorized contact person is ill, on
vacation, in training, or other short-term change of work availability
status has occurred or is expected to occur. The LMC is to provide the
ILS maintenance coordinator the change information as soon as it is
known, using procedures in Section 21.1.

## 21.3 Hardware Repairs - Standard

The LMC is the DAACÕs focal point for directing and coordinating
corrective maintenance of ECS hardware. The users, operators,
maintainers, and managers of the ECS hardware require the aggregate
capability to report, track, control, monitor, open, and close hardware
problems and solutions to ensure the ECS hardware suite is retained in
an approved configuration. Each of these people (users, maintainers, and
managers) has specific capability requirements not needed by others. The
users and operators of the ECS hardware need only to be able to report
encountered problems to a central office for resolution and obtain
timely feedback on the outcome of the problem resolution effort. The
maintainers need a capability to be promptly informed of problems, open
and close trouble tickets and maintenance work orders, dispatch
system/network administrators or repair technicians/engineers, update
repair efforts as they occur, capture and retain problems and solutions
for future reference including part numbers, serial numbers, location,
and equipment ID. Managers need a capability to track repair status, use
hardware information to update property records when changes occur,
review proposed and actual repair actions for appropriateness, ensure
repair actions maintain approved ECS configuration, and identify support
problem areas. These capabilities are provided for use by each area
through the use of the Remedy applicationÕs trouble ticket, XRPÕs
inventory, logistics and maintenance capability (includes a maintenance
work order/dispatch ticket), system configuration and baseline
management capability, and DDTS for configuration change requests (CCR)
recording and processing.

Users, operators, and support personnel who encounter a problem will
report the problem using the Remedy trouble Ticket application and
guidance in document 609-CD-003-001, Version 2.0 Operations Tool Manual.
Maintainers will be notified of problems, dispatch appropriate
personnel, ensure accuracy of information and updates, track repairs,
and close out the dispatch work order using procedures in this section
and guidance in document 609-CD-CD-003-001, Version 2.0 Operations Tool
Manual.

Managers of ECS hardware will use both the work order information and
the configuration management information contained in this section and
Section 9, Configuration Management, of this document and guidance found
in document 609-CD-CD-003-001, Version 2.0 Operations Tool Manual

### 21.3.1 Hardware Problem Reporting

COTS hardware problems are tracked by the LMC from the initial
notification of a COTS HW problem through its successful repair. The
vehicle for tracking COTS HW problems is a combination of the trouble
ticket described in Section 8, Configuration Management, and the
Maintenance Work Order (MWO) from the XRPÕs Inventory, Logistics, and
Management application described in document 609-CD-003-001, Version 2.0
Operations Tools. Anyone may open a trouble ticket. Thus, the first
person \[this may be either a user, operator, LMC, system administrator
(SA), network administrator (NA), or responsible engineer (RE)\]
experiencing or observing a COTS hardware problem will initiate a
trouble ticket. Usually the opening of a COTS HW trouble ticket is by a
user or operator since they are most likely to be the first person to
experience a problem with the ECS COTS products. In all cases the
trouble ticket is annotated with the specifics of the problem, equipment
it is on and the equipmentÕs location. Once the problem information is
keyed in, the trouble ticket is forwarded to the User Services office.
Typically, the User Services office assigns the trouble ticket to an
administrator (systems/network) for initial verification of the problem.
After verification and any repair efforts by the administrator (to be
annotated on the trouble ticket) it is forwarded by the administrator to
the organization best capable of performing the repair/fix. If the
problem is with COTS HW, the administrator forwards (by E-mail) the
trouble ticket to the LMC for resolution., constituting notification of
a problem.

> *Note: Remedy has a second level hardware information screen that was
> developed for a previous release but is not used in Release B0 or
> subsequent releases.*

Upon receiving the trouble ticket the LMC opens a MWO using the XRP ILM
application. The ILM application generates a unique number for each MWO
opened and prints it on the MWO. The trouble ticket and the MWO are
cross linked by unique reference numbers by the LMC. The LMC cross
references the trouble ticket to the MWO by keying into the MWO the
trouble ticketÕs own unique identifying number. The MWO is referenced to
the trouble ticket by the LMC keying into the trouble ticketÕs Comment
block the MWOÕs unique identification number.

The MWO may be printed and given as a hard copy to the
technician/engineer assigned to correct the problem. The
technician/engineer must complete the hard copy MWO with thorough and
legible entries. The LMC is responsible for accurately and timely
entering the hand scribed information into the electronic copy of the
MWO that is part of the ILM tool. If more convenient, the
technician/engineer assigned to repair the COTS hardware may enter the
data directly to the electronic MWO. However, the LMC is still
responsible for verifying the accuracy of the information. If a change
needs to be made to a MWO after it has been completed and forwarded, the
ILM application provides such a capability through a change screen. It
is the LMCÕs responsibility to ensure such changes are made or
personally accomplish the changes..

The MWO is used by the LMC to dispatch and monitor site and OEM
personnel in the resolution of the problem indicated on the trouble
ticket and the recording of events and information associated with the
problem solution and closure. The LMC will ensure that the MWO contains
a complete and accurate sequence of events as requested in the ILM data
fields. Information as to the applications that were running, the
commands given immediately prior to the apparent COTS hardware problem,
the troubleshooting accomplished, and embedded diagnostics that were
exercised needs to be included in the narrative portion of the MWO.
Information provided by the maintenance contractorÕs technical support
center, as well as the results of using that information, is also to be
annotated in the narrative portion of the MWO. The accuracy and
completeness of the MWO information is paramount . It is the LMCÕs
responsibility to ensure this information is accurately entered.

Upon completion of the MWO resulting in the failed item being made
operational, the trouble ticket is annotated with a briefly comment to
indicate the problem was resolved and to ensure the link to the MWO Ôs
unique identification number is accurate. An electronic copy of the MWO
and the trouble ticket is forwarded to the Configuration Administrator
by the LMC. The Configuration Administrator acts as closing authority
for opened trouble tickets. The LMC will ensure that the procedures of
Section 8 and guidance in the 609 document are followed in annotating
the trouble ticket and the MWO as well as when forwarding trouble
tickets and MWOÕs to the Configuration Administrator.

### 21.3.2 Initial Troubleshooting/Diagnostics

Once failure occurs and the operator, SA, and/or NA are the first to
recognize the problem, they will perform diagnostics to isolate the
problem to its source (i.e., Operating System, COTS software, ECS
software, science software, network, or COTS hardware) using the actions
in Table 21.3-1, Initial Troubleshooting/Diagnostics Procedures. If
these efforts do not correct the problem, they will open a trouble
ticket and include all actions they have take, results of those actions,
and a recommendation as to who the trouble ticket should be forwarded to
for corrective action.

Table 21.3-1. Initial Troubleshooting/Diagnostics Procedures

+----+----------+-----------------------------------------------------+
| St | Oc       | Action                                              |
| ep | currence |                                                     |
+----+----------+-----------------------------------------------------+
| 1  | System   | a\. Review error message against the applicable     |
|    | problem  | hardware/software operator manual.                  |
|    | di       |                                                     |
|    | scovered | b\. Verify that power, network, and interface       |
|    | by an    | cables are properly connected and functioning       |
|    | o        | properly.                                           |
|    | perator, |                                                     |
|    | SA or    | c\. Run internal systems and/or network             |
|    | NA.      | diagnostics.                                        |
|    |          |                                                     |
|    |          | d\. Review system logs for evidence of previous     |
|    |          | related problems or configuration changes that may  |
|    |          | be contributing to the problem.                     |
|    |          |                                                     |
|    |          | e\. Attempt to reboot the system.                   |
|    |          |                                                     |
|    |          | f\. If problem is fixed a trouble ticket is not     |
|    |          | needed.                                             |
|    |          |                                                     |
|    |          | g\. If the problem is not fixed go to Table 21.3-2  |
|    |          | of this document.                                   |
+----+----------+-----------------------------------------------------+
| 2  | System   | a\. User Services office forwards trouble ticket to |
|    | problem  | the SA and/or NA who do the following:              |
|    | reported |                                                     |
|    | to User  | > 1\) Review error message against the applicable   |
|    | Services | > hardware/software operator manual. Record the     |
|    | office   | > error message on the trouble ticket.              |
|    | by other | >                                                   |
|    | than an  | > 2\) Verify that power, network, and interface     |
|    | o        | > cables are properly connected and functioning     |
|    | perator, | > properly.                                         |
|    | SA or NA | >                                                   |
|    |          | > 3\) Run internal systems and/or network           |
|    |          | > diagnostics.                                      |
|    |          | >                                                   |
|    |          | > 4\) Review system logs for evidence of previous   |
|    |          | > related problems or configuration changes that    |
|    |          | > may be contributing to the problem.               |
|    |          | >                                                   |
|    |          | > 5\) Attempt to reboot the system.                 |
|    |          |                                                     |
|    |          | b\. Problem is resolved                             |
|    |          |                                                     |
|    |          | > 1\) Complete the trouble ticket and forward to    |
|    |          | > Configuration Administrator.                      |
|    |          |                                                     |
|    |          | c\. Problem is not resolved                         |
|    |          |                                                     |
|    |          | > 1\) Annotate trouble ticket with all actions      |
|    |          | > taken and their results                           |
|    |          | >                                                   |
|    |          | > 2\) Forward trouble ticket to the organization    |
|    |          | > you believe responsible for correcting the        |
|    |          | > problem.                                          |
|    |          | >                                                   |
|    |          | > NOTE: Report software related problem to the      |
|    |          | > SA/Sustaining Engineering Organization (SEO)      |
|    |          | > (refer to Section 22, Software Maintenance) If    |
|    |          | > the problem is hardware related notify the LMC.   |
+----+----------+-----------------------------------------------------+

Should User Services office receive the initial problem report from
other than the operator, SA, and/or NA the trouble ticket is routed by
User Services office to the SA and/or NA for initial
troubleshooting/diagnostics using procedures in Table 21.3-1, Initial
Troubleshooting/Diagnostics Procedures. If their efforts succeed in
correcting the problem they close out the trouble ticket and forward it
to the Configuration Administrator. If their efforts do not result in a
correction of the problem the SA and/or NA annotate the trouble ticket
with the actions they have taken and the results before forwarding the
trouble ticket to the organization they believe to be the appropriate
one to perform the needed corrective action.

### 21.3.3 Hardware Corrective Maintenance Actions

Hardware problems are reported to the LMC if the initial troubleshooting
does not resolve the problem. The LMC will attempt to identify the cause
of the problem and employ DAAC resources to resolve the problem using
procedures of Table 21.3-2, Hardware Corrective Maintenance Actions. If
unable to correct the problem using DAAC resources, the LMC arranges for
maintenance vendor on-site support in accordance with Section 21.3.4,
Contract On-Site hardware Support.

Table 21.3-2. Hardware Corrective Maintenance Actions (1 of 2)

+----+------------+---------------------------------------------------+
| St | Occurrence | Action                                            |
| ep |            |                                                   |
+----+------------+---------------------------------------------------+
| 1  | COTS       | a\. Open a trouble ticket                         |
|    | hardware   |                                                   |
|    | problem    | b\. Annotate trouble ticket with all actions      |
|    | discovered | taken and their results                           |
|    | by an      |                                                   |
|    | operator,  | c\. Indicate the organization you recommend to    |
|    | SA or NA.  | perform corrective action to find and fix the     |
|    | not        | problem (COTS SW is SEO, COTS HW is LMC.          |
|    | resolved   |                                                   |
|    | by initial | d\. Forward trouble ticket to the User Services   |
|    | trouble    | office                                            |
|    | -shooting. |                                                   |
+----+------------+---------------------------------------------------+
|    | COTS       | a\. Annotate trouble ticket with all actions      |
|    | hardware   | taken and their results                           |
|    | problem    |                                                   |
|    | forwarded  | b\. Forward trouble ticket to the organization    |
|    | to SA or   | you believe responsible for correcting the        |
|    | NA. by     | problem.                                          |
|    | User       |                                                   |
|    | Services   | > NOTE: Report software related problem to the    |
|    | not        | > SA/Sustaining Engineering Organization (SEO)    |
|    | resolved   | > (refer to Section 22, Software Maintenance) If  |
|    | by initial | > the problem is hardware related notify the LMC. |
|    | trouble    |                                                   |
|    | -shooting. |                                                   |
+----+------------+---------------------------------------------------+
| 2  | LMC        | a\. LMC reviews the trouble ticket.               |
|    | attempts   |                                                   |
|    | to         | b\. LMC opens a MWO and annotates actions and     |
|    | identify   | results to date.                                  |
|    | cause of   |                                                   |
|    | problem.   | c\. LMC enters the unique identification number   |
|    |            | of the trouble ticket into the MWO.               |
|    |            |                                                   |
|    |            | d\. LMC enters the unique identification number   |
|    |            | of the MWO into the trouble ticket.               |
|    |            |                                                   |
|    |            | e\. LMC contacts SA or NA to assist in additional |
|    |            | troubleshooting, as necessary.                    |
|    |            |                                                   |
|    |            | f\. LMC, SA, and/or NA accomplish the following:  |
|    |            |                                                   |
|    |            | > 1\) Performs initial troubleshooting, including |
|    |            | > that described in the COTS hardware manuals.    |
|    |            | >                                                 |
|    |            | > 2\) Record all actions and results into the     |
|    |            | > MWO.                                            |
+----+------------+---------------------------------------------------+

Table 21.3-2. Hardware Corrective Maintenance Actions (2 of 2)

+----+------------+---------------------------------------------------+
| St | Occurrence | Action                                            |
| ep |            |                                                   |
+----+------------+---------------------------------------------------+
| 3  | Problem    | a\. If no hardware replaced.                      |
|    | resolved   |                                                   |
|    | by local   | > 1\) Correct problem, verify resolution.         |
|    | staff.     | >                                                 |
|    |            | > 2\) Close MWO, annotating actions taken.        |
|    |            |                                                   |
|    |            | b\. hardware replaced with maintenance spare      |
|    |            |                                                   |
|    |            | > 1\) Failed LRU is replaced.                     |
|    |            | >                                                 |
|    |            | > a\) Old part number recorded in MWO             |
|    |            | >                                                 |
|    |            | > b\) Old serial number recorded in MWO           |
|    |            | >                                                 |
|    |            | > c\) Old model/version recorded in MWO           |
|    |            | >                                                 |
|    |            | > d\) New part number recorded in MWO             |
|    |            | >                                                 |
|    |            | > e\) New serial number recorded in MWO           |
|    |            | >                                                 |
|    |            | > f\) New model/version recorded in MWO           |
|    |            | >                                                 |
|    |            | > g\) Down time recorded in clock hours in MWO    |
|    |            | >                                                 |
|    |            | > h\) Delay time identified by reason and         |
|    |            | > recorded by clock hours in MWO.                 |
|    |            | >                                                 |
|    |            | > 2\) CM requirements are accomplished following  |
|    |            | > procedures in Section 9.                        |
|    |            | >                                                 |
|    |            | > 3\) Failed LRU replacement ordered in           |
|    |            | > accordance with Section 21.4.1.                 |
|    |            | >                                                 |
|    |            | > 4\) Failed LRU routed in accordance with        |
|    |            | > Sections 21.3.5 or 21.4.3.                      |
|    |            |                                                   |
|    |            | c\. LMC closes MWO noting actions taken to        |
|    |            | resolve the problem                               |
|    |            |                                                   |
|    |            | d\. LMC forwards completed trouble ticket and MWO |
|    |            | to the Configuration Administrator.               |
+----+------------+---------------------------------------------------+
| 4  | Hardware   | a\. SA/NA notifies the LMC that hardware problem  |
|    | problem is | remains open. Maintenance contract support is     |
|    | not        | needed.                                           |
|    | resolved   |                                                   |
|    | by staff.  |                                                   |
+----+------------+---------------------------------------------------+

### 21.3.4 Contract On-Site Hardware Support

The LMC will notify the applicable maintenance contractor and request
assistance in diagnosing the problem or dispatch of a maintenance
engineer to the site if local DAAC resources can not resolve the
problem. The call for support will be documented in the MWO by the LMC,
noting the date and time the contractor was called. Table 21.3-3
identifies the steps associated with obtaining contract maintenance
support. It is important that all vendor maintenance activities and
start/stop times associated with them are recorded in ILMÕs MWO. This is
the principal means of monitoring, measuring, and managing the
maintenance vendorÕs contractual performance in support of the ECS
system downtime goals. Data fields have been specifically created in
ILMÕs MWO to capture this information.

Table 21.3-3. Obtaining On-Site Hardware Support (1 of 2)

+----+-------------+--------------------------------------------------+
| St | Occurrence  | Action                                           |
| ep |             |                                                  |
+----+-------------+--------------------------------------------------+
| 1  | Local       | a\. LMC gathers information needed to obtain     |
|    | support     | contract maintenance support.                    |
|    | effort did  |                                                  |
|    | not resolve | > 1\) Make, model, serial number, and location   |
|    | the         | > of failed systems.                             |
|    | problem.    | >                                                |
|    |             | > 2\) Description of problem and symptoms (from  |
|    |             | > trouble ticket).                               |
|    |             | >                                                |
|    |             | > 3\) Criticality of the COTS hardware           |
|    |             | > experiencing the problem.                      |
|    |             |                                                  |
|    |             | b\. Using the ILS web page at                    |
|    |             | http://edhs1.gsfc.nasa.gov/ the LMC determines:  |
|    |             |                                                  |
|    |             | > 1\) Name and telephone number of maintenance   |
|    |             | > provider                                       |
|    |             | >                                                |
|    |             | > 2\) Access code needed to obtain support       |
|    |             | >                                                |
|    |             | > 3\) Telephone number of the support providerÕs |
|    |             | > technical support center.                      |
|    |             | >                                                |
|    |             | > 4\) Site authorized contact person             |
|    |             |                                                  |
|    |             | c\. LMC records the information in a and b,      |
|    |             | above, into the MWO.                             |
+----+-------------+--------------------------------------------------+
| 2  | LMC calls   | a\. Provides information from Step 1 above to    |
|    | the         | the support provider to establish a need for     |
|    | appropriate | on-site support.                                 |
|    | support     |                                                  |
|    | providerÕs  | b\. Obtains the reference case number from the   |
|    | technical   | COTS hardware support provider.                  |
|    | support     |                                                  |
|    | center to   | c\. Updates the MWO to reflect time and date of  |
|    | obtain      | the call and                                     |
|    | on-site     |                                                  |
|    | assistance. | > case reference number.                         |
|    |             |                                                  |
|    |             | d\. Notifies problem originator that contractor  |
|    |             | is on the way.                                   |
+----+-------------+--------------------------------------------------+
| 3  | Maintenance | a\. LMC arranges for site access using local     |
|    | technician  | established procedures. Records arrival time.    |
|    | arrives at  |                                                  |
|    | the site.   | b\. LMC escorts maintenance technician to the    |
|    |             | hardware.                                        |
|    |             |                                                  |
|    |             | c\. LMC assists the maintenance technician in    |
|    |             | resolving the problem. This includes:            |
|    |             |                                                  |
|    |             | > 1\) Arranging for the equipment to be shut     |
|    |             | > down.                                          |
|    |             | >                                                |
|    |             | > 2\) Demonstrating the problem.                 |
|    |             |                                                  |
|    |             | 3\) Obtaining site available technical           |
|    |             | references, when needed.                         |
+----+-------------+--------------------------------------------------+

Table 21.3-3. Obtaining On-Site Hardware Support (2 of 2)

+----+-------------+--------------------------------------------------+
| St | Occurrence  | Action                                           |
| ep |             |                                                  |
+----+-------------+--------------------------------------------------+
| 4  | Maintenance | a\. If a part is replaced, LMC:                  |
|    | technician  |                                                  |
|    | corrects    | > 1\) Obtains from the failed part:              |
|    | the         | >                                                |
|    | problem.    | > a\) serial number                              |
|    |             | >                                                |
|    |             | > b\) Equipment identification number (EIN) (the |
|    |             | > number on the silver label)                    |
|    |             | >                                                |
|    |             | > c\) model/version                              |
|    |             | >                                                |
|    |             | > 2\) Obtains from the new part:                 |
|    |             | >                                                |
|    |             | > a\) part number                                |
|    |             | >                                                |
|    |             | > b\) serial number                              |
|    |             | >                                                |
|    |             | > c\) manufacturerÕs model number (if different  |
|    |             | > from part removed, a configuration change      |
|    |             | > request \[CCR\] is required for CM)            |
|    |             | >                                                |
|    |             | > 3\) Updates the MWO with following             |
|    |             | > information:                                   |
|    |             | >                                                |
|    |             | > a\) actions taken to correct the problem.      |
|    |             | >                                                |
|    |             | > b\) part number of the new item.               |
|    |             | >                                                |
|    |             | > c\) serial number of the old and new item.     |
|    |             | >                                                |
|    |             | > d\) Model/version of the old and new item      |
|    |             | >                                                |
|    |             | > e\) EIN of the old part, (if applicable)       |
|    |             | >                                                |
|    |             | > f\) EIN assigned to the new item, (if          |
|    |             | > applicable)                                    |
|    |             | >                                                |
|    |             | > g\) replacement LRUÕs model number             |
|    |             | >                                                |
|    |             | > h\) name of the item replaced.                 |
|    |             | >                                                |
|    |             | > I\) time and date repair started               |
|    |             | >                                                |
|    |             | > j\) time and day repair completed repair       |
|    |             | >                                                |
|    |             | > k\) any delay time experienced in completing   |
|    |             | > the repairs and reason for delay.              |
|    |             |                                                  |
|    |             | b\. If no parts were replaced the LMC updates    |
|    |             | the MWO with:                                    |
|    |             |                                                  |
|    |             | > 1\) Actions taken to correct the problem.      |
|    |             | >                                                |
|    |             | > 2\) Time and date repair started               |
|    |             | >                                                |
|    |             | > 3\) Time and date repair completed             |
|    |             | >                                                |
|    |             | > 4\) Delay time experienced in completing the   |
|    |             | > repairs:                                       |
|    |             | >                                                |
|    |             | > a\) start and stop times                       |
|    |             | >                                                |
|    |             | > c\) reason for each delay                      |
|    |             |                                                  |
|    |             | c\. Forward the completed MWO and trouble ticket |
|    |             | to the Configuration Administrator               |
+----+-------------+--------------------------------------------------+

### 21.3.5 Return-to-Depot Support

In some cases (e.g., X terminals) on-site support is not provided by the
OEM. Instead, a return-to-depot support is provided whereby an advance
replacement LRU is requested from the vendor by the LMC prior to
returning the failed repair. The vendor ships the requested part and
provides a return materials authorization (RMA) number for the return of
the failed LRU. When the replacement unit arrives, it is placed in
spares inventory using the procedures in this document in Section 23,
Property Administration, if an on-site spare was used to replace the
failed unit. Otherwise, the new LRU is installed in the equipment. The
failed unit is returned to the vendor using procedures in Section
21.4.3.

## 21.4 Maintenance Spares

Replacement LRUs are provided by the organization performing the
maintenance support (i.e., the OEM or vendor for warranty maintenance,
the COTS hardware support provider for post-warranty maintenance
support, or the ECS Project for self-maintenance support by local DAAC
M&O staff.) Replacement LRUs provided by the OEM/vendor/COTS hardware
maintenance support provider will typically be obtained from within the
metropolitan area where the DAAC is located, and will seldom be stocked
on the DAAC site.

The ECS ILS manager may elect to procure selected maintenance spares to
provide a more rapid return to service for failed critical units. These
spares may be centrally stocked at GSFC, stored on-site in the DAAC
property room, or as installed spares in equipment. Maintenance spares
will be identified by the ILS maintenance coordinator in document
618-CD-002-001, Release B Replacement Parts List and Spare Parts List.
Maintenance spares are procured, replenished, and managed by the ILS
Office using the process identified in Paragraphs 4.6.3 and 4.6.4 of
Release b COTS Maintenance Plan, document 613-CD-003-001; Section 23,
Property Administration; and appropriate local DAAC policies and
procedures.

### 21.4.1 Installed Maintenance Spares

Some COTS hardware items have extra LRUs installed to provide
hot-swappable spares or excess capability so that a failure does not
bring the system down or the unit can be made operational expeditiously.
Such spares include RAID disks, power supplies, network cards and tape
drives.

When a maintenance spare fails, the LMC, after returning the system to
an operational status, contacts the appropriate hardware support
provider, vendor, or OEM, as appropriate, and arranges for an advance
replacement to be shipped to the DAAC. The LMC also obtains an RMA
number for the return of the failed LRU. The LMC uses the information
from the ILS web siteÕs COTS Hardware Support spreadsheet to determine
the appropriate maintenance support provider to contact. The advanced
replacement part from the maintenance support provider is air-shipped to
the DAAC. The LMC will return the failed item to the location specified
on the RMA using the carton in which the advanced replacement part was
received. The RMA number will be prominently displayed on the carton and
on the failed item. The receipt, installation, and return of the LRU is
accomplished using procedures of this document found in Section 8,
Problem Management; Section 9, Configuration Management; Section 23,
Property Management; guidance found in document 609, Version2.0
Operations Tools; and local procedures.

### 21.4.2 Use of Maintenance Spares

The LMC will control the use of on-site maintenance spares. Centrally
stocked spares can be requested from the ILS coordinator using
procedures in Section 21.1. Replacement or repair of maintenance spares
is performed by the COTS hardware maintenance provider using procedures
in Section 21.3.

### 21.4.3 Return of Failed LRUs

During the warranty period and subsequent maintenance contract, the
failed unit belongs to the support provider who is responsible for
repairing or replacing the failed LRU. The LMC is responsible for the
return of the failed LRUs if either site spares or advanced replacement
LRUs (e.g., systems under return-to-depot support) are used to replace
the failed LRU. After a failed LRU that is to be returned by the LMC is
removed it will be tagged and processed for return to a maintenance
facility for repair or replacement. The LMC will use the RMA number
provided with the advance replacement LRU or request an RMA number from
the appropriate site spare provider if a site spare was used. The failed
unit is returned to the vendor (unless the vendor specifies another
location) using the same carton in which the replacement unit was
shipped. The RMA number will be prominently displayed on the outside of
the return carton and on the tag attached to the failed item. The LMC
will ensure that replacement of LRUs is documented in the trouble
ticket.

## 21.5 Non-standard Hardware Support

Non-standard COTS hardware support consists of maintenance support
outside the PPM (support incurring time and materials charges) or
escalated support actions by the maintenance support provider.

### 21.5.1 Escalation of COTS Hardware Support Problem

Hardware support providers have escalation policies based on elapsed
time from start of the corrective effort. The escalation policies direct
increased management attention and/or resources to the problem, which is
relatively invisible to the DAACs. Escalation of a hardware support
providerÕs efforts may also be requested anytime the corrective effort
is not progressing satisfactorily. The LMC may request escalation by
calling the maintenance contractorÕs technical support center and
providing the case number generated when the problem was first reported.
The LMC can also contact the ILS maintenance coordinator for assistance
in obtaining a satisfactory resolution.

### 21.5.2 Time and Material (T&M) Hardware Support

T&M support is expensive and generally exceeds \$150/hour, two-hour
minimum. It is to be used only as a last resort for mission critical
repairs. The LMC will obtain authorization from the ILS maintenance
coordinator or the System Monitoring and Coordination Center (SMC) in
the ILS maintenance coordinatorÕs absence, for T&M support. Once T&M
support is approved, the ILS maintenance coordinator (or LMC if the ILS
maintenance coordinator is not available) calls the appropriate COTS
hardware providerÕs technical support center and requests T&M support.
The LMC will verify the time of arrival and departure of T&M
technicians; delay time; travel times; replaced LRU part number, serial
number, name; and total dollar charge. This information and a legible
photo copy of the completed T&M work order are to be sent to the ILS
maintenance coordinator within one business day.

# 22. Software Maintenance

## 22.1 Introduction

The ECS organization provides maintenance and operations for ECS
hardware, software, and firmware systems delivered under the ECS
contract to the ECS sites. The functions performed by each of the M&O
organizations are described in the M&O Management Plan, CDRL
601-CD-001-001.

In general, ECS organizations procure, produce, deliver, and document
the modifications, and enhancements made to ECS software, firmware, and
hardware. No custom firmware has been identified as part of the ECS
program. Commercial Off-The-Shelf software (COTS SW), firmware, and
hardware will be maintained in accordance with the COTS Maintenance
Plan, CDRL 613-CD-001-001. The Project maintenance philosophy for
software is to provide ECS centralized support for developed items and
vendor support for COTS SW.

Specific software support procedures are discussed in this section. ECS
Project software consists of COTS, custom-developed, and science
software. Science software, developed for use on the ECS project, is the
responsibility of the scientific community (see section 22.3). Staffing
requirements are discussed in the [ECS Operations Plan,]{.underline}
CDRL 608-CD-001-002. Training is discussed in the [ECS Training
Plan]{.underline} , CDRL 622-CD-001-002. Specific schedules of
programmed activities are contained in the [Intermediate Logic Network
Diagrams]{.underline}, CDRL 108-CD-tbd.

Software maintenance includes:

¥ COTS support contract with the software vendor for license to use;
telephone assistance in resolving COTS SW problems, as well as obtaining
patches and upgrades.

¥ Resources, including equipment, software tools and personnel to
maintain ECS in accordance with specified functional performance, and
availability requirements.

¥ Services required to produce, deliver, integrate, install, test,
validate and document modifications of existing ECS software and
firmware. The maintenance activity includes: software configuration
management (CM) including support for change control, configuration
status accounting, audit activities, and software quality assurance
(QA). Each site is the CM authority over its own resources subject to
ESDIS delegation of roles for ECS management.

The siteÕs Local Maintenance Coordinator, System Administrator, Network
Administrator, and COTS Software License Administrator handle COTS
software problem administration, issues concerning upgrade installation,
troubleshooting, and vendor liaison. The ECS Sustaining Engineering
Office (SEO) provides assistance when COTS software issues exceed the
capabilities of the site System Administrator or the Network
Administrator to resolve. M&O tasks for COTS software support are
described in Section 22.1, based on 613-CD-003-001, ÒRelease B COTS
Maintenance Plan for the ECS ProjectÓ and 194-602-OP1-001, "Property
Management Plan for the ECS Project."

Custom software is maintained by the SEO, FOS, M&O, and ECS developers.
The SEO Software Librarian acts as the configuration manager for the M&O
master software library (ECS operational baseline) and distributes
copies of this library to SEO who in turn, provides both a distributed
backup and local availability of the library. The SEO Configuration
Management Administrator, SEO Software Maintenance Engineer, and the
ECS-site (DAAC, EOC, and SMC CM Administrators and Software Maintenance
Engineers) implement changes to the ECS baseline. M&O tasks for custom
software maintenance are described in Section 22.2, based on
614-CD-001-003, "Developed Software Maintenance Plan"; 102-CD-001-002,
"M&O Configuration Management Plan"; ECS#301-CD-003-001, "System
Implementation Plan for ECS Turnovers"; 605-CD-001-003, "Operations
Scenarios for the ECS Project: Release A"; and 194-602-OP1-001,
"Property Management Plan for the ECS Project."

### 22.1.1 COTS Software Maintenance

Installation of patches, upgrades and software problem isolation is
accomplished by M&O personnel at the site. COTS software procured for
the ECS contract is supported by the COTS software vendor through a
support contract. (NB: The term software vendor refers to the company
having the legal right to authorize software use and to modify the
software code.) COTS software vendor support consists of telephone
support for resolution of usage and interface problems, access to an
on-line solution database, providing upgrades and patches, escalation of
support, and resolving COTS software code problems.

The Activity Checklist table that follows provides an overview of COTS
Software Support procedures. Column one (Order) shows the order in which
tasks might be accomplished. Column two (**Role**) lists the
Role/Manager/Operator responsible for performing the task. Column three
(**Task**) provides a brief explanation of the task. Column four
(**Section)** provides the Procedure (P) section number or Instruction
(I) section number where details for performing the task can be found.
Column five (**Complete?**) is used as a checklist to keep track of
which task steps have been completed.

Table 22.1-1. COTS Maintenance - Activity Checklist

+--------+-----------+----------------------------+---------+---------+
| Order  | Role      | Task                       | Section | Co      |
|        |           |                            |         | mplete? |
+--------+-----------+----------------------------+---------+---------+
| 1      | Site      | Assist System              | \(I\)   |         |
|        | Local     | Administrator in obtaining | 22      |         |
|        | Ma        | COTS SW support            |         |         |
|        | intenance |                            |         |         |
|        | Co        |                            |         |         |
|        | ordinator |                            |         |         |
+--------+-----------+----------------------------+---------+---------+
| 2      | COTS      | Manage COTS Software       | \(I\)   |         |
|        | Software  | Maintenance Contracts      | 22.1.3  |         |
|        | License   |                            |         |         |
|        | Admi      |                            |         |         |
|        | nistrator |                            |         |         |
+--------+-----------+----------------------------+---------+---------+
| 3      | COTS      | Manage Software Licenses   | \(I\)   |         |
|        | Software  |                            | 22.1.3  |         |
|        | License   |                            |         |         |
|        | Admi      |                            |         |         |
|        | nistrator |                            |         |         |
+--------+-----------+----------------------------+---------+---------+
| 4      | SEO       | Distribute Software        | (I      |         |
|        | Librarian |                            | )22.1.4 |         |
+--------+-----------+----------------------------+---------+---------+
| 5      | System    | Install and troubleshoot   | \(I\)   |         |
|        | /Network  | COTS Software              | 22.1.5  |         |
|        | Admi      |                            |         |         |
|        | nistrator |                            |         |         |
+--------+-----------+----------------------------+---------+---------+
| 6      | S         | Interface with CCB         | \(I\)   |         |
|        | ustaining |                            | 22.1.4  |         |
|        | En        | (SW License Admin. may go  | -22.1.5 |         |
|        | gineering | before CCB whenever there  |         |         |
|        | O         | is an upgrade in current   |         |         |
|        | ffice/(SW | software version, new      |         |         |
|        | License   | patches, or a need to      |         |         |
|        | Admin.)   | purchase additional        |         |         |
|        |           | software licenses in order |         |         |
|        |           | to satisfy project         |         |         |
|        |           | requirements. SW License   |         |         |
|        |           | Admin. should be informed  |         |         |
|        |           | and involved in any        |         |         |
|        |           | overall change to the      |         |         |
|        |           | baseline.)                 |         |         |
+--------+-----------+----------------------------+---------+---------+
| 7      | A         | Obtain COTS Software       | \(P\)   |         |
|        | uthorized | Support                    | 22.1.6  |         |
|        | Site      |                            |         |         |
|        | Personnel |                            |         |         |
+--------+-----------+----------------------------+---------+---------+

### 22.1.2 COTS SW Maintenance Survey Results

A COTS Maintenance Survey of ECS software has been undertaken to
determine the proper approach to maintenance issues related to the
following:

¥ Existing maintenance agreements

¥ SW dependencies (compatible versions),

¥ Freeware,

¥ Expert point of contact (Responsible Engineer) information,

¥ Where deployed (SMC, DAAC, EDF, EOC)

¥ Release schedule, and

¥ OEM, vendor, terms and conditions, and maintenance costs.

One major objective was to conduct a survey on the process of purchasing
future maintenance for COTS SW products. Specifically, what is the
nature of customization for each product in order to judge the
feasibility of continuing vendor maintenance; are there any other
options available; what would be the value of pursuing agreements from
the vendors; and ensuring an informed maintenance approach for each COTS
product. The results are being maintained in a matrix index of
information related to each product such as level of customization, code
ownership, complexity factor related to number/ type of interfaces,
maintenance approach, etc. This survey should be made available to the
EDF for future purchasing and to the Sustaining Engineering Office (SEO)
for maintenance planning. The results are closely held due to
proprietary rights and volatility of the data.

A complexity factor for upgrade activities was considered and assigned
as low (L), medium (M), or high (H) to indicate the degree of required
knowledge of the product to install an upgrade at the site. It was noted
that in some cases, such as for the BMC Optima SNMP toolkit, a low-level
of knowledge was indicated, even though at some point in the update a
high degree of knowledge is required. An upgrade of this COTS SW would
require a recompile of all ECS custom code, but the upgrade at the site
would be a fairly simple reinstall of the custom code (no
reconfiguration required). This corresponding upgrade method information
poses serious concerns for our maintenance approach, i.e., central vs.
distributed; expert vs. novice installer; regression testing strategy,
etc. We have surveyed appropriate engineers to determine the best
approach and understand the myriad problems to be avoided.

Freeware presents special problems for maintenance consideration. As an
example, we have had a request to upgrade PERL, which is freeware that
is available from several convenient places on the internet. PERL is
extensively used in the Pre-Release B.0 Testbed for scripting. It would
easily become problematic for each DAAC to download any freeware version
they choose and thus induce incompatibilities. The SEO will provide a
single point of contact for sourcing this freeware. We will also
consider companies who will package freeware on media with appropriate
documentation and possible support. Whether sourced from a vendor or
centrally collected, having a single point of control will allow us to
control versions of freeware in use, avoid incompatibilities, and
perform consistent regression testing and verification activities. At a
minimum, the SEO with proper technical consultation of developers,
operators, and users will look at freeware and determine the proper
source for it. The maintenance problems can then be minimized to a great
degree by agreement on a single sourcing point for freeware. The SEO
will then track these source points for ongoing maintenance and
upgrades.

We will not field any SW which does not have an identifiable source. On
freeware, we can become that source and thus police the version which is
deployed. It will be up to the ESDIS CCB to determine the use of SW
add-ons which will be used that cannot be proven to have a deleterious
effect on ECS. We will keep track of DAAC-unique add-ons and determine
which ones we will not support in subsequent releases. A free-flow of
information is essential to safe-guard ECS assets from the effects of SW
add-ons at the DAACs. Thus, such information will be maintained under
configuration management control. Where determined necessary, developers
may bundle freeware with custom code products in ECS SW releases.

The survey results have implications for allocation of staffing to SEO
SW maintenance and on-site support regarding specific expertise
required, number of billets, locations of support elements, and
deployment of ECS logistics support components (training, documentation,
maintenance release deployment strategy, test and support, technical
data, and computer resources).

### 22.1.3 Management of COTS Software Maintenance Contracts

COTS software vendor support is contracted by the ECS procurement office
at the EDF. After the first year of warranty support, support is
contracted for a period of one or more years and extended or modified as
operationally required. Information related to COTS software support
contracts is maintained in a database used by the COTS Software License
Administrator to monitor/track the expiration dates and contract terms.

For Version 2, the COTS SW License Administrator will track software
licenses via the ILM tool. . As a COTS SW vendor support requirement
nears its expiration date, the COTS SW License Administrator determines
through consultation with SEO, the need for continued COTS software
support. Once a determination is made to continue COTS software support,
the COTS SW License Administrator coordinates with the ECS procurement
office for extension/modification of the support contract. Requested
changes to COTS SW support contracts should be provided by the site
System Administrator to the COTS SW License Administrator. The COTS SW
License Administrator may be contacted by email at ilsmaint@eos.hitc.com
and by dialing 1-800-ECS-DATA, Option #3, then dial extension 0726.

### 22.1.4 Management of COTS Software Licenses 

Functions of the COTS SW License Administrator include the following:

1\. Maintain accountability for all COTS SW licenses procured for the
ECS contract. Accountability includes tracking and reporting the
as-installed location of all licenses procured for the ECS contract.
This information will be generated from the findings of the software
physical configuration audit. Once the software configuration audit has
been performed, the software licenses will be tracked by monitoring the
status of COTS SW CCRs as they are implemented and through configuration
data maintained in Inventory and Logistics Manager (ILM), ECS' inventory
database.

2\. Assist the SEO organization in impact analysis of proposed COTS SW
upgrades and patches on other COTS SW applications incorporated in the
ECS system design

3\. Maintain a database containing license keys of all project-purchased
COTS SW. The COTS SW installation team will provide the machine
identifications to the COTS SW License Administrator, who shall then
obtain the necessary license keys from vendors for SW installation and
populating the COTS SW database.

4\. Provide SW upgrades, with vendor-provided release notes and version
description documents, to SEO for testing and distribution.

5\. Keep SEO and all other ECS sites informed as to how they may access
vendor patch libraries for use in resolution of software problems.

COTS software licenses vary by the type of software and the software
vendors' policies. COTS software license types include: floating, per
site, specific number of concurrent users, unlimited users, and lifetime
use without regard to number of users or location. The quantity and type
of COTS software licenses initially required are identified to the ECS
procurement office by ECS design engineers. COTS software licenses are
received and entered into the ILM database by the COTS SW License
Administrator. The COTS SW License Administrator maintains the master
copy of COTS SW license agreements, as well as a COTS software license
database.

### 22.1.5 COTS Software Installation

The COTS software upgrades are subject to ECS CCB approval before they
may be loaded on any platform. The COTS SW License Administrator, using
procedures contained in Section 9, ÒConfiguration Management,Ó notifies
the SEO organization of the upgrades that have been received. The COTS
SW License Administrator distributes the COTS software upgrade as
directed by the CCB. The site Software Maintenance Engineer, Network
Administrator, and the System Administrator are responsible for
upgrading the software on the host machine and providing follow-up
information to the Configuration Management Administrator (CMA) and the
COTS SW License Administrator. The site Local Maintenance Coordinator
will notify the appropriate personnel (Release Installation Team, System
Administrator, Network Administrator, Software Maintenance Engineer)
when the COTS software is received.

COTS software patches may be provided by the COTS software vendor in
response to a DAACÕs call requesting assistance in resolving a COTS
software problem. The problem may or may not exist at other locations.
When a COTS software patch is received directly from a COTS software
vendor (this includes downloading the patch from an on-line source), the
DAACÕs CCB will be informed via CCR prepared by the requesting Operator,
System Administrator, Network Administrator, or site Software
Maintenance Engineer. It is the responsibility of the Operator, System
Administrator, Network Administrator, or site Software Maintenance
Engineer to notify the CCB of the patchÕs receipt, purpose, and
installation status, using procedures contained in Section 9,
ÒConfiguration Management,Ó and to comply with the CCB decisions. The
Operator, System Administrator, Network Administrator, or site Software
Maintenance Engineer installs COTS SW patches as directed by the CCB.

In addition to providing patches to resolve problems at a particular
site, the software vendor will periodically provide upgrades of COTS
software in order to improve the product. Such upgrades are issued are
issued to all licenses covered by a software maintenance contract.
Therefore, the COTS software upgrades will be shipped to the COTS
Software License Administrator, who receives and enters them into
inventory. When there is a desire to upgrade to a more current version
of the software, a CCR must be submitted for approval by the ESDIS CCB.
Once the upgrade is approved and installed, and the CCR closed, the CM
Administrator and COTS SW License Administrator are notified of the
configuration change for updating of ECS records.

### 22.1.6 Obtaining COTS Software Support

COTS SW support involves both site capability and contracted support.
Site capability is provided by the System Administrator, Network
Administrator, and site Software Maintenance Engineer. Contracted
support is provided by the COTS SW vendor. When the System
Administrator, Network Administrator, or site Software Maintenance
Engineer confirm that a problem is attributed to the COTS SW, the COTS
SW vendorÕs technical support center is contacted by authorized
personnel at the site.

The software vendorÕs technical support center will verify contract
support authorization and then assist in pinpointing the COTS SW problem
to provide a recommended solution. The solution may comprise of a patch,
work-around, or include the fix in a future release. If a patch exists
to correct the problem, the patch will be identified and provided by the
software vendor over the internet or mailed to the requester. If a patch
is required but not available, the site and vendor together determine
the seriousness of the problem. If the problem is critical, a temporary
patch or work-around may be provided, if non-critical, the solution to
the software problem may be scheduled by the software vendor to be
incorporated in a future update or release. (NB: The DAAC and ESDIS CCBs
must authorize the patch to be installed as a permanent installation.
This decision may be made after-the-fact. That is, if the patch is
needed in order to proceed with operations, notify the CCB of the
requirement in accordance with Section 9, ÒConfiguration Management.Ó
Applicable requirements of Section 8.1, ÒTrouble Ticket System
Procedures,Ó must be followed.)

The COTS Software License Administrator obtains the support
authorization codes from the vendors and arranges for specified
personnel to become an authorized contact person, based upon the
limitations imposed by the vendor, and the needs of individual DAACS.
The software vendorÕs technical support telephone numbers, the names of
personnel authorized (by site and software) to contact the vendor, and
the authorization/access codes will be provided to the siteÕs Local
Maintenance Coordinator by the COTS Software License Administrator
through the M&O web site entitled "COTS Hardware- Software Maintenance".
This web site can be accessed via the EDHS 1 Server, as a subsection of
the Maintenance and Operations segment. The web address to the EDHS 1
Server is (http://edhs1.gsfc.nasa.gov/). Once the EDHS Server has been
accessed, click on the yellow section labeled "Maintenance and
Operations". This leads to a lists which details all information
available on the Maintenance and Operations web page, then select "COTS
Hardware-Software Maintenance". Once the maintenance web presentation
has materialized, click on the purple highlighted section entitled "DAAC
Site Selection". Select the appropriate DAAC location, click on the
software option. A software matrix will appear. In order to receive
detailed vendor information, select from vendor names on the left side
of the page. The software matrix fields include the following:

<table>
<colgroup>
<col style="width: 6%" />
<col style="width: 9%" />
<col style="width: 7%" />
<col style="width: 9%" />
<col style="width: 7%" />
<col style="width: 8%" />
<col style="width: 8%" />
<col style="width: 15%" />
<col style="width: 15%" />
<col style="width: 12%" />
</colgroup>
<tbody>
<tr class="odd">
<td>SW Mfgr</td>
<td>SW Product</td>
<td>Sites Using SW</td>
<td>Tech. Support #</td>
<td>Fax #</td>
<td>Tele. Spt. Hrs.</td>
<td>Auth. Callers</td>
<td>Info. Needed for obtaining Spt.</td>
<td><p>Response Time/</p>
<p>Commitment</p></td>
<td>Internet Address</td>
</tr>
</tbody>
</table>

Changes to the information in the ÒCOTS Software SupportÓ web site are
to be provided to the COTS Software License Administrator as they occur,
for updating the web site. Specifically, the need to identify or replace
the authorized contact person, must be provided by the site's Local
Maintenance Coordinator to the COTS Software License Administrator.
E-mail is the preferred notification method. The Local Maintenance
Coordinator will follow these steps:

1\. Send e-mail to ilsmaint@eos.hitc.com.

2\. If e-mail is not available, call 1-800-ECS-DATA, Option 3; then dial
extension 0728.

3\. Identify the change as either a permanent or temporary change. A
temporary change may occur when the authorized contact person is ill, on
vacation, in training, or other short-term change of work availability
status has occurred or is expected to occur.

4\. Provide the COTS Software License Administrator the change
information as soon as it is known.

### 22.1.7 COTS Software Problem Reporting

The first person experiencing or observing a potential COTS SW problem
(an Operator, System Administrator, Network Administrator, or the site
Software Maintenance Engineer) will initiate a trouble ticket according
to the procedures found in Section 8.1, ÒTrouble Ticket System
Procedures.Ó The site user service desk forwards the trouble ticket to
the System Administrator, Network Administrator, or the site Software
Maintenance Engineer as appropriate and in accordance with Subsection
8.1.4.

### 22.1.8 Troubleshooting COTS Software

The initial troubleshooting/diagnostics procedures are accomplished by
the siteÕs engineering staff. The Operator, System Administrator,
Network Administrator, or the site Software Maintenance Engineer will
attempt to isolate the source of the problem to hardware, the network,
COTS, custom-developed, or science software.

If it is confirmed to be a COTS SW problem, the System Administrator,
Network Administrator, or the site Software Maintenance Engineer
contacts the vendor's technical support center for assistance.
Information on contacting the software vendorÕs technical support center
is in Section 22.1.4.1, ÒCOTS Software Problem Reporting.Ó

One avenue to troubleshooting the COTS SW problem is to scan the
software vendorÕs web siteÕs solutions database to learn of any
solutions for similar problems. The software vendorÕs web site address
can be obtained as stated in Section 22.1.6, above. Another avenue to
troubleshooting the COTS SW problem is to exercise any software
diagnostic routine embedded or down-loadable that will determine the
status of the COTS SW on the equipment. As part of this effort, the
System Administrator, Network Administrator, or the site Software
Maintenance Engineer reviews the troubleshooting-diagnostics and
corrective actions taken to date. Additional troubleshooting/diagnostic
procedures and problem isolation techniques may be performed. These
troubleshooting, diagnostics, and/or isolation procedures/techniques may
be contained in the software/hardware vendorÕs operational/technical
manuals or in locally devised troubleshooting/diagnostic procedures.

COTS SW problems that cannot be corrected using site and contracted
software support may be escalated to the ECS SEO. The SEO is staffed
with Senior Systems Engineers knowledgeable on COTS SW and can assist in
diagnosing the problem. The site Local Maintenance Coordinator may go
directly to the software vendor or to the ILS Maintenance Coordinator to
obtain an escalation of software vendor support if the software vendorÕs
efforts have not produced satisfactory results within a reasonable
period of time. The escalation may result in increased vendor management
review of the problem resolution, the assignment of additional resources
to resolve the problem, and/or a more highly qualified technician
assigned to resolve the software problem.

### 22.1.9 Corrective Action Reporting

COTS software corrective action reporting follows the procedures
contained in Section 8, ÒProblem ManagementÓ and the configuration
control requirements contained in Section 9, ÒConfiguration Management,Ó
when a configuration item is removed and/or replaced with a different
version or release.

The appropriate site engineer (System Administrator, Network
Administrator, or site Software Maintenance Engineer) will record the
solution and results provided by the COTS software vendorÕs technical
support center on the trouble ticket. The site Local Maintenance
Coordinator verifies that trouble tickets contain accurate sequence of
events and are correctly entered into the trouble ticket management
system. The trouble ticketÕs sequence of events includes stating the
applications that were running and the commands given immediately prior
to the apparent COTS software problem. Include on the trouble ticket all
troubleshooting accomplished and any embedded/down-loaded diagnostics
software that were exercised. Also note on the trouble ticket, the
corrective actions resulting from the troubleshooting/diagnostics
efforts.

This page intentionally left blank.

# 23. Property Management

This section describes procedures for the receipt, control, and
accountability of ECS property at ECS sites. The ÒProperty Management
Plan for the ECS ProjectÓ is the base document that addresses the
process and policies regarding how ECS property is to be managed
throughout the Project. This section supplements the information
contained in that plan. Personnel assigned responsibility for site
property administration functions should be thoroughly familiar with the
contents and policies contained within that document.

Personnel designated as site Property Administrators are responsible for
the day-to-day receipt, inspection, storage, issue, inventory,
recording, accounting, and reporting of ECS property at ECS sites.
Property Administrators will follow the process and policies approved in
the Property Management Plan, procedures contained herein, and
ECS-approved local procedures developed between site Property
Administrators and local facility property control offices. Locally
developed procedures will be forwarded to and reviewed by the ECS ILS
Manager to ensure that they are in conformance with provisions of the
ECS Property Management Plan and are conducive to the maintenance of
control and accountability of ECS property at the site.

The Activity Checklist table that follows provides an overview of
Property Management. Column one (**Order**) shows the order in which
tasks might be accomplished. Column two (**Role**) lists the
Role/Manager/Operator responsible for performing the task. Column three
(**Task**) provides a brief explanation of the task. Column four
(**Section)** provides the Procedure (P) section number or Instruction
(I) section number where details for performing the task can be found.
Column five (**Complete?**) is used as a checklist to keep track of
which task steps have been completed.

Table 23.1-1. Property Management - Activity Checklist

  -----------------------------------------------------------------------------
  Order   Role            Task                           Section    Complete?
  ------- --------------- ------------------------------ ---------- -----------
  1       Property        Receive equipment and software \(I\) 23.1 
          Administrator                                             

  2       Property        Tag equipment                  \(I\) 23.2 
          Administrator                                             

  3       Property        Maintain site property records \(I\)      
          Administrator                                  23.3.1     

  4       Property        Compile and submit inventory   \(I\)      
          Administrator   reports                        23.3.2     

  5       Property        When necessary, report lost,   \(I\)      
          Administrator   damaged, or destroyed property 23.3.3     

  6       Property        When appropriate, submit       \(I\)      
          Administrator   information for Relief From    23.3.4     
                          Accountability in a timely                
                          manner                                    

  7       Property        Prepare and submit all reports \(I\) 23.4 
          Administrator   for equipment relocation                  

  8       Property        Perform scheduled inventories  \(I\) 23.5 
          Administrator   and audits                                

  9       Property        Ensure secure storage          \(I\) 23.6 
          Administrator   facilities                                

  10      Property        Ensure timely preparation of   \(I\) 23.7 
          Administrator   pre-shipping activities                   
  -----------------------------------------------------------------------------

## 23.1 Receipt of Equipment and Software

Upon receipt of ECS equipment the site Property Administrator will
perform a receiving inspection to verify correctness of delivery and
quantity received, and to determine if there is evidence of damage
during shipment. The Property Administrator will verify the number of
pieces received against the carrierÕs shipping document and the
condition of the material. On both the carrierÕs and siteÕs copy of the
shipping document, the receiving individual will note any damages or
shortages to the shipment and will sign the documents, thereby
acknowledging receipt of the shipment. The signature of the carrierÕs
representative will be obtained when container shortages, damages, and
other transit-related discrepancies are identified.

Property received will be promptly moved from the loading dock to a
controlled inventory area for unpacking, inspection, detailed component
level inventory, and preparation of a Receiving/ Inspection Record.
Items received will be reconciled against purchase orders, shipping
documents, packing lists, or related documents to ensure accountability
for all items, attachments, and accessories. Item identification,
configuration, and quantity will be matched against the vendorÕs packing
list, the purchase order, and/or the detailed configuration list (if
supplied by the Purchasing Office). Receiving/Inspection Records will
contain the following information:

-   Dates of receipt and inspection.

-   Names of shipper and carrier.

-   ShipperÕs bill of lading number and/or carrierÕs shipping document
    number.

-   Purchase order number or Return Maintenance Authorization (RMA)
    number (if indicated on shipping documents ).

-   List of items received (shipperÕs document or packing list can be
    referenced, if attached, and annotated to indicate receipt of
    items).

-   Serial numbers of serially numbered items.

-   Model/version numbers (shipperÕs document or packing list can be
    referenced if models/versions received are same as indicated on the
    shipperÕs document/packing list).

-   Equipment identification numbers (EIN) or equipment control numbers
    (ECN) of tagged items. \[Note: EINs are assigned to ECS-purchased
    reportable equipment and ECNs apply to government-furnished property
    (GFP).\]

-   Condition of shipment (note all damages to material).

-   Shipment discrepancies (include shortages, overages, and incorrect
    items/quantities/models).

The Receiving/Inspection Record will be forwarded electronically (i.e.
fax of internet) to the ECS Property Administrator within on day of
receipt of the shipment (fax \# 303-925-0438, internet
ecsprop@eos.hitc.com). The Receiving/Inspection Record will be attached
to the shipping document and retained in the siteÕs property files as a
permanent record of the property received. Such documents will be
retained until accountability is transferred to the Government.

The Receiving/Inspection Record is used by the ECS Property
Administrator to build the inventory records in the Inventory,
Logistics, and Maintenance (ILM) database. Site Receiving/Inspection
Records will be retained until accountability is transferred to the
Government. Site property records will be maintained by the Site
Property Administrator to reflect any changes to the original data
(i.e., maintenance failures/repairs, returns, location changes).

Received property will be secured and protected from electrostatic
discharge (ESD) during movement, storage, and while awaiting assembly
and issue. Access to the property storage areas will be controlled at
all times to prevent unauthorized entry.

## 23.2 Equipment Tagging

ECS equipment (e.g., contractor-acquired and GFP) that is separately
identifiable and/or meets the criteria for controlled equipment
(described in the Property Management Plan) will be tagged with ECS
property tags containing an ECS equipment identification number (EIN).
(Figure 23.2-1 illustrates ECS property tags.) ECS equipment shipped to
the EDF for staging and subsequent shipment to the sites is tagged by
the ECS Property Administrator. Tags will be supplied by the ECS
Property Administrator. ECS equipment shipped directly from the vendor
to the site will be tagged by the site Property Administrator
immediately upon receipt. Tags will be placed on the equipment in an
area so that they are visible and easily accessible by bar-code
scanners. Vendor-loaned and HITS capital equipment will not be tagged
with ECS property tags.

![](media/image1.wmf){width="6.0in" height="2.0694444444444446in"}

Figure 23.2-1. ECS Property Tags (actual size)

Components of major equipment that are not separately identifiable or
stocked for use as spares/repair parts for a higher assembly (i.e.,
workstations, servers, data storage devices and major communications
equipment) will not be assigned an EIN, but will be tagged with the
round non-reportable property tag (e.g., mice, keyboards, or other
devices that can be moved and have a value of less than \$1,000.00).
Such components will be controlled as inventory items and reported by
manufacturer, description, model/version, serial number (if assigned),
stock location, condition code, quantity, and, when installed as a
component of centrally reportable equipment, by next higher assembly.

Property tags of loaned GFP equipment containing a NASA equipment
control number will not be removed by the site Property Administrator.
At the time of receipt of such property, the Property Administrator will
affix an ECS property tag with EIN next to the government tag. The NASA
ECN will be recorded in the property record and cross referenced to the
EIN.

Prior to disposing of ECS equipment (or prior to returning it to the
Government), ECS property tags will be removed. Removed ECS property
tags will be retained with the property turn-in document by affixing the
tag to the document or to a blank sheet attached to the turn-in
document. Turn-in documents will be forwarded to the ECS Property
Administrator for retention in the accountable records.

## 23.3 Property Records and Reporting

The ECS Project uses the ILM property to support the property
management, control and reporting functions for Contractor-acquired and
GFP equipment. Property records are created and maintained in this
system by the ECS Property Administrator. At Version 2 this database
will be accessible to Site Property Administrators on a read-only basis.
However, Site Property Administrators will be able to do queries and to
prepare reports.

At a minimum, site property records in ILM will contain the following
fields:

-   Manufacturer.

-   Item description.

-   EIN \[for controlled equipment (see Section 23.2)\].

-   Parent EIN (for components of equipment having an EIN; this is the
    EIN of the next higher assembly).

-   Date received.

-   Location (e.g., GSFC, EDC).

-   Building Number.

-   Room Number.

-   Serial Number.

-   Model/Version Number.

-   Purchase Order Number.

Property records will contain a line for each item having an EIN (e.g.,
workstation, server, monitor, printer) and each of its major components
(e.g., network interface cards, RAM chips, graphics card, hard drive,
tape drive, CD ROM, SCSI card, operating system, COTS applications, RAID
drives). For an example of the level of detail, refer to the
Installation Receipt Report provided at the time of site installation.
It is the responsibility of the Site Property Administrator to notify
the ECS Property Administrator of inventory, configuration, and location
changes so that site property records can be maintained current

### 23.3.1 Maintaining Property Records

Site Property Administrators will update local property records within
one business day of property changes, including the following: receipt
and/or installation of new equipment, relocation's, dispositions, and
component configuration changes.

Support documentation used to post changes to property records will be
retained by the site Property Administrator to provide complete,
current, and auditable accountability data. Some of the documents to be
used for posting changes to property records include the following:

-   **Receiving/Inspection Record** \-- Prepared by the Site Property
    Administrator at the time equipment is received. It is used to
    record the physical condition of property received, quantities
    received, shipping data, date received, and the name of the persons
    who received and inspected the property.

-   **Installation Receipt Report** \-- This report is provided by the
    installation team at the time equipment is installed and is used to
    record receipt of equipment installed at the site. It can be used to
    update site property records with installed location, date, and name
    of the person accepting receipts. This report is signed by the Site
    Property Administrator to acknowledge receipt of new equipment.

-   **Inventory Report** \-- Prepared by the Site Property Administrator
    and used to report the results of 100 percent annual inventories
    required to be conducted of ECS equipment and GFP.

-   **Trouble Tickets** \-- Prepared by the site LMC and used to update
    property record with equipment changes resulting from maintenance or
    relocation actions (e.g., serial/model changes, component
    replacements, and relocation at the site).

-   **Disposition Record** \-- Prepared by the site Property
    Administrator and used to record the disposition of ECS equipment.
    Includes the identification of ECS equipment disposed of as a result
    of disposition actions approved by the NASA Property Administrator
    (i.e., DCMAO).

### 23.3.2 Property Reporting

The Site Property Administrator will submit the following reports:

-   **Inventory Change Report (ICR)** -- Forwarded within on business
    day of receipt/inspection of additional ECS property at the site or
    of inventory changes resulting from maintenance actions, relocation,
    or reconfigurations that will result in changes to the ECS property
    records.

-   **Annual Inventory Report** -- Forwarded upon completion of the
    annual property inventory conducted in accordance with the ECS
    Property Management Plan. The Site Property Administrator will
    generate an Inventory Reconciliation Report from the ILM system for
    use in the conduct of the inventory. Annual Inventory Reports will
    identify dates of the inventory, person(s) conducting the inventory,
    discrepancies noted, and actions taken to resolve the discrepancies.

### 23.3.3 Reporting Loss, Theft, Damage or Destruction

If ECS or GFP property at the site is lost, stolen, damaged, or
destroyed, the site Property Administrator will notify the ECS Property
Administrator and supply following information:

-   Description of items lost, stolen, damaged, or destroyed.

-   Cost of property lost, stolen, damaged, or destroyed and cost of
    repairs in instances of damage (if actual costs are not known,
    reasonable estimates will be given).

-   Date, time, and cause of the loss, theft, damage, or destruction.

-   Actions taken by Contractor to prevent further loss, theft, damage,
    or destruction and to prevent repetition of similar incidents.

-   Other facts or circumstances relevant to the determination of
    liability and responsibility for repair or replacement.

-   Statement that no insurance costs or other means of covering loss,
    theft, damage, or destruction of Government property were charged to
    the contract, if applicable.

-   Statement that, in the event the Contractor was or will be
    reimbursed or compensated for loss, damage, or destruction of
    Government property (e.g., reimbursement by a subcontractor) the
    Government will receive equitable reimbursement.

The ECS Property Administrator will provide this information without
delay to the ECS Contracts Manager, who will immediately notify the
Government Property Administrator (e.g., DCMAO).

### 23.3.4 Obtaining Relief from Accountability

The Site Property Administrator will provide all pertinent information
(e.g., type of equipment, reasons for obtaining relief from
accountability) without delay to the ECS Contracts Manager, who will
immediately notify the Government Property Administrator (e.g., DCMAO).

## 23.4 Equipment Relocation

This section provides instructions for equipment relocation within a
DAAC (intra-site relocation); between ECS sites, and between ECS sites
and non-ECS sites (inter-site relocation); to a vendor (off-site
relocation); and transfer to outside the contract (external transfer).

### 23.4.1 Intra-site Relocation 

Requirements for equipment reallocations within the facility or between
facilities at the same site will be processed through the site Property
Administrator to maintain control and accountability of equipment
inventories. A Trouble Ticket will be used to document and forward the
relocation request to the site Property Administrator. The site Property
Administrator will approve the request and schedule the relocation.
Configuration management authorization is required prior to
reconfiguring equipment or software within a DAAC. When completed, the
Trouble Ticket will be closed by the site Property Administrator and the
new location entered into the property record.

### 23.4.2 Inter-site Relocation

Requests to relocate equipment to another ECS site (or to a non-ECS
site) will be forwarded by the losing site's Property Administrator to
the ECS Property Administrator. Such requests will identify by EIN and
equipment description what is to be moved, where and when it is to be
moved, and the reason for the relocation. The ECS Property Administrator
will coordinate the relocation resources and schedule between the losing
and gaining Property Administrators. Once completed, the gaining
Property Administrator will report completion of the relocation to the
ECS Property Administrator, who will update the property record with the
new location and date of the action. Any loss or damage to the equipment
will be reported using the procedure described in section 23.3.3 when it
occurs or is first discovered. Configuration management authorization is
required prior to reconfiguring equipment or software between DAACs.
Inter-site relocations will be reported by site Property Administrators
as they occur.

### 23.4.3 Relocation Off-site for Vendor Repairs

For equipment returned to a maintenance vendor for repair, the repair
document will be retained in a "pending actions" file until the item is
returned. The repair document will identify RMA number, date of
shipment, expected return date, and vendor point of contact. Status code
"O" (out for repair) will be entered into the site property record.
Property tags will not be removed. Once returned, the date of return
will be recorded on the repair document, and the serial number and EIN
will be verified. The equipment status code in the property record will
be changed to reflect status "R" (received) or "I" (installed). In the
event the original equipment is replaced by the vendor because it is
beyond repair, the site Property Administrator should attempt to recover
the property tag from the vendor. In any case the site Property
Administrator shall report the item as unserviceable/non-repairable to
the ECS Property Administrator, who will archive the property record. If
a component is non-reparable and must be replaced via another
procurement action, the site administrator should initiate actions
described in section 23.3.4 to obtain relief from accountability.

### 23.4.4 External Transfers

Transfers of ECS property outside the contract must be approved by the
ECS Contracting Officer. Upon approval by the ECS Contracting Officer,
transfers of ECS property to the Government or to other contracts will
be as directed in written instructions provided by the site Property
Administrator.

## 23.5 Inventories and Audits

Site Property Administrators will complete a 100 percent physical
inventory of controlled ECS property and GFP at the site not later than
August 30 annually. Notification of the scheduled date of the inventory
will be provided to the ECS Property Administrator 30 days prior to the
inventory start date, who will advise the Government Property
Administrator of the site inventory schedules. ECS personnel responsible
for maintaining property records will not assist in the conduct of these
inventories. Inventories will be designed to achieve the following
objectives:

-   Verify that accountable equipment is still on hand.

-   Confirm or determine current locations and custodial responsibility
    for equipment and material.

-   Identify unrecorded equipment which qualifies for control.

-   Locate or identify missing equipment.

-   Identify unused or underutilized equipment and equipment or material
    in need of repair or rehabilitation.

The site Property Administrator will, at the time of completion of the
annual inventory, forward a marked up copy of the Inventory
Reconciliation Report to the ECS Property Administrator. The Inventory
Report will be signed by the siteÕs ECS Manager attesting that a 100
percent inventory was conducted and that all equipment is accounted for
except for those indicated as not on hand. All discrepancies will be
explained.

## 23.6 Storage

ECS property will be stored in clean, orderly, and secure areas
conforming to the environmental controls for temperature, humidity, and
electrostatic discharge (ESD) specified in the ECS Environmental Control
Plan. Access will be limited to authorized personnel and controlled by
the site Property Administrator. ECS property tags will appear on all
COTS equipment to distinguish contractor-acquired and vendor-loaned
equipment from GFP. Material will be inspected to determine
serviceability before being stored. Material that is unserviceable will
be segregated and disposed of following procedures described in the ECS
Property Management Plan, Section 19.

### 23.6.1 Segregation Requirements

Contractor-owned and vendor-loaned property will be segregated from
Government-owned property during storage. Site Property Administrators
will ensure that storage areas are kept in a clean, orderly manner.
Material will be stored on shelves, in bins or drawers as appropriate,
and its storage location entered into the site property record. Special
storage areas or controls will be provided for items subject to
corrosion, humidity, and temperature. Such items will be inspected
semi-annually by the site Property Administrator.

### 23.6.2 Stock Rotation

Material designated as "stock," such as computer tapes and CD's, will be
distributed on a first-in, first-out basis.

### 23.6.3 Physical Security

ECS property will be stored in secured areas where access will be
limited to authorized personnel and controlled by the site Property
Administrator.

## 23.7 Packing and Shipping

Prior to shipping centrally reportable equipment to the EDF or other ECS
sites, the site Property Administrator will report intent-to-ship to the
ECS Property Administrator or receiving site Property Administrator. The
report will identify the expected shipment date, carrier, shipping
document number, estimated weight and cube, number of pieces, shipper
and ship-to-address. Prior to shipment, a pre-shipment inspection will
be performed to verify the following:

-   Correct identification of equipment on packing lists and shipping
    documents including configurations, serial numbers, number of
    containers, and ship-to address.

-   Adherence to packaging and marking standards.

-   Inclusion of appropriately prepared documents within shipping
    containers.

ECS property being shipped from vendors and the EDF will be shipped to
the DAAC facility to the attention of the site Property Administrator.
Local policy at some sites may require delivery to a site central
receiving point. In such cases, written procedures will be developed
between the site Property Administrator and the site's central receiving
office regarding notification of receipts, documentation required, and
provisions for local delivery to the DAAC facility. The delivery of ECS
equipment to site central receiving points versus direct delivery to the
DAAC facility will be determined based on agreements and procedures
established between the host facility and the DAAC.

This page intentionally left blank.

# 24. Installation Planning

The ECS Facilities Plans (DID 302) and the Installation Plans developed
for each ECS release are the products of the facility and installation
planning process. DAAC Facilities Plans are distributed 30 days after
each Release Critical Design Review (CDR). These plans identify space,
power, and cooling requirements based on design information available at
CDR. The Installation Plans are distributed two months prior to
installation of equipment at each ECS Release. As such, the Facilities
Plan identifies facility preparation requirements and general
installation planning that is based on final design information. The
Installation Plans provide the detailed planning required by
installation teams and the sites to make final preparation for
installing Release equipment. Both documents are provided in draft to
the DAACs for review and comment prior to publication.

## 24.1 Responsibilities

Installation planning and coordination is the responsibility of the ECS
Installation Planner, who is part of the Integrated Logistics Support
(ILS) Office within M&O. Using information obtained during site surveys,
the Installation Planner prepares the Facility Plans and the
Installation Plans and coordinates actions needed to prepare for and
conduct the installations. DAAC M&O personnel support the Installation
Coordinator by providing information to complete the Site Survey
Questionnaire; reviewing the Facility Plan and the Installation Plan;
ensuring that site preparations/coordination are completed on schedule;
facilitating receipt and installation of the hardware; and accepting
installation of the hardware and software by signing the Installation
Receipt Report.

## 24.2 Process Description

DAAC site surveys have been previously conducted to obtain DAAC-specific
information needed to begin the installation planning process. This
information was documented in a Survey Questionnaire prepared for each
DAAC and is used in the preparation of the Facility Plans and the
Installation Plans. This information, plus design and equipment
specifications, is used to prepare the Facilities Plans, which project
facility requirements and provide a preliminary plan for the placement
of systems within the DAACs. DAACs review this information and provide
requested changes, which are considered in the preparation of the
Installation Plans.

Two months prior to the installation of hardware, a detailed
Installation Plan is produced to identify the planned placement of
hardware in the facility and how the hardware will be configured and
networked, and to identify site preparations necessary to support the
installation. Installation teams use the Installation Plan to install
the systems and networks. After the equipment is installed and tested,
the installation team leader obtains the DAAC Manager's signature on the
Installation Receipt Reports, which details the locations and equipment
that have been installed and networked. Within three weeks following the
installation, the Installations Planner will update the facility
diagrams and network diagram to reflect the as-installed configuration
at the site. These diagrams are submitted to the ECS CCB and, when
approved, become part of the operations baseline for the site. The
baselined diagrams are provided to the site and are the responsibility
of the LMC to update as changes occur.

## 24.3 Maintenance of Facility and Network Diagrams

Facility and network diagrams reflect the as-installed configuration.
The baseline version of these diagrams is maintained by the ECS
Installations Coordinator. As changes to these diagrams occur (e.g.,
relocation of equipment within the site, additions/deletions to the
LAN), the LMC will inform the ECS Installations Coordinator by redlining
the diagrams. The Installations Coordinator will update the CAD system
to reflect the change(s) and provide an updated facility drawing to the
site's LMC.

## 24.4 Maintenance of LAN Cable Management Scheme

Within three weeks of the completed hardware installation, a LAN Cable
Management Scheme is supplied to the DAAC LMC by the ECS Installations
Coordinator. This matrix will identify the cable number, type, length,
decibel loss rating, and location of cables installed; and will identify
the IP addresses of the equipment connected by the cables. The LMC will
update this matrix as LAN changes occur.

# 25. COTS Training

The procedures to request COTS training have been developed based on
these sources: DID 622 ECS Training Plan, DID 525 Training and
Certification Records, DID 626 M&O Certification Plan, and DID 607 M&O
Manual for the ECS Project. The (SEO ECS Operations Trainer) arranges
for COTS training by working with the COTS Training Coordinator, the ILS
Contractor Manager, as well as the potential students.

The Activity Checklist table that follows provides an overview of these
procedures. Column one (**Order**) shows the order in which tasks might
be accomplished. Column two (**Role**) lists the Role/Manager/Operator
responsible for performing the task. Column three (**Task**) provides a
brief explanation of the task. Column four (**Section**) provides the
Procedure (P) section number or Instruction (I) section number where
details for performing the task can be found. Column five
(**Complete?**) is used as a checklist to keep track of which task steps
have been completed.

Table 25.1-1. COTS Training - Activity Checklist

  ----------- -------------- ------------------------------ ------------- ---------------
  **Order**   **Role**       **Task**                       **Section**   **Complete?**

  1           (COTS Training Submit request for COTS        \(P\) 25.1    
              Coordinator)   Training                                     

  2           COTS Training  Forward request to ILS Manager \(P\) 25.2    
              Coordinator                                                 

  3           (COTS Training Arrange for equipment and      \(P\) 25.2    
              Coordinator)   classroom space                              

  4           (COTS Training Ensure that initial            \(I\) 25.3    
              Coordinator)   registration will be filled or               
                             arrange for cancellation                     
                             without penalty                              

  5           (COTS Training Maintain COTS training records \(I\) 25.4    
              Coordinator)                                                

  6           (COTS Training Monitor DAAC COTS training     \(I\) 25.5    
              Coordinator)   budget                                       
  ----------- -------------- ------------------------------ ------------- ---------------

## 25.1 Requesting COTS Training

The COTS Training Coordinator must request training to initiate the
following procedures **at least** **30 days prior** to the desired
training date. The procedures are accomplished in the following order:

**1** The COTS Training Coordinator requests training using COTS
Training Request Format via cc:mail to the SEO ECS Operations Trainer,
the COTS Training Request Format must include the following information:

¥ Student(s) name and position/role

¥ Training need

¥ COTS course requested

¥ Dates preferred

-   (Price of COTS course))

-   (Manager approving purchase of training)

-   (Course location)

-   (Duration of course)

2 (SEO ECS Operations Trainer) verifies that the training requested
meets the following criteria:

¥ Relates to an ECS M&O function

¥ Relates to COTS product in the ECS system design

¥ Is cost effective and within budget constraints

3 COTS Training Coordinator determines the proposed training details,
including the following:

¥ Training vendor

¥ Individual or group training, based on cost effectiveness

¥ On-site or off-site class location

¥ Available vendor training dates

4 (All COTS training must be approved by both the SEO ECS Operations
Trainer and the ILS Contractor Manager prior to procurement.) COTS
Training Coordinator forwards the training request to the (SEO ECS
Operations Trainer) for approval, once approved by the SEO ECS
Operations Trainer, it is then forwarded to the ILS Contractor Manager.
The ILS Contractor Manager will either approve or deny the request.

5 COTS Training Coordinator maintains record of approval of training
purchase.

## 25.2 Coordinating COTS Training

If the ILS Manager approves the request for COTS training, the COTS
Training Coordinator will provide all vendor training details to the
COTS Purchasing Manager. The COTS Purchasing Manager produces the
purchase order and provides a copy to the COTS Training Coordinator, who
will then order the training from the vendor. The procedures to
coordinate training are accomplished in the following order:

**1** When approved, the COTS Training Coordinator submits all training
details to the COTS Purchasing Manager.

**2** The COTS Training Coordinator orders training from the vendor.

**3** Purchasing Manager processes the purchase order and provides a
copy to the COTS Training Coordinator.

**4** The COTS Training Coordinator forwards the purchase order to the
vendor to reserve training.

**5** The COTS Training Coordinator generates a notice to students that
includes training vendor, course, date(s), other relevant information.

6 For on-site training, COTS Training Coordinator makes necessary
arrangements for classroom space and equipment configuration;
coordinates use of any operational equipment required for course, with
on-going operations; forwards site location details to vendor
instructor.

7 Students attend training.

8 Prior to COTS training, the COTS Training Coordinator provides
students with a COTS Training Evaluation Form, which evaluates the
effectiveness of the course. In cases when COTS training is found to be
substandard or ineffective, the COTS Training Coordinator contacts the
SEO ECS Operations Trainer, ILS Contractor Manager, and the DAAC or site
manager, together they come to a consensus as to whether or not to
pursue compensation for the training.

9 Depending upon the decision rendered, the COTS Training Coordinator
seeks refund, replacement training seat, or training credit from the
vendor.

10 The COTS Training Coordinator maintains training records at site in
accordance with DIDs 622 and 525. Required record fields include:
student name, vendor name, course name and number, course dates, and
location.

11 The (COTS Training Coordinator) will forward a copy of the training
record to the SEO ECS Operations Trainer in format specified by DID 622.

## 25.3 Canceling/Rescheduling COTS Training

COTS training vendors generally withhold all or part of registration
fees for course seats canceled too close to the start date of training.
The deadline for cancellation without penalty varies between vendors,
**but the maximum deadline is three weeks prior to course start date**.
In order to preserve ECS COTS training funds, any cancellations of COTS
training by ECS personnel must be made within three weeks of the start
date to avoid these financial penalties.

If student(s) need to cancel after this three-week deadline, the (DAAC
or site manager) will be responsible for substituting an equally
qualified individual to attend the course, and for notifying the COTS
Training Coordinator to ensure proper record keeping and registration
with the vendor.

## 25.4 Maintenance of COTS Training Records

The (COTS Training Coordinator) will maintain records of all training
accomplished as specified in DID 622. For COTS training, these records
will include the following fields:

-   Student name

-   Vendor name

-   Course title and number

-   Course location

-   Course duration

-   Dates attended

-   Certification (if applicable)

The COTS Training Coordinator will submit training record information to
the SEO ECS Operations Trainer for DID 525 as specified in DID 622.

## 25.5 Contractor COTS Training Funds Accounting

COTS training funds will be allocated to each ECS M&O organization,
based upon staffing levels and functions performed at the site. The ECS
M&O organizations to be allocated funds for COTS training are as follow:

-   DAACs

-   SMC

-   SEO

-   ILS

-   ECS Development Facility operations

While the coordination and purchasing responsibilities for COTS training
fall primarily with the ECS COTS Training Coordinator, the (SEO ECS
Operations Trainer) will be responsible to the M&O organization for
spending the allocated COTS training budget judiciously.

Travel funds are not included in the COTS training budget. These must be
secured from the organization to which each student belongs.

The COTS Training Coordinator will distribute quarterly balance reports
to the (SEO ECS Operations Trainer) for planning purposes.