# 8. Problem Management

ECS Problem Management is administered through system-level and
site-level reviews. These reviews oversee the analysis, recommendations,
and actions taken to resolve ECS system/site problems concerning
hardware, software, documentation, and procedures. The SEO and the
site-level maintenance organization resolve routine maintenance issues
at the system-level and site-level, respectively, using the Trouble
Ticket System for making maintenance changes. High priority issues
require concurrence of both Government and Contractor project managers
for resolution. Maintenance changes are recorded in the Remedy Action
Request System. Configuration changes are recorded in the Baseline
Manager tool. The Configuration Control Board (CCB) process will approve
all configuration changes, as stated in the M&O Configuration Management
Plan, 102-CD-001-002.

The Trouble Ticket System is the vehicle to record and report problems
with operational-phase configuration items. Trouble Tickets can be
generated by operations, maintenance, development, and customer
personnel as well as users. The Trouble Ticket System is wholly
automated and all problem resolution activities are recorded in the
database tool, the Remedy Action Request System. Documentation that is
not in electronic form is handled by the local CM Administrator (CMA)
and is listed as an attachment to the trouble ticket.

Any CM Administrator (site or system-level) can serve as a Trouble
Ticket database administrator at the local site. The CMA is responsible
for tracking ECS system-level issues discovered at the sites and for
propagating system problem resolutions to the site level. The CMA also
supports the activities of the Trouble Ticket Review Board. This
includes generating status reports, and implementing resolutions,
instructions, and changes as directed by the Board. User Services
Representatives monitor trouble tickets to notify users concerning
problem resolution and status. Maintenance engineers record all
activities in the trouble ticket. This information can be used to
determine critical maintenance concerns related to frequency of
occurrence, criticality level, and the volume of problems experienced.
The maintainability analysis will guide critical changes, volume and
type of support components to be utilized, and will focus further ECS
release development.

This section provides an overview of the Trouble Ticketing process and
defines the M&O procedures for processing and resolving trouble ticket
submissions. In addition, this section provides instructions for
diagnosing network problems.

## 8.1 The Problem Resolution Process --- An Overview

Users submit trouble tickets to the User Services Desk. External users
submit trouble tickets through the Internet \[using a series of
hypertext mark-up language (HTML) screens (see Section 8.3, below\].
DAAC personnel submit trouble tickets via Remedy. User Services submits
the trouble ticket through Remedy for problem resolution. The Operations
Supervisor reviews the trouble ticket and assigns it to a Software
Maintenance Engineer.

Trouble tickets are first evaluated to determine the severity of the
problem and assignment of on-site responsibility/cognizance. Every
trouble ticket is logged into the database for record keeping purposes.
Trouble tickets that can be resolved locally are assigned and tracked at
the local center. Matters of sufficient importance are escalated to the
Trouble Ticket Telecon agenda for further discussion and disposition.

Trouble Tickets are discussed at a weekly trouble ticket Telecon. This
meeting functions as the ECS Failure Review Board in compliance with the
*EOS Performance Assurance Requirements for ECS,* GSFC 420-05-03,
Section 7.12.2.2. The telecon is held to coordinate trouble ticket
activities within the M&O organization as well as with development,
customer, and user organizations. Attendees include: Customer
representatives; ECS SEO engineering teams leads (one is designated as
chairperson of the meeting); ECS ILS engineering support; ECS
engineering team leads and operations representatives; ECS M&O support
staff; ECS development organization representatives (including
management, technical, configuration management and quality assurance);
SCF(s) representatives.

Agenda items may be supplemented or replaced by hardcopy or softcopy
reports. Material from this meeting is distributed within each ECS
organization and to customer and user organizations as required. A
typical agenda might include:

• Review and prioritize each trouble ticket opened at each center.

• Review and re-prioritize older trouble tickets (as required).

• Assign trouble ticket work-off responsibility to one organization.

• Review distribution of trouble tickets by organization, priority and
age.

• Discuss trouble ticket issues with development organizations.

The Maintenance & Operations Problem Management Concept is outlined in
the following procedures, which are illustrated in Figures 8.2-1 and
8.2-2:

1\. User or Operator discovers a problem with ECS configuration item(s)
(hardware, software, documentation, procedure) and documents this
problem for later resolution. He or she submits to User Services: by
calling up the Trouble Ticket System via the Internet (see Section 8.3,
below); by going on-line with the Trouble Ticket database (Remedy); by
phoning User Services; or by sending an e-mail message to Remedy. (See
Section 8.2.2.)

2\. The trouble ticket is logged into the system. Remedy automatically
assigns \"New\" status to the trouble ticket and notifies the Operations
Supervisor for assignment and prioritization. Remedy notifies the
Operations Supervisor via email, or through Remedy\'s notification tool,
or both (see Section 8.2.1.4). Trouble tickets are subsequently statused
and reported by the CMA.

3\. The trouble ticket log is prioritized according to a triage system
of maintenance priorities which will be determined in relation to the
effect of a problem on mission success which shall be differentiated by
scope of impact, frequency of occurrence, and the availability of an
adequate work-around. The Performance Assurance Requirements document,
NASA 420-05-03, identifies problem categories which correspond to the
triage system of maintenance priorities:

**Category 1: System/Service cannot perform critical function or imposes
major safety hazard. (Priority 1)**

Presents an immediate impact to development, operations, services, or
data processing functions; imposes major safety hazard to personnel,
systems, or space mission resources; or results in loss of one or more
essential mission objectives.

**Category 2: System/Service substantially impaired. (Priority 2)**

Substantially impacts development, operations, services, or data
processing functions; fails to operate within critical performance
specifications; or cannot effectively or efficiently fulfill baseline
requirements.

**Category 3: System/Service slightly impaired. (Priority 3)**

Causes minor or no substantial impact to development, operations,
services, or data processing functions. Support may be degraded, but
mission can still be accomplished.

The Trouble Ticket System tool (Remedy) has coded the triage as HIGH,
MEDIUM, and LOW. The User can designate a priority level for the
problem. However, the official priority is assigned by the Operations
Supervisor and maintained by the CM Administrator. All Category 1
(Priority 1) trouble tickets will be elevated to the Government Failure
Review Board and will require both Government and Contractor Project
Manager approval for final close-out. SEO applies these additional
priorities:

**Priority 4: Nuisance Problem:** Includes the arrangement of video
screens, color, and so on.

**Priority 5: Closed Problem:** A known issue with a prior disposition.

4\. All affected Operations Supervisors at the sites (SMC, DAACs, EOC,
EDF) are notified of the problem and solicited for inputs to problem
assessment (impact) and resolution.

5\. The Trouble Ticket database is updated with inputs received by the
Operations Supervisor, and the trouble ticket may be modified to reflect
this new information/coordination activity.

6\. The Operations Supervisor assigns the problem to a Problem
Investigator for further follow-up.

7\. The Problem Investigator coordinates input from SEO, developers,
vendors, and external organizations to effect the local resolution. The
Problem Investigator presents significant issues at the contractor
Telecon sponsored by SEO.

8\. The Problem Investigator updates the trouble ticket database.

9\. The Problem Investigator forwards any information regarding
proposed/implemented fixes to the established notification list.

10\. The proposed resolution is then presented to the Trouble Ticket
Review Board (and Government Failure Review Board for Priority 1 and
Priority 2 problems) for review, ratification, or revision.

11\. Changes that do not affect configuration controlled items may be
approved and implemented by the Failure Review Board/Trouble Ticket
Review Board and closed. CCBs must approve changes that affect
configuration controlled items. Temporary changes are updated in
Baseline Manager (see Section 9.9 of this document). Permanent changes
are proposed in a CCR to either on-site or system-level CCBs. Emergency
fixes can be made and then reported to the CCB after the crisis is
resolved. (See Section 8.4.)

12\. The CCBs may approve, reject, or revise the change proposal.

13\. The on-site Trouble Ticket Review Board may revise the proposed
solution.

14\. The off-site problem resolution process will be managed by the SEO
Trouble Ticket Review Board, who may also revise the on-site proposed
solution because of any system-level effect(s). This is part of the
problem escalation process which also includes appropriate submission of
CCR(s).

15\. The CCR may be escalated to higher level CCBs for system and/or
external elements that may be involved in the resolution process.

![](media/image1.wmf){width="6.05625in" height="8.031944444444445in"}

Figure 8.2-1. ECS Problem Management Concept Part I

![](media/image2.wmf){width="6.498611111111111in"
height="8.096527777777778in"}

Figure 8.2-2. ECS Problem Management Concept Part II

## 8.2 Using the Trouble Ticket System Tool

The Remedy Action Request System provides a distributed Trouble
Ticketing Service that furnishes DAACs a common environment and means of
classifying, tracking, and reporting problem occurrence and resolution
to both ECS users and operations personnel. The Trouble Ticketing
Service:

-   Provides a GUI for operations personnel to access all Trouble Ticket
    services.

-   Provides a common Trouble Ticket entry format.

-   Stores Trouble Tickets.

-   Retrieves Trouble Tickets via ad hoc queries.

-   Allows operations personnel to forward problems from one DAAC to
    another.

-   Produces stock and common reports.

-   Interfaces with user's and operator's e-mail to provide automatic
    notification.

-   Offers an application programming interface through which
    applications can submit trouble tickets.

-   Provides summary information to the SMC from each DAAC to allow
    trend reports regarding trouble tickets.

-   Defines a consistent "life cycle" for trouble tickets.

-   Allows each DAAC a degree of customization through definition of
    further re-prioritization and action rules.

Rules for re-prioritization are time-activated events, which execute on
trouble tickets that meet a set of specified criteria (see Section
8.2.9, below). Actions which can be taken include notification (either
to a user or to a support staff member), writing to a log file, setting
a field value on the trouble ticket, or even running a custom written
process. Qualifications can be expressed on any trouble ticket data
tracks. Active links are similar to escalation rules with the exception
that they are defined to take place on a specified action rather than at
a given time.

In addition to the functionality provided by Remedy's Action Request
System, the Trouble Ticketing Service utilizes a set of custom HTML
pages (\"screens\") to provide registered users with the ability to
submit new trouble tickets and query the current status of any of their
previous entries. Access to the Trouble Ticketing System through this
technique provides users an easy method for reporting problems in an
environment with which most are already familiar. (See Section 8.3,
below.) Additionally, as another means of trouble ticket entry, the
Trouble Ticket System provides a text e-mail template through which
automated entry of trouble tickets is possible. Support staff members
are able to enter Trouble Tickets through the Remedy's Action Request
System interface for problems received via other methods (for example,
phone calls).

In addition to tracking trouble tickets, the Remedy Action Request
System also functions as the User Contact Log. Remedy's Action Request
System is configured to have a separate schema that contains the entries
that User Services personnel enter for each contact that they receive
from a user. The User Contact Log allows a trouble ticket to be
initiated from a log entry with the push of a button --- the Trouble
Ticket will be populated with information from the contact log.

Sections 8.2.1 through 8.2.9 provide procedures using HTML and Remedy.
If you need more information about using HTML, see Section 8.3, below.
For more information using Remedy, see the Remedy User\'s Guide.

The Activity Checklist table that follows provides an overview of the
Trouble Ticket System. Column one (**Order**) shows the order in which
tasks might be accomplished. Column two (**Role**) lists the
Role/Manager/Operator responsible for performing the task. Column three
(**Task**) provides a brief explanation of the task. Column four
(**Section)** provides the Procedure (P) section number or Instruction
(I) section number where details for performing the task can be found.
Column five (**Complete?**) is used as a checklist to keep track of
which task steps have been completed.

***Trouble Ticket System - Activity Checklist***

  ----------------------------------------------------------------------------------------
  **Order**   **Role**        **Task**                       **Section**   **Complete?**
  ----------- --------------- ------------------------------ ------------- ---------------
  1           ECS users       Access the Trouble Ticket      8.2.1, 8.3.1  
                              System                                       

  2           ECS users       Submit Trouble Ticket          8.2.2         

  3           Maintenance     Modify Open Trouble Ticket     8.2.3         
              Engineer                                                     

  4           Operations      Forward Trouble Ticket         8.2.4         
              Supervisor,                                                  
              Maintenance                                                  
              Engineer                                                     

  5           Database        Add Users to Remedy            8.2.5         
              Administrator                                                

  6           CMA             Modify Remedy User Privileges  8.2.6         

  7           CMA             Modify Remedy\'s Configuration 8.2.7         

  8           Maintenance     Generate Reports               8.2.8         
              Engineer, CMA                                                

  9           Maintenance     Maintain Escalation Time Table 8.2.9         
              Engineer                                                     
  ----------------------------------------------------------------------------------------

### 8.2.1 Accessing the Trouble Ticket System

The Trouble Ticket System is accessed through either HTML or Remedy. The
Trouble Ticket HTML is used by both User Services and the end user to
submit trouble tickets without going through Remedy. It is accessed by
clicking on the Trouble Ticket icon. Through HTML, the user can submit,
obtain a list, and view details of trouble tickets.

Through Remedy, the User clicks on the User Tool icon, which opens the
RelB-Trouble Tickets schema to submit, query, or work a Trouble Ticket.
The Main Remedy Trouble Ticket screen is used to select the appropriate
schema for submitting, modifying, or displaying a trouble ticket. The
Main Page data fields are identified in Table 8.2.1-1.

The RelB-Trouble Tickets Schema provides the following options, which
are initiated by clicking on the appropriate button:

-   **Forward** \-- Forwards this Trouble Ticket to the site specified
    in the \"Forward-to\" field.

-   **Hardware Information** \-- Opens a window that is associated with
    this Trouble Ticket to hold hardware information. (See Section
    8.2.1.3.)

-   **List All Masters** \-- All Trouble Tickets that are duplicates of
    each other have one master. This button will lists all master
    Trouble Tickets.

-   **List This Trouble Ticket\'s Duplicate(s)** \-- List all Trouble
    Tickets that have duplicates associated to this Trouble Ticket.

-   **Go to Contact Log** \-- If this Trouble Ticket was created from a
    Contact Log, then this button will open a window to that Contact
    Log.

Several time-saving features are available through Remedy: the Admin
Tool, GUI Import tool, the Hardware Information schema, and the GUI
Notification tool. Brief descriptions are provided in Sections 8.2.1.1
through 8.2.1.4.

Table 8.2.1-1. RelB-Trouble Ticket Field Description (1 of 2)

+-----------------+--------+-----+---------+-------------------------+
| Field Name      | Data   | S   | Entry   | Description             |
|                 | Type   | ize |         |                         |
+-----------------+--------+-----+---------+-------------------------+
| Ticket-Id       | Cha    | 15  | System  | Ticket number which is  |
|                 | racter |     | ge      | set and maintained by   |
|                 |        |     | nerated | the system              |
+-----------------+--------+-----+---------+-------------------------+
| Ticket Status   | Sel    | 4   | R       | Status of the Trouble   |
|                 | ection |     | equired | Ticket                  |
+-----------------+--------+-----+---------+-------------------------+
| As              | Sel    | 4   | R       | Priority of Trouble     |
| signed-Priority | ection |     | equired | Ticket assigned at the  |
|                 |        |     |         | site (HIGH, MEDIUM,     |
|                 |        |     |         | LOW)                    |
+-----------------+--------+-----+---------+-------------------------+
| Short           | Cha    | 128 | R       | Short Description of    |
| Description     | racter |     | equired | the problem             |
+-----------------+--------+-----+---------+-------------------------+
| Submitter       | Sel    | 4   | O       | Impact of the problem   |
| Impact          | ection |     | ptional | to the submitter        |
+-----------------+--------+-----+---------+-------------------------+
| L               | Cha    | 255 | O       | Long Description of the |
| ong-Description | racter |     | ptional | problem                 |
+-----------------+--------+-----+---------+-------------------------+
| Resolution Log  | Diary  | Un  | O       | General steps in the    |
| (End User Sees) |        | lim | ptional | resolution of the       |
|                 |        |     |         | problem                 |
+-----------------+--------+-----+---------+-------------------------+
| Detailed        | Diary  | Un  | O       | Detailed steps in the   |
| Resolution Log  |        | lim | ptional | resolution of the       |
|                 |        |     |         | problem                 |
+-----------------+--------+-----+---------+-------------------------+
| Submitter ID    | Cha    | 30  | R       | User Id of the          |
|                 | racter |     | equired | Submitter.              |
+-----------------+--------+-----+---------+-------------------------+
| Submitter Name  | Cha    | 30  | O       | Full Name of the        |
|                 | racter |     | ptional | Submitter               |
+-----------------+--------+-----+---------+-------------------------+
| Submitter Phone | Cha    | 30  | O       | Phone number of the     |
|                 | racter |     | ptional | Submitter               |
+-----------------+--------+-----+---------+-------------------------+
| Submitter eMail | Cha    | 64  | O       | E-mail address of the   |
|                 | racter |     | ptional | Submitter               |
+-----------------+--------+-----+---------+-------------------------+
| Submitter Home  | Cha    | 60  | O       | Home DAAC of the        |
| DAAC            | racter |     | ptional | Submitter               |
+-----------------+--------+-----+---------+-------------------------+
| History         | Diary  | Un  | O       | Upon submission or      |
|                 |        | lim | ptional | modification, the       |
|                 |        |     |         | person assigned to the  |
|                 |        |     |         | ticket and the ticket   |
|                 |        |     |         | status will be          |
|                 |        |     |         | indicated in the        |
|                 |        |     |         | History field.          |
|                 |        |     |         |                         |
|                 |        |     |         | Due to a limitation in  |
|                 |        |     |         | Remedy, this            |
|                 |        |     |         | information will only   |
|                 |        |     |         | be written when the     |
|                 |        |     |         | Assigned-to and Status  |
|                 |        |     |         | fields are modified     |
+-----------------+--------+-----+---------+-------------------------+
| Assigned-To     | Cha    | 30  | O       | Person that Trouble     |
|                 | racter |     | ptional | Ticket has been         |
|                 |        |     |         | assigned to             |
+-----------------+--------+-----+---------+-------------------------+
| L               | Cha    | 30  | System  | Person that last        |
| ast-modified-by | racter |     | ge      | modified the Trouble    |
|                 |        |     | nerated | Ticket                  |
+-----------------+--------+-----+---------+-------------------------+
| Create-date     | Dat    | 4   | System  | Date Trouble Ticket was |
|                 | e/Time |     | ge      | created at the present  |
|                 |        |     | nerated | site                    |
+-----------------+--------+-----+---------+-------------------------+
| Las             | Dat    | 4   | System  | Date the Trouble Ticket |
| t-Modified-date | e/Time |     | ge      | was last modified       |
|                 |        |     | nerated |                         |
+-----------------+--------+-----+---------+-------------------------+
| Related CCR     | Cha    | 60  | O       | ID of a related CCR     |
|                 | racter |     | ptional |                         |
+-----------------+--------+-----+---------+-------------------------+
| Key Words       | Cha    | 255 | O       | Key words to help       |
|                 | racter |     | ptional | identify this Trouble   |
|                 |        |     |         | Ticket                  |
+-----------------+--------+-----+---------+-------------------------+
| Problem Type    | Cha    | 30  | O       | Type of problem         |
|                 | racter |     | ptional | addressed by this       |
|                 |        |     |         | Trouble Ticket          |
+-----------------+--------+-----+---------+-------------------------+

Table 8.2.1-1. RelB-Trouble Ticket Field Description (2 of 2)

+-----------------+--------+-----+---------+-------------------------+
| Field Name      | Data   | S   | Entry   | Description             |
|                 | Type   | ize |         |                         |
+-----------------+--------+-----+---------+-------------------------+
| Closing Code    | Cha    | 60  | O       | Origin of the problem   |
|                 | racter |     | ptional | that this Trouble       |
|                 |        |     |         | Ticket resulted from    |
+-----------------+--------+-----+---------+-------------------------+
| Closed-by       | Cha    | 60  | O       | Person that closed this |
|                 | racter |     | ptional | Trouble Ticket          |
+-----------------+--------+-----+---------+-------------------------+
| Close-date      | Dat    | 4   | O       | Date this Trouble       |
|                 | e/Time |     | ptional | Ticket was closed       |
+-----------------+--------+-----+---------+-------------------------+
| Software        | Cha    | 60  | O       | Software Resource that  |
| Resource        | racter |     | ptional | the problem came from   |
+-----------------+--------+-----+---------+-------------------------+
| Hardware        | Cha    | 60  | O       | Hardware Resource that  |
| Resource        | racter |     | ptional | this problem came from  |
+-----------------+--------+-----+---------+-------------------------+
| Duplicate       | Cha    | 25  | O       | The Master Ticket-ID of |
| Master Id       | racter |     | ptional | this Trouble Ticket     |
+-----------------+--------+-----+---------+-------------------------+
| Forward-to      | Cha    | 60  | O       | Site that this Trouble  |
|                 | racter |     | ptional | Ticket was last         |
|                 |        |     |         | forwarded to            |
+-----------------+--------+-----+---------+-------------------------+
| Forwarded-from  | Cha    | 60  | O       | Site that forwarded     |
|                 | racter |     | ptional | this Trouble Ticket     |
+-----------------+--------+-----+---------+-------------------------+
| Forwarded-by    | Cha    | 60  | O       | Contact person at the   |
|                 | racter |     | ptional | forwarding site         |
+-----------------+--------+-----+---------+-------------------------+
| Forward-date    | Dat    | 4   | O       | Date Trouble Ticket was |
|                 | e/Time |     | ptional | forwarded               |
+-----------------+--------+-----+---------+-------------------------+
| Un              | Cha    | 20  | O       | Unique identifier which |
| ique-Identifier | racter |     | ptional | is established at the   |
|                 |        |     |         | origination site        |
|                 |        |     |         |                         |
|                 |        |     |         | This identifier should  |
|                 |        |     |         | NEVER be changed once   |
|                 |        |     |         | set                     |
+-----------------+--------+-----+---------+-------------------------+
| Forwarded-to-1  | Cha    | 60  | O       | First site to have been |
|                 | racter |     | ptional | forwarded this Trouble  |
|                 |        |     |         | Ticket                  |
+-----------------+--------+-----+---------+-------------------------+
| Forwarded-to-2  | Cha    | 60  | O       | Second site to have     |
|                 | racter |     | ptional | been forwarded this     |
|                 |        |     |         | Trouble Ticket          |
+-----------------+--------+-----+---------+-------------------------+
| Forwarded-to-3  | Cha    | 60  | O       | Third site to have been |
|                 | racter |     | ptional | forwarded this Trouble  |
|                 |        |     |         | Ticket                  |
+-----------------+--------+-----+---------+-------------------------+
| Forwarded-to-4  | Cha    | 60  | O       | Fourth site to have     |
|                 | racter |     | ptional | been forwarded this     |
|                 |        |     |         | Trouble Ticket          |
+-----------------+--------+-----+---------+-------------------------+
| Associated      | Cha    | 30  | O       | ID number of the        |
| Contact Log Id  | racter |     | ptional | Associated Contact Log  |
+-----------------+--------+-----+---------+-------------------------+

#### 8.2.1.1 Remedy\'s GUI Admin Tool

The Admin Tool is used to notify or set fields as soon as the trouble
ticket reaches a particular state or to escalate the problem once a
trouble ticket is in a particular state too long. This tool is accessed
in two ways:

1\. By clicking on Admin Tool to open correct filter, escalation, or
active link. (Problem escalation is discussed in Section 8.2.9.)

2\. By clicking on User Tool icon and opening RelB-TT-Times schema to
review/modify a Trouble Ticket.

For more information on the Admin Tool, refer to the Remedy
Administration Manual.

#### 8.2.1.2 Remedy\'s GUI Import Tool

The GUI Import tool is used to import existing entries rather than
retyping information manually. It also enables the user to import
entries into a schema from a file generated by the Admin tool. This tool
is accessed by clicking on the Remedy Import Tool icon. For more
information on the Import tool, refer to the Remedy User Guide.

#### 8.2.1.3 Remedy\'s Hardware Information Schema

If detailed hardware information needs to be provided beyond what can be
entered on the Trouble Tickets schema. The User Tool, Hardware
Information schema, provides the vehicle to add a description of a
hardware problem that corresponds to a trouble ticket. Through this
schema, the user can enter detailed information about failed hardware
components (e.g., part and serial numbers) and the actions taken to
correct the problem. This schema is accessed by clicking on the User
Tool icon and opening RelB-Hardware Information schema, or via Hardware
Information link from Trouble Tickets schema.

#### 8.2.1.4 Remedy\'s GUI Notification Tool

The GUI Notification Tool is used as an alternative to email
notification to notify the user of a Remedy event. This tool is accessed
by clicking on the Remedy Notification Tool icon. It allows properties
and options to be modified via pull-down menus. Examples of GUI
notification include a beep, a pop-up window, a flashing message. In
addition, both an email and a GUI notification can be sent if the site
so desires.

### 8.2.2 Submit a Trouble Ticket

When a problem is either found by or reported to User Services, follow
the procedure applicable to your system, to create and log trouble
tickets. Trouble tickets can be submitted via HTML or via Remedy\'s user
tool -- RelB-Trouble Tickets schema. Remedy\'s Contact Log schema is
used to classify, track, and report contacts of ECS users and operators
and also to submit a trouble ticket from a log entry. E-mail is another
method of submitting a trouble ticket. The template is available from
your System Administrator.

1.  For HTML submission:

    a.  Access HTML Trouble Ticketing Main page.

        Select Submit link which opens the Submit page.

        Fill out the impact, short description, and detailed description
        fields.

        Select Submit.

    For submission through Remedy (See the Remedy User Guide, Chapter 3,
    "Submitting an Action Request" for the general steps):

    a.  Access Remedy User Tool (See the Remedy User Guide, Chapter 2,
        "Getting Started with the User Tool", page 2-3, section on
        "Starting the User Tool").

        Access RelB-Trouble Ticket schema (See the Remedy User Guide,
        Chapter 2, "Getting Started with the User Tool", page 2-18,
        section on "Using Schemas").

        Select Open Submit from the File menu.

        Fill out at least Short Description and User Identification.

        Fill out any other pertinent fields.

        Select Apply.

    For submission from a Remedy Contact Log entry:

    a.  Click on User Tool icon and open RelB-Contact Log schema.

        Fill out Contact Log ID and Contact Information. If the contact
        is a registered Remedy user, the contact information is filled
        out automatically.

        Fill in Short Description (limit is 128 characters).

        Click on **Create TT** button.

    For submission via E-mail:

    a.  Obtain Template from your System Administrator.

        Address the message to
        arsystem@\_\_\_\_\_\_\_\_\_\_\_\_.\_\_\_\_\_\_\_\_\_\_\_.\_\_\_\_\_\_\_\_.

        Copy template into message area. DO NOT INCLUDE AS AN
        ATTACHMENT. DO NOT ALTER TEMPLATE. The template is presented in
        Figure 8.2.2-1. The \# sign indicates comments, which are not
        read by Remedy. **Enter data as indicated in Figure 8.2.2-1.**
        Send message.

> \#
>
> \# File exported Wed Feb 28 19:01:27 1996
>
> \#
>
> Schema: RelB-Trouble Tickets
>
> Server: remedy server name
>
> Login:
>
> Password:
>
> Short Description ! 8!:
>
> Submitter Impact !536870922!: Low
>
> \# Values: Low, Medium, High
>
> Long-Description ! 9!:
>
> Submitter ID ! 2!:
>
> Submitter Name !536870917!:
>
> Submitter Phone !536870918!:
>
> Submitter eMail !536870921!:
>
> Submitter Home DAAC !536870919!:

Figure 8.2.2-1: Trouble Ticket E-mail Template

### 8.2.3 Reviewing and Modifying Open Trouble Tickets

Trouble tickets may need to be modified based on better understanding of
the nature of problems defined and revised resolutions from the
Maintenance Engineer investigations, Sustaining Engineering inputs,
Developer inputs, Trouble Ticket Review Board decisions, Change Control
Board decisions, and/or Failure Review Board decisions. The results will
be factored into revisions and/ or additions to the Trouble Ticket log.

1.  For HTML Review and Modification of Trouble Tickets:

    a.  Access HTML Trouble Ticketing Main (see Section 8.3). Trouble
        Tickets can be ***submitted, queried or modified***.

        Select List link which opens the List page and shows each
        Trouble Ticket's Identification, Short Description, and Status.

        Select the Trouble Ticket Id to get a more detailed description
        of that particular Trouble Ticket.

    For Reviewing and Modifying Trouble Tickets through Remedy (See the
    Remedy User Guide, Chapter 4, "Reviewing and Modifying Action
    Request" for the general steps):

    a.  Access Remedy User Tool (See the Remedy User Guide, Chapter 2,
        "Getting Started with the User Tool", page 2-3, section on
        "Starting the User Tool").

        Access Release B-Trouble Ticket schema (See the Remedy User
        Guide, Chapter 2, "Getting Started with the User Tool", page
        2-18, section on "Using Schemas").

        Select List from the Query menu.

        From the List pick the Trouble Ticket(s) that you would like to
        Review/Modify.

        Select Modify Individual from the Query menu of the List window
        review and modify Trouble Ticket.

### 8.2.4 Forwarding Trouble Tickets

Trouble ticket administrative reports are forwarded for local and
system-wide usage. The trouble ticket contains all forwarding
information; once forwarded, it goes to the RelB-TT-ForwardToSite
holding area (transparent to the user). The RelB-TT-Sites schema is used
to indicate the site name and email address to be used in forwarding. To
forward a trouble ticket:

1.  Click on User Tool icon and open RelB-Trouble Ticket schema.

2.  Set the status to "Forwarded".

3.  Select a value for the Forward-to field from its picklist.

4.  Select the Forward button.

5.  Select Apply.

If necessary, a site name and email address can be modified, added, or
deleted to update the picklist of Release B sites by authorized Remedy
users: DAACs, SMC, NSI, EBnet. To modify picklist:

1.  Click on User Tool icon and open RelB-TT-Sites schema.

2.  Modify data.

### 8.2.5 Adding Users to Remedy

The database administrator uses the RelB-User schema to grant access to
the Remedy tool. See Remedy Administrator's Guide for OSF/ Motif,
Chapter 3, "Setting Up Users and Groups", page 3-11, section on "Adding
Users". Users who change jobs can be deleted.

### 8.2.6 Changing Privileges in Remedy

This procedure is used by the CM Administrator to control privileges of
those who have been granted access. For more information, refer to the
Remedy Administrator's Guide.

**NOTE:** No group should be modified without proper configuration
change approval.

To change Privileges in Remedy:

1.  See Remedy Administrator's Guide, Chapter 3, "Setting Up Users and
    Groups", page 3-2, section on "Understanding Access Control".

2.  See Remedy Administrator's Guide, Chapter 3, "Setting Up Users and
    Groups", page 3-4, section on "Access Control Groups".

3.  Groups have already been created to accommodate all privileges
    needed by Remedy Users for Release B. These groups are identified in
    see Table 8.2.6-1.

Table 8.2.6-1. Table of Access Control Groupings

  --------------------------------------------------------------------------
  Groups                Description                        Access Type
  --------------------- ---------------------------------- -----------------
  Operator              Submits trouble ticket internally. Change

  User Services         Submits trouble ticket internally  Change
                        for user.                          

  Operations Supervisor Assigns problem priority and       Change
                        resolution responsibility. Can     
                        forward trouble ticket to another  
                        site.                              

  Resource Manager      Assigns problem priority and       Change
                        resolution responsibility. Can     
                        forward trouble ticket to another  
                        site.                              

  Resolution Technician Attempts to resolve problem.       Change

  Trouble Ticket Review Reviews proposed solutions         Change
  Board Chair Person                                       

  Administrator         Adds groups and users. Changes     Change
                        permissions. Sets escalation       
                        times. Sets menu items. Etc.       

  Sub-Administrator     Same functions as Administrator    Change
                        but only with certain Schemas.     

  Browser               Read only permission.              Read

  Customize             Can use all features of the        Change
                        customize facility.                

  Submitter             Place holder for anyone that       NA
                        submits a trouble ticket.          

  Assignee              Place holder for anyone that is    NA
                        assigned a trouble ticket.         

  Public                Read only permission. Guest users  Read
                        are automatically put in this      
                        group.                             

  NotifyNewEscal        Everyone that will be notified on  Read
                        an escalation due to trouble       
                        ticket being in "New" status.      

  NotifyAssignedEscal   Everyone that will be notified on  Read
                        an escalation due to trouble       
                        ticket being in "Assigned" status. 

  NotifySolPropEscal    Everyone that will be notified on  Read
                        an escalation due to TT being in   
                        "Solution Proposed" status.        

  NotifyImpSolEscal     Everyone that will be notified on  Read
                        an escalation due to trouble       
                        ticket being in "Implement         
                        Solution" status.                  

  NotifySolImpEscal     Everyone that will be notified on  Read
                        an escalation due to TT being in   
                        "Solution Implemented" status.     
  --------------------------------------------------------------------------

### 8.2.7 Modifying Remedy\'s Configuration

RelB-Trouble Ticket schemas\' pulldown menus can be customized.
Customization is achieved through the User Tool by modifying the
RelB-Menu-Closing Codes, RelB-Menu-Hardware Resources,
RelB-Menu-Software Resources, RelB-Menu-Key Words, RelB-Menu-Problem
Type, Sites schemas.

To modify the Remedy environmental variables, refer to the Remedy User's
Guide and Remedy Administrator's Guide as indicated.

1.  See Remedy User's Guide, Chapter 7, "Customizing the Environment."

2.  See Remedy Administrator's Guide, Chapter 1, "Using the
    Administrator Tool."

**NOTE:** No administrative configuration should be made without proper
configuration change approval.

### 8.2.8 Generating Trouble Ticket Reports

A set of predefined reports will be placed in a public directory that
should be downloaded to your personal configuration directory (see the
Remedy User Guide, Chapter 2, "Getting Started with the User Tool," page
2-31, section "Sharing Macros, User Commands and Custom Reports,"
sub-section "Copying Files"). These reports are trouble ticket
administrative reports generated for local and system-wide usage. See
Remedy User's Guide, Chapter 5, "Reports."

### 8.2.9 Re-prioritization of Dated Trouble Ticket Logs

Remedy provides automated prioritization of trouble tickets based on
delinquency status of out-dated trouble ticket logs. The File Tickler
System automatically assigns higher priority to promote timely
resolution.

1.  Access Remedy User Tool (See the Remedy User Guide, Chapter 2,
    "Getting Started with the User Tool," page 2-3, section on "Starting
    the User Tool").

2.  Access RelB-Times schema (See the Remedy User Guide, Chapter 2,
    "Getting Started with the User Tool", page 2-18, section on "Using
    Schemas").

3.  Select List from the Query menu.

4.  From the List pick the Time(s) that you would like to Review/Modify.

5.  Select Modify Individual from the Query menu of the List window to
    review and/or modify the Time (in seconds).

## 8.3 Using Hypertext Mark-up Language (HTML) Screens

The hypertext mark-up language (HTML) Trouble Ticket Main Screen ("ECS
Trouble Ticketing: Menu") provides an introduction on how to use the
Trouble Ticketing HTML, and is used by registered ECS users to go to
either the Submit page or List page.

Selecting **[Submit a Trouble Ticket]{.underline}** will bring up the
Trouble Ticketing Submit screen.

Selecting **[List the \[username\] Trouble Tickets]{.underline}** will
bring up the Trouble Ticketing List screen.

Help on the Trouble Ticket HTML screens is available by clicking on the
Trouble Ticket Help icon at the bottom of the screen .

### 8.3.1 ECS Trouble Ticketing HTML Submit Screen

The HTML Trouble Ticket Submit screen is used by registered ECS users to
submit a Trouble Ticket.

Table 8.3.1-1 below provides a description of the Trouble Ticket HTML
Submit Screen fields.

Table 8.3.1-1. Trouble Ticket HTML Submit Screen Field Description

  -----------------------------------------------------------------------------
  Field Name         Data Type   Size   Entry       Description
  ------------------ ----------- ------ ----------- ---------------------------
  ID                 character   30     System      Submitter Id
                                        generated   

  Name               character   30     System      Submitter Name
                                        generated   

  E-mail address     character   64     System      Submitter E-mail Address
                                        generated   

  Phone              character   30     System      Submitter Phone Number
                                        generated   

  Home DAAC          character   60     System      Submitter Home DAAC
                                        generated   

  Impact             selection   4      Required    Impact to Submitter

  Short description  character   125    Required    Short description of
                                                    problem

  Detailed problem   character   245    Optional    Long description of problem
  description                                       
  -----------------------------------------------------------------------------

When the information is completed, the user can submit the Trouble
Ticket by clicking on the **Submit** button on the lower half of the
screen. The Problem Information Fields can be cleared by clicking on the
**Reset** button. The user also has the choice of returning to the
Trouble Ticketing Homepage or going to the Trouble Ticket Help screen by
clicking on the respective icons at the bottom of the page.

### 8.3.2 ECS Trouble Ticketing HTML Success Screen

The HTML Trouble Ticket Success screen is used by registered ECS users
to ensure successful submission and report Trouble Ticket Id.

From this screen, the user is provided with the following
information/options:

-   Confirmation that the trouble ticket was successfully submitted, the
    trouble ticket identification number, and who submitted the trouble
    ticket.

-   Notification that an E-mail message has been sent to the user
    indicating that a Trouble Ticket has been submitted and when it was
    closed. Selecting [this Trouble Ticket]{.underline} will open the
    Trouble Ticket Detailed Screen.

-   Instructions telling the user how to check the progress of Trouble
    Ticket resolution.

The user also has the choice of returning to the Trouble Ticketing
Homepage or going to the Trouble Ticket Help screen by clicking on the
respective icons at the bottom of the page.

### 8.3.3 ECS Trouble Ticketing HTML List Screen

The HTML Trouble Ticket List screen is used by registered ECS users to
List Trouble Tickets for a user and links the listed Trouble Ticket
Number to the Trouble Ticket Detailed Screen.

Table 8.3.3-1 below provides a description of the Trouble Ticket HTML
List Screen fields.

Table 8.3.3-1. Trouble Ticket HTML List Screen Field Description

  ------------------ ----------- ------ ----------- ---------------------------
  Field Name         Data Type   Size   Entry       Description

  Trouble Ticket     character   15     System      Trouble Ticket Id
  Number                                generated   

  Problem Short      character   125    System      Short Description of
  Description                           generated   Problem

  Status             character   20     System      Status of Trouble Ticket
                                        generated   
  ------------------ ----------- ------ ----------- ---------------------------

The user also has the choice of returning to the Trouble Ticketing
Homepage or going to the Trouble Ticket Help screen by clicking on the
respective icons at the bottom of the page.

### 8.3.4 ECS Trouble Ticketing HTML Detailed Screen

The HTML Trouble Ticket Detailed screen is used by registered ECS users
to see a more detailed output of a Trouble Ticket.

Table 8.3.4-1 below provides a description of the Trouble Ticket HTML
Detailed Screen fields.

Table 8.3.4-1. Trouble Ticket HTML Detailed Screen Field Description

+-----------------+--------+----+---------+--------------------------+
| Field Name      | Data   | Si | Entry   | Description              |
|                 | Type   | ze |         |                          |
+-----------------+--------+----+---------+--------------------------+
| ID              | cha    | 30 | System  | Submitter Id             |
|                 | racter |    | ge      |                          |
|                 |        |    | nerated |                          |
+-----------------+--------+----+---------+--------------------------+
| Name            | cha    | 30 | System  | Submitter Name           |
|                 | racter |    | ge      |                          |
|                 |        |    | nerated |                          |
+-----------------+--------+----+---------+--------------------------+
| E-mail address  | cha    | 64 | System  | Submitter E-mail Address |
|                 | racter |    | ge      |                          |
|                 |        |    | nerated |                          |
+-----------------+--------+----+---------+--------------------------+
| Phone           | cha    | 30 | System  | Submitter Phone Number   |
|                 | racter |    | ge      |                          |
|                 |        |    | nerated |                          |
+-----------------+--------+----+---------+--------------------------+
| Home DAAC       | cha    | 60 | System  | Submitter Home DAAC      |
|                 | racter |    | ge      |                          |
|                 |        |    | nerated |                          |
+-----------------+--------+----+---------+--------------------------+
| Status          | sel    | 4  | System  | Status of Trouble Ticket |
|                 | ection |    | ge      |                          |
|                 |        |    | nerated |                          |
+-----------------+--------+----+---------+--------------------------+
| Impact          | sel    | 4  | System  | Impact to Submitter      |
|                 | ection |    | ge      | (low, medium, high)      |
|                 |        |    | nerated |                          |
+-----------------+--------+----+---------+--------------------------+
| Short           | cha    | 1  | System  | Short description of     |
| description     | racter | 25 | ge      | problem                  |
|                 |        |    | nerated |                          |
+-----------------+--------+----+---------+--------------------------+
| Detailed        | cha    | 2  | System  | Long description of      |
| problem         | racter | 45 | ge      | problem                  |
|                 |        |    | nerated |                          |
| description     |        |    |         |                          |
+-----------------+--------+----+---------+--------------------------+
| Log             | cha    | un | System  | Diary of problem         |
|                 | racter | li | ge      | resolution               |
|                 |        | m. | nerated |                          |
+-----------------+--------+----+---------+--------------------------+

The user also has the choice of returning to the Trouble Ticketing
Homepage or going to the Trouble Ticket Help screen by clicking on the
respective icons at the bottom of the page.

### 8.3.5 ECS Trouble Ticketing HTML Help Screen

The HTML Trouble Ticket Help screen is used by registered ECS users to
get help with the HTML screens.

This screen provides general information on the following:

-   Index \-- links that scroll the screen to the Introduction, Submit
    Page, and List Page sections listed below.

-   Introduction -- provides information about the Trouble Ticket Help
    page

-   Menu Page -- describes the Trouble Ticketing Menu page.

-   Submit Page -- describes the Trouble Ticket Submit page.

-   Success Page -- describes the Trouble Ticket Success page.

-   List Page -- describes the Trouble Ticket List page.

-   Detailed Page - describes the Trouble Ticket Detailed page.

## 8.4 Emergency Fixes

Any emergency may be dealt with on an ad hoc basis, but contingency
plans, contact points for supervisors, responsible engineers, Sustaining
Engineering Organization, vendors, and general guidelines need to be in
place to provide a common framework for emergency response to
crisis-level situations. Emergency fixes may have already been
implemented on a temporary basis by the Trouble Ticket Review Board with
concurrence from the CCB Chair who later receives the CCR to
document/implement the permanent change. Urgent items will be reviewed
by the next CCB meeting.

The scenario presented below, involves a hardware failure. The problem
needs to be resolved quickly to bring a system back into operation. The
resolution requires emergency replacement of a component that is of a
later version than is contained in the original equipment. The scenario
is summarized in table 8.4-1.

**Scenario --- An Example of an Emergency Change Procedure**

It is 7:00 on a Saturday evening. The DAAC operator detects a problem
with the automated tape library (ATL) and reports the problem to the
Trouble Ticket System. The trouble ticket is routed to the System
Administrator, who confirms that the system will not operate and
notifies the site Maintenance Engineer. After running further
diagnostics, the Maintenance Engineer reports the problem and symptoms
to the OEM's maintenance desk. The original equipment manufacturer (OEM)
maintenance representative arrives and concludes that a controller card
has failed. The only card the OEM has immediately available is of a
later version and no spares are available on site. It will be Monday at
the earliest before a replacement board of the same revision level can
be located. The site maintenance engineer reports this to the operations
Crew Chief (i.e., shift leader) for a decision.

The DAAC cannot afford to have the ATL down until Monday. The Crew Chief
calls the DAAC manager at home, apprises him of the situation, and
obtains approval to replace the board with the later version if tests
conclude that it works properly. The OEM's maintenance representative
installs the board. The site's Sustaining Engineer tests the new
controller board, finds that it works properly, and brings the ATL back
on-line. The Sustaining Engineer generates a CCR to document the
configuration change and the authority for the change. The site
Maintenance Engineer records the replacement action in the Trouble
Ticket, references the related CCR, and closes the trouble ticket. The
site Maintenance Engineer updates the property record with the model,
version, and serial number of the new board.

The site CM Administrator reviews the CCR, determines whether it
requires review by the site CCB, and updates the Baseline Manager with
the new configuration and CCR number authorizing the change. The CCR is
also reviewed by the ECS SEO to assess whether there may be impacts to
the ECS and/or applicability to other sites. The ESDIS CCB is provided
an information copy of the CCR for their review and concurrence.

In the event that it is later discovered that the new version controller
board has adverse impacts when operating in the ECS configuration, a
board of the original version will have to be obtained to replace the
newer version. In such cases, the action will be recorded on a new CCR,
referencing the previous CCR.

Table 8.4-1 summarizes emergency procedures that might be taken during
an after-hours, over-the-weekend emergency hardware failure.

Table 8.4-1. Example of Emergency Change Procedure

  --------------------------------------------- -------------------------
  Operator/User                                 System

  Operator prepares trouble ticket to report    Trouble ticket recorded.
  ATL controller failure.                       

  System Administrator and Maintenance Engineer Diagnosis and vendor call
  confirm ATL controller failure, call ATL      recorded in trouble
  maintenance vendor, report call and time in   ticket.
  Trouble ticket.                               

  Maintenance vendor isolates failure to the    
  controller card. The later version card is    
  the only card available.                      

  Crew Chief notified of situation and decision 
  needed to bring ATL up to full operating      
  capability. Approves use of the newer version 
  card, records decision in the trouble ticket, 
  forwards trouble ticket to Sustaining         
  Engineer.                                     

  Maintenance vendor installs card, tests using 
  hardware diagnostics. Crew Chief authorizes   
  controller to be brought back on-line.        

  Maintenance Engineer records card             Trouble ticket action
  installation by model/version into the        recorded.
  trouble ticket.                               

  Sustaining Engineer reads trouble ticket and  Install action recorded
  prepares for discussion at 8:30 am meeting.   in CCR. CCR routed to the
  Generates a CCR.                              CMA.

  CMA updates site baseline, forwards CCR to    Site ATL baseline updated
  the CCB. When CCB approves the action, CMA    in Baseline Manager. CCR
  closes the CCR.                               closed.

  SEO reviews emergency CCR, checks for         
  applicability to other sites, opens new CCR   
  if other sites require change.                
  --------------------------------------------- -------------------------

## 8.5 Diagnosing Network Problems

Previously supplied by Ken Park.