# Presentation: 125345

## SYSTEM TROUBLESHOOTING

**ECS Release 5A Training**

## Overview of Lesson

**Introduction**
**System Troubleshooting Topics**
**System Performance Monitoring**
**Problem Analysis/Troubleshooting**
**Trouble Ticket (TT) Administration**
**Practical Exercise**

## Objectives

**Overall:  Proficiency in methodology and procedures for system troubleshooting for ECS**
**Conduct system performance monitoring**
**Perform COTS problem analysis and troubleshooting**
**Prepare Hardware Maintenance Work Order**
**Perform Failover/Switchover**
**Perform general checkout and diagnosis of failures related to operations with ECS custom software**
**Set up trouble ticket users and configuration**

## Importance

**DAAC Computer Operator, System Admin-istrator, and Maintenance Coordinator**
**SOS/SEO System Administrator, System Engineer, System Test Engineer, and Software Maintenance Engineer**
**DAAC System Engineers, System Test Engineers, Maintenance Engineers**

**Lesson helps prepare several ECS roles for effective system troubleshooting, maintenance, and problem resolution:**

## System Performance Monitoring

**Maintaining Operational Readiness**
**System operators -- close monitoring of progress and status**
**Notice any serious degradation of system performance**
**System administrators and system maintenance personnel -- monitor overall system functions and performance**
**Administrative and maintenance oversight of system**
**Watch for system problem alerts**
**Use monitoring tools to create special monitoring capabilities**
**Check for notification of system events**

## Checking Network Health & Status

**HP Open View system management tool**
**Site-wide view of network and system resources**
**Status information on resources**
**Event notifications and background information**
**Operator interface for starting servers and managing resources**
**HP Open View monitoring capabilities**
**Network map showing elements and services with color alerts to indicate problems**
**Indication of network and server status and changes**
**Creation of submaps for special monitoring**
**Event notifications**

## HP Open View Network Maps

## Network Discovery and Status

**HP OpenView discovers and maps network and its elements**
**Configured to display status**
**Network maps set for read-write**
**IP Map application enabled**
**HP OpenView Network Node Manager start-up**
**Network management processes must be running**
**ovwdb, trapd, ovtopmd, ovactiond, snmpCollect, netmon**
**Check using the command:  ovstatus**
**Status categories**
**Administrative:  Not propagated**
**Operational:  Propagated from child to parent**
**Compound Status:  How status is propagated**

## HP OpenView Default Status Colors

## Monitoring:  Check for Color Alerts

**Open a map**
**Compound Status set to default**
**Color indicates operational status**
**Follow color indication for abnormal status to isolate problem**

## Monitoring:  Check for New Nodes

**IP Map application enabled**
**Automatic discovery of IP-addressable nodes**
**Creation of object for each node**
**Creation and display of symbols**
**Creation of hierarchy of submaps**
**Internet submap**
**Network submaps**
**Segment submaps**
**Node submaps**
**Autolayout**
**Enabled:  Symbols on map**
**Disabled:  Symbols in New Object Holding Area**

## Monitoring:  Special Submaps

**Logical *****vs*****. physical organization**
**Create map tailored for special monitoring purpose**
**Two types and access options**
**Independent of hierarchy, opened by menu and dialog**
**Child of a parent object, accessible through symbol on parent**

## Monitoring:  Event Notifications

**Monitoring:  Event Notifications**

**Event:  a change on the network**
**Registers in appropriate Events Browser window**
**Button color change in Event Categories window**

**Event Categories**
**Error events**
**Threshold events**
**Status events**
**Configuration events**
**Application alert events**
**ECS events (various)**
**All events**

## Accessing the EBnet Web Page

**EBnet is a WAN for ECS connectivity**
**DAACs, EDOS, and other EOSDIS sites**
**Interface to NASA Internet (NI)**
**Transports spacecraft command, control, and science data**
**Transports mission critical data**
**Transports science instrument data and processed data**
**Supports internal EOSDIS communications**
**Interface to Exchange LANs**
**EBnet home page URL**
**http://bernoulli.gsfc.nasa.gov/EBnet/**

## EBnet Home Page

## Analysis/Troubleshooting: System

**COTS product alerts and warnings**
**(e.g., HP OpenView, AutoSys/Xpert)**
**COTS product error messages and event logs**
**(e.g., HP OpenView)**
**ECS Custom Software Error Messages**
**Listed in 609-CD-500-001**

## Systematic Troubleshooting

**Thorough documentation of the problem**
**Date/time of problem occurrence**
**Hardware/software**
**Initiating conditions**
**Symptoms, including log entries and messages on GUIs**
**Verification**
**Identify/review relevant publications (e.g., COTS product manuals, ECS tools and procedures manuals)**
**Replicate problem**
**Identification**
**Review product/subsystem logs**
**Review ECS error messages**
**Analysis**
**Detailed event review (e.g., HP OpenView Event Browser)**
**Troubleshooting procedures**
**Determination of cause/action**

## ECS Assistant vs. HP OpenView

**HP OpenView**
**Dynamic**
**Real-time**
**SNMP**
**Only one instance at a time can be used to manage system, and ECS implementation is somewhat limited**
**ECS Assistant**
**Independently available at each host**
**Limited Monitoring (i.e., server status)**

## ECS Assistant Manager Windows

## ECS Assistant Monitor Windows

**ECS Assistant Monitor Windows**

## Analysis/Troubleshooting: Hardware

**ECS hardware is COTS**
**System troubleshooting principles apply**
**HP OpenView for quick assessment of status**
**HP OpenView Event Log Browser for event sequence**
**Initial troubleshooting**
**Review error message against hardware operator manual**
**Verify connections (power, network, interface cables)**
**Run internal systems and/or network diagnostics**
**Review system logs for evidence of previous problems**
**Attempt system reboot**
**If problem is hardware, report it to the DAAC Mainten-ance Coordinator, who prepares a maintenance Work Order using ILM software**

## XRP-II Main Screen

## XRP-II ILM Hierarchical Menu Structure

**ILM Main Menu**

## ILM Work Order Entry Screen

**ILM Work Order Entry Screen**

## Hardware Problems:  (Continued)

**Difficult problems may require team attack by Maintenance Coordinator, System Administrator, and Network Administrator:**
**specific troubleshooting procedures described in COTS hardware manuals**
**non-replacement intervention (e.g., adjustment)**
**replace hardware with maintenance spare**
**locally purchased (non-stocked) item**
**installed spares (e.g., RAID storage, power supplies, network cards, tape drives)**

## Hardware Problems:  (Continued)

**If no resolution with local staff, maintenance support contractor may be called**
**Update ILM maintenance record with problem data, support provider data**
**Call technical support center**
**Facilitate site access by the technician**
**Update ILM record with data on the service call**
**If a part is replaced, additional data for ILM record**
**Part number of new item**
**Serial numbers (new and old)**
**Equipment Identification Number (EIN) of new item**
**Model number (Note:  may require CCR)**
**Name of item replaced**

## ILM Work Order Modification

**Completion of Work Order Entry copies active children of parent EIN into the work order**
**Use Work Order Modification screen to enter down times, and vendor times and notes**
**From Work Order Modification screen, Items Page is used to record details**
**Which item (or items) failed**
**New replacement items**
**Notes concerning the failure**

## Non-Standard Hardware Support

**For especially difficult cases, or if technical support is unsatisfactory**
**Escalation of the problem**
**Obtain attention of support contractor management**
**Call technical support center**
**Time and Material (T&M) Support**
**Last resort for mission-critical repairs**

## Failover/Switchover

**Hardware consists of one pair of SGI servers (e.g., ICL - Ingest Server)**

**One server in the pair acts as the “hot” server, the other is a “warm” standby backup**

**RAID device between the two servers is Dual Ported to both machines (each machine “sees” the entire RAID); a “virtual IP” is established**

## Preventive Maintenance

**Elements that may require PM are the EMASS/STK robot, tape drives, stackers, printers**
**Scheduled by local Maintenance Coordinator**
**Coordinated with maintenance organization and using organization**
**Scheduled to be performed by maintenance organization  and to coincide with any corrective maintenance if possible**
**Scheduled to minimize operational impact**
**Documented using ILM Preventive Maintenance record**

## Troubleshooting COTS Software

**Issues**
**Software use licenses**
**Obtaining telephone assistance**
**Obtaining software patches**
**Obtaining software upgrades**
**Vendor support contracts**
**First year warranty**
**Subsequent years contracts**
**Database at ILS office**
**Contact ILS Logistics Engineer**
**E-mail:  ilsmaint@eos.hitc.com**
**Telephone:  1-800-ECS-DATA (327-3282)**

## COTS Software Licenses

**Maintained in a property database by ECS Property Administrator**

## COTS Software Installation

**COTS software is installed with any appropriate ECS customization**
**Final Version Description Document (VDD) available**
**Any residual media and commercial documentation should be protected (e.g., stored in locked cabinet, with access controlled by on-duty Operations Coordinator)**

## COTS Software Support

**Systematic initial troubleshooting**
**Software Event Browser (e.g., HP OpenView Event Browser) to review event sequence**
**Review error messages, prepare Trouble Ticket (TT)**
**Review system logs for previous occurrences**
**Attempt software reload**
**Report to Maintenance Coordinator (forward TT)**
**Additional troubleshooting**
**Procedures in COTS manuals**
**Vendor site on World Wide Web**
**Software diagnostics**
**Local procedures**
**Adjustment of tunable parameters**

## COTS Software Support (Cont.)

**Organize available data, update TT**
**Locate contact information for software vendor technical support center/help desk (telephone number, name, authorization code)**
**Contact technical support center/help desk**
**Provide background data**
**Obtain case reference number**
**Update TT**
**Notify originator of the problem that help is initiated**
**Coordinate with vendor and CM, update TT**
**Work with technical support center/help desk (e.g., troubleshooting, patch, work-around)**
**CCB authorization required for patch**

## COTS Software Support (Cont.)

**Escalation may be required, e.g., if there is:**
**Lack of timely solution**
**Unsatisfactory performance of technical support center/help desk**
**Notify SOS/SEO**
**Senior Systems Engineers**
**ILS Logistics Engineer coordination for escalation within vendor organization**

## Troubleshooting of Custom Software

**Code maintained at ECS Development Facility**
**ClearCase******** for library storage and maintenance**
**Sources of maintenance changes**
**ESDIS CCB directives**
**Site-level CCB directives**
**Developer modifications or upgrades**
**Trouble Tickets**

## Implementation of Modifications

**Responsible Engineer (RE) selected by each ECS organization**
**SOS RE establishes set of CCRs for build**
**Site/Center RE determines site-unique extensions**
**System and center REs establish schedules for implementation, integration, and test**
**SOS RE maintains CCR lists and schedule**
**SOS RE maintains VDD**
**RE or team for CCR at EDF obtains source code/files, implements change, performs programmer testing, updates documentation **

## Custom Software Support

**Science software maintenance not responsibility of ECS on-site maintenance engineers**
**Sources of Trouble Tickets for custom software**
**Anomalies**
**Apparent incorrect execution by software**
**Inefficiencies**
**Sub-optimal use of system resources**
**TTs may be submitted by users, operators, customers, analysts, maintenance personnel, management**
**TTs capture supporting information and data on problem**

## Custom Software Support (Cont.)

**Troubleshooting is ad hoc, but systematic**
**For problem caused by non-ECS element, TT and data are provided to maintainer at that element**

## General ECS Troubleshooting

**    ****(Note:  Lesson Guide has introduction and summary table prior to general procedure)**
**Source of problem likely to be specific operations**
**General approach starts with checks of communications, DCE, database accessibility**
**Series of checks on basic operational capabilities**
**Servers can be started from HP OpenView**
**ESDTs are installed and the collections are mapped**
**There is connectivity between the V0 Gateway (V0GTWY) and Data Dictionary (DDICT)**
**Science Data Server (SDSRV) can notify Subscription Server (SBSRV) to trigger subscriptions**
**Storage Management (STMGT) is inserting and archiving granules correctly; SDSRV is storing associated metadata**

## General ECS Troubleshooting (Cont.)

**Checks on basic capabilities**** (Cont.)**
**PGEs are registered, Production Requests can be created, needed inputs are available, and a production plan can be created and activated**
**Ingest functions are working**
**FTPpush distribution is working**
**A Delivered Algorithm Package (DAP) can be inserted and acquired**
**Science Software Integration and Test (SSI&T) tools and Graphical User Interfaces (GUIs) are working**
**The Quality Assurance (QA) Monitor function is working**
**FTPpull distribution is working**
**Functions to locate data using data search are working**
**Functions associated with data order by a registered user are working**

## General ECS Troubleshooting (Cont.)

## HP OpenView:  System/Server Checks

**HP OpenView:  System/Server Checks**

**Root Window**

**Service Submap Window**

## HPOV:  System/Server Checks (Cont.)

**HPOV:  System/Server Checks (Cont.)**

**Mode Submap Window**

**OPS Mode Submap Window**

## HPOV:  System/Server Checks (Cont.)

**HPOV:  System/Server Checks (Cont.)**

**Server Status Submap Window**

**Zoom View, OPS Mode**
**    ****Submap Window**

## Problem Starting Server with HPOV

**Requirements for Using HPOV to start server:**
**EcMsAgDeputy (Deputy Agent, on HPOV host)**
**EcMsCmEcsd (Start Daemon, on HPOV host)**
**EcMsAgSubAgent (SubAgent, on host for server to be started)**
**Alternatives if HPOV does not start server:**
**Use ECS Assistant**
**Use command line**

## General ECS Troubleshooting (Cont.)

**General ECS Troubleshooting (Cont.)**

## Connectivity/DCE Problems

**ECS depends on communications in a Distributed Computing Environment (DCE)**
**Review of server log files may point the way**
**Both the called server and the calling server**
**Ensure servers are up**
**Ping by name**
**Run *****dceverify***** and** ***dcestatus***
**Use *****cdsbrowser***** to check DCE entries for a server**
**Ensure that the DCE entry being used by the calling server/client matches the DCE entry for the called server **

## Cdsbrowser Screens

## General ECS Troubleshooting (Cont.)

**General ECS Troubleshooting (Cont.)**

## Database Access Problems

**Most ECS data stores use the Sybase database engine**
**Sybase hosts listed in Document 920-TD*****x*****-009****(*****x *****= E for EDC, = G for GSFC, = L for LaRC, ****= N for NSIDC)**
**On Sybase host, *****ps -ef | grep dataserver***** and *****ps -ef | grep sqs***** to check that SQS was started after Sybase dataserver processes (*****Note*****:  This applies only to host for SDSRV database)**
**On application host, *****grep Sybase *****<logfilename> to check for Sybase errors**

## General ECS Troubleshooting (Cont.)

## Mount Point/File Access Problems

**ECS depends on remote access to files**
**Ensure file is present in path where a client is seeking it**
**Ensure correct file permissions**
**Check for lost mount point and re-establish if necessary**
**Engineering Technical Directive:  *****NFS Mount Point Installation/Update Standard Procedure***

## General ECS Troubleshooting (Cont.)

**General ECS Troubleshooting (Cont.)**

## ESDT Problems

**Each ECS data collection is described by an ESDT**
**Descriptor file has collection-level metadata attributes and values, granule-level metadata attributes (values supplied by PGE at run time), valid values and ranges, list of services**
**Check SDSRV GUI to ensure ESDT is installed**
**Check SBSRV GUI to ensure events are registered**
**Check that IOS and DDICT are installed and up**
**Check SDSRV GUI for event registration in ESDT Descriptor information**
**Check log files for errors in communication between SDSRV, IOS, SBSRV, and DDICT**
**If necessary, perform collection mapping for DDICT**

## General ECS Troubleshooting (Cont.)

## V0GTWY/DDICT Connectivity Problem

**ECS Data Search depends on correct functioning of V0GTWY and DDICT, communication, and ****up-to-date mapping of valid attributes and keywords**
**Review V0GTWY log to check that V0GTWY is using a valid *****isql***** query**
**Ensure compatibility of collection mapping database used by DDICT and the EOS Data Gateway Web Client search tool**
**If necessary, perform collection mapping for DDICT (using DDICT Maintenance Tool)**
**Contact EOSDIS V0 Information Management System to check status of any recently exported ECS valids**

## General ECS Troubleshooting (Cont.)

## SBSRV Problem

**SBSRV plays key role in many ECS functions**
**Ensure SBSRV is up and listening**
**Use SBSRV GUI to add a subscription for FTPpush of a small data file**
**Get Sybase username and password from EcSbSubServer.CFG, then attempt to log in to Sybase (on the SBSRV database host) with that username and password**

## General ECS Troubleshooting (Cont.)

## Granule Insertion Problems

**ECS depends on successful archiving functions**
**Check SDSRV logs and Archive Server logs for communications errors**
**List files in Archive to check for file insertion (/dss_stk1/<*****mode*****>/<*****data_type_directory*****>)**
**Database Administrator check SDSRV Inventory database for file entry**
**Check mount points on Archive and SDSRV hosts**
**If dealing with Ingest, check for staging disk in *****drp-***** or *****icl-*****mounted staging directory**
**Archive Manager check volume group set-up and status**

## Granule Insertion Problems (Cont.)

**Check SDSRV and SBSRV logs to ensure that subscription was triggered by the insertion**
**Check SBSRV log for Acquire request to SDSRV**
**Check DDIST log for sending of e-mail notification to user**
**Check for Acquire failure**
**Check SDSRV GUI for receipt of Acquire request**
**Check SDSRV logs for Acquire activity**
**Check Archive Server log for Acquire activity**
**Check DDIST staging area for file and metadata**
**Check Staging Disk log for Acquire activity errors**
**Check space available in the staging area on the DDIST server**

## General ECS Troubleshooting (Cont.)

**General ECS Troubleshooting (Cont.)**

## PDPS Plan Creation/Activation and PGE Problems

**Production Planning and Processing depend on registration and functioning of PGEs, and on data insertion and archiving**
**Initial troubleshooting by PDPS personnel**
**Check logs for evidence of communications problems between PDPS and SDSRV**
**Have PDPS check for failed PGE granule; refer problem to SSI&T?**
**Insert small file and check for granule insertion problems**
**Check that PDPS mount point is visible on SDSRV and Archive Server hosts**

## PDPS Plan Creation/Activation and PGE Problems (Cont.)

**Have PDPS create and activate a plan for sample PGEs (e.g., ACT and ETS)**
**Ensure necessary input and static files are in SDSRV**
**Ensure necessary ESDTs are installed**
**Ensure there is a subscription for output (e.g., AST_08)**
**Check for PDPS run-time directories**
**Determine if the user in the subscription received e-mail concerning the FTPpush**
**Determine if the files were pushed to the correct directory**
**Execute *****cdsping***** of machines with which DDIST communicates from *****x*****0dis02**

## General ECS Troubleshooting (Cont.)

## Ingest Problems

**Ingest problems vary depending on type of Ingest**
**Ingest GUI should be the starting point; Ingest technician/Archive Manager may resolve many Ingest problems (e.g., Faulty DAN, Threshold problems, disk space problems, FTP error, Ingest processing error)**
**Have technician perform a test ingest of appropriate type**
**Check for granule insertion problems**
**Check Archive and Inventory databases for appropriate entries**

## General ECS Troubleshooting (Cont.)

**General ECS Troubleshooting (Cont.)**

## Problems with FTPpush Distribution

**FTPpush process is central to many ECS functions**
**Use DDIST GUI to determine if DDIST is handling a request for the data, and to monitor progress**
**Check server logs (FtpDis, DDIST) to ensure file was pushed to correct directory**
**Check that the directory exists**
**Check FtpDis logs for permission problems**
**Check for Archive Server staging of file; check staging disk space**
**Check server logs to find where communication broke down**

## General ECS Troubleshooting (Cont.)

## Problems with DAP Insertion/Acquire and SSI&T Tools/GUIs

**Delivered Algorithm Packages (DAPs) are the means to receive new science software**
**Check that Algorithm Integration and Test Tools (AITTL) are installed**
**Check that ESDTs are installed**
**Check for granule insertion problems**
**Check archive for presence of the DAP**
**Check for problems with FTPpush distribution**

## General ECS Troubleshooting (Cont.)

## QA Monitor Problems

**QA Monitor GUI is used to record the results of a QA check on a science data product (update QA flag in the metadata)**
**Operator may handle error messages identified in Operations Tools Manual (Document 609)**
**Check that the data requested are in the Archive**
**Check SDSRV logs to ensure that the data query from the QA Monitor was received**
**Check QA Monitor GUI log to determine if the query results were returned**
**If not, check SDSRV logs for communications errors**

## General ECS Troubleshooting (Cont.)

**General ECS Troubleshooting (Cont.)**

## Problems with FTPpull Distribution

**FTPpull is key mechanism for data distribution**
**Use DDIST GUI to determine if DDIST is handling a request for the data, and to monitor progress**
**Check that the directory to which the files are being pulled exists**
**Check FtpDis logs for permission problems**
**Check for Archive Server staging of file**
**Check server logs to find where communication broke down**
**Execute *****cdsping***** of machines with which DDIST communicates from *****x*****0dis02**

## General ECS Troubleshooting (Cont.)

**General ECS Troubleshooting (Cont.)**

## Data Search Problems

**Data Search functions are key to user access**
**List files in Archive to check for presence of file (/dss_stk1/<*****mode*****>/<*****data_type_directory*****>)**
**Check SDSRV logs for problems with search**
**Check V0GTWY/DDICT connectivity**

## General ECS Troubleshooting (Cont.)

**General ECS Troubleshooting (Cont.)**

## Data Order Problems

**Registered user must be able to order products**
**Check for data search problems**
**Use DDIST GUI to determine if DDIST is handling a request for the data, and to monitor progress**
**Determine if the user in the subscription received e-mail notification**
**Check server logs to determine where the order failed; check SDSRV GUI to determine if SDSRV received the Acquire request from V0GTWY**

## Data Order Problems (Cont.)

**Check DDIST staging area for presence of data; check staging disk space**
**Execute *****cdsping***** of machines with which DDIST communicates from *****x*****0dis02**
**Use ECS Order Tracking GUI to check that the order is reflected in MSS Order Tracking; check database**
**If order is for L7 Scene data, check HDFEOS Server .ALOG to determine if the HDFEOS Server received the request**

## Trouble Ticket (TT)

**Documentation of system problems**
**COTS Software (Remedy)**
**Documentation of changes**
**Failure Review Board**
**Emergency fixes**
**Configuration changes ********CCR**

## Using Remedy

**Creating and viewing Trouble Tickets**
**Adding users to Remedy — Database Administrator**
**Controlling and changing privileges in Remedy — Configuration Management Administrator**
**Modifying Remedy’s configuration  — Configuration Management Administrator.**
**Generating Trouble Ticket reports — System Administrator, others**

## Remedy RelB-User Schema Screen

## Adding Users to Remedy

**Status**
**License Type **
**Login Name**
**Password**
**Email Address**
**Group List**

**Full Name**
**Phone Number**
**Home DAAC**
**Default Notify Mechanism**
**Full Text License**
**Creator**

## Changing Privileges in Remedy

**Access privileges (for fields)**
**View**
**Change**
**Privilege change methods**
**Change group assignment**
**Change privileges of a group**
**Use Admin tool to define group access for schemas (Remedy databases)**

## Remedy Admin Tool - Schema List

## Remedy Admin - Group Access

## Remedy Admin - Modify Schema

## Changing Remedy Configuration

**User Contact Log, Category**
**User Contact Log, Contact Method**
**Configuration Item (CI)**

## Remedy Admin - Modify Menu

## Generating Trouble Ticket Reports

**Assigned-to Report**
**Average Time to Close TTs**
**Hardware Resource Report**
**Number of Tickets by Status**
**Number of Tickets by Priority**
**Review Board Report**
**SMC TT Report**
**Software Resource Report**
**Submitter Report**
**Ticket Status Report**
**Ticket Status by Assigned-to**

## Remedy Admin - Reports

## Operational Work-around

**Managed by the ECS Operations Coordinator at each center**
**Master list of work-arounds and associated trouble tickets and configuration change requests (CCRs) kept in either hard-copy or soft-copy form for the operations staff**
**Hard-copy and soft-copy procedure documents are “red-lined” for use by the operations staff**
**Work-arounds affecting multiple sites are coordinated by the ECS organizations and monitored by ECS M&O Office staff**