IBM Health Checks and Introduction to Troubleshooting on a PureApplication System - United States  TECHNOTE (TROUBLESHOOTING)

PROBLEM(ABSTRACT)
 This document describes how to check the health of your system, and when needed, where to start troubleshooting a problem on a PureApplication System. 

RESOLVING THE PROBLEM
Use this document to confirm the health of your system. 

Consider forwarding the PureApplication System Events to an external monitoring system, such as IBM Netcool/OMIbus. and enabling the call home feature. 

 

With PureApplication System firmware versions 2.1.0.1 or newer you can use the System Health check report to see a consolidated view of some of the PureApplication System UI pages. 

System > System Troubleshooting, click on System Health Check. 
[/support/docview.wss?uid=swg21675742&aid=1] [/support/docview.wss?uid=swg21675742&aid=1] 

The Lite Health Checks section lists a set of items which you can review every couple of days. These checks take about 15 minutes. 

Full Health Checks lists a set items you can review every week to help you confirm your system is operational. There are more checks in this section. 

Introduction to Troubleshooting and Contacting IBM Support and sending in your information are included to help when the information found during a health check is not as expected. 

The checks include accessing information on both the System and Workload console. You will need System and Workload administration access. 



LITE HEALTH CHECKS 
Along with forwarding the events to a monitoring system, you might want to check the following items every couple of days. This should take about 15 minutes. 

If the information displayed is not as expected, expand all sections of the web page and copy the page to a word processing document. Proceed to the Contacting IBM Support and sending in your information section of this document. Start with Step 1 where you will add this information from a word processing document. 

 

 

Check Web Console Page Description High Availability Status of the system System Console > Troubleshooting, then expand the High Availability section. Under Service Code, the overall status should be "Online"when this feature is working properly. Problems System Console > System > Problems This view should be empty. However, if problems have been identified by the system, they will be listed here. 

You can sort problems by any of the column headers. The "Created on" date is usually the most helpful. Investigate any problems that are reported here. When the issue has been resolved or determined to not be a real problem, delete the records. 

Problems are not automatically deleted from this view. It is important to delete problems on resolution. Problems are often (but not always) associated with Events. There may be some overlap between information in this view and that presented in the Event Viewer.
Search the Support Portal [http://www.ibm.com/support/entry/portal/product/puresystems/pureapplication_system]for content that may help you avoid opening a service request. Events System Console > System > Events Events which have been identified by the system will be listed here. You can sort Events by any of the column headers but the "Updated on" and "Severity" are usually the most helpful. Some of the events are from problems experienced by the workload users and are critical to that workload but not to the entire system. Look for "critical" and "Call HOME events" 
Search the Support Portal [http://www.ibm.com/support/entry/portal/product/puresystems/pureapplication_system]for content that may help you avoid opening a service request.

Compute Nodes System Console > Hardware > Compute Nodes Click on any Errors or Warning Events and review the "Health statistic" and other information on these pages. 

Check that none of the nodes are in quiesced or stopped state. Click on each compute node in the system. Look for the word "Temperature". There are 2 values. Ambient Temperature and Maximum Ambient Temperature. Hover over the 'temperature icons' for these field to see the expected temperature range. 


On the W1700, a compute node with an LED on might be an indication of a AIX Operation System problem on one if the Virtual Machines which is or was running. See: Node <node_nn> message: (System Event) system hardware failure, CMM Event: 77777703, Node Event: B200A101, Check log (Informational LED) is lit on compute node. [http://www.ibm.com/support/docview.wss?uid=swg21669085] This is not a hardware problem



Management Nodes System Console > Hardware > Management Nodes Look at the status of elements and events associated with the two PureSystems Manager nodes and the other two. 

Click on any Errors or Warning Events and review the "Health statistic" and other information on these pages. 

One of the PureSystem Managers should be marked as the leader with the word "Primary" in the "Type" field. 

Any errors that are listed for management nodes can indicate significant problems. Job Queue System Console > System > Job Queue. You want to ensure that jobs are starting and finishing on this page. 

Check the Started Queue for failures. 

Click the check-box to show internal jobs to get a complete picture of the job queue. 

It is quickest to sort by "Status" which will pull all failures to the top. Failures of individual deployment jobs should be reviewed with the team managing the system workload. Scan the information to look for pervasive system issues here. Failed Deployments Workload Console > Instances > All Instances Look over the list of all instances to find those that are failed that appear to be stuck initializing. 

A large number of deployments in these failed or stalled states can indicate system wide problems that should be investigated. A few failed or error deployments will occasionally be there because of the way that these systems are being used by the team managing the work loads. Confirm all is well with the team working on these projects. 

Scan the information to look for pervasive system issues here. FULL HEALTH CHECKS 


Along with forwarding the events to a monitoring system, you might want to check the following items every week. The "Lite Health Check" are included in the "Full Health Check" list below. 

If the information displayed is not as expected, expand all sections of the web page and copy the page to a word processing document. Proceed to the Contacting IBM Support and sending in your information section of this document. Start with Step 1 where you will add more information to the word processing document. 


Check Web Console Page Description High Availability Status of the system System Console > Troubleshooting, then expand the High Availability section. Under Service Code, the overall status should be "Online" when this feature is working properly. Temperature System Console > Hardware > Infrastructure Map Click on the "Temperature" in the top menu bar. 

 




Check the temperature values for the system Hover over the value to see acceptable range for this value. When checking component status in the following checks, look at the Temperature readings for the component. 

Then check the temperature of each component in the following systems using the information under "Component Status" below. 

Prior to V2.0, the temperature check on the storage node component is incorrect. Check for events concerning storage node (V7000 Storewize ) component instead. 



Component Status System Console > Hardware > Infrastructure Map 


For the components in the system, check the status shown on the web page. 

If there is a red "!" , in the picture of the rack, click on the icon. The "Default section" will be presented with more information. If there are numbers after the red "!", click on the number and review the events posted. Check the "Updated on" column to see when the event was posted. 

[/support/docview.wss?uid=swg21675742&amp;aid=2] System Console > Hardware > Infrastructure Map Click on "Switch to Tree View" 

Flex Chassis > 

Chassis Management Module

Check that there are 2 Chassis Management Modules (CMM) and a primary and secondary listed for each chassis. Look for any alerts and review the information shown on the web page. 
Click on each Chassis in the system. Look for the word "Temperature". There are 2 values. Ambient Temperature and Maximum Ambient Temperature. Hover over the 'temperature icons' for these field to see the expected temperature range. 

[/support/docview.wss?uid=swg21675742&amp;aid=2] System Console > Hardware > Compute Nodes Click on any Errors or Warning Events and review the health statistic and other information on these pages. 

Check that none of the nodes are in quiesced or stopped state. 
Click on each compute node in the system. Look for the word "Temperature". There are 2 values. Ambient Temperature and Maximum Ambient Temperature. Hover over the 'temperature icons' for these field to see the expected temperature range. 

 

On the W1700, or W2700, a compute node with an LED on might be an indication of a AIX Operation System problem on one if the Virtual Machines which is or was running. See: Node <node_nn> message: (System Event) system hardware failure, CMM Event: 77777703, Node Event: B200A101, Check log (Informational LED) is lit on compute node. [http://www.ibm.com/support/docview.wss?uid=swg21669085] This is not a hardware problem

[/support/docview.wss?uid=swg21675742&amp;aid=2] System Console > Hardware > Infrastructure > Management Node

Check the management nodes to make sure they are powered on and available. 

One of the PureSystem Managers should have 'primary' in the type field. 

Review the "Health statistic" 

Look at the status of elements and events associated with the two PureSystems Manager nodes and the two Manager nodes. Click on any Errors or Warning Events and review the health statistic and other information on these pages. 

One of the PureSystem Managers should be marked as the leader with "Primary" in the "Type" field. 

Any errors that are listed for any management nodes can indicate significant problems. [/support/docview.wss?uid=swg21675742&amp;aid=2] System Console > Hardware > Storage Devices Look at all the Storage Nodes and Storage Node Expansions to make sure the Disk Drives, LUNs, and Storage Controller Ports are all available for each node. Prior to V2.0, the temperature check on the storage node component is incorrect. Check for events concerning storage node (V7000 Storewize ) component instead. 

In V2. the temperature is under the "Canister" section. Hover over the temperature gauge icon to confirm the temperatures are normal. 

Problems System Console > System > Problems This view should be empty. 

However, if problems have been identified by the system, they will be listed here. You can sort problems by any of the column headers but the "Created on" date is usually the most helpful. Investigate any problems that are reported here and when the issue has been resolved or determined to not be a real problem, delete the records. 

Problems are not automatically deleted from this view. It is important to delete them on resolution. Problems are often (but not always) associated with Events. There may be some overlap between information in this view and that presented in the Event Viewer. Events System Console > System > Events Events which have been identified by the system they will be listed here. You can sort Events by any of the column headers but the "Updated on" and "Severity" are usually the most helpful. Some of the events are from problems experienced by the workload users and are critical to that workload but not to the entire system. 

Job Queue System Console > System > Job Queue. You want to ensure that jobs are starting and finishing on this page. 

Check the Started Queue for failures. 

Click the check-box to show internal jobs to get a complete picture of the job queue. It is quickest to sort by Status which will pull all failures to the top. Failures of individual deployment jobs should be reviewed with the team managing the system workload. Scan the information to look for pervasive system issues here. Failed Deployments Workload Console > Instances > All Instances Look over the list of all instances to find those that are failed that appear to be stuck initializing. 

A large number of deployments in these failed or stalled states can indicate system wide problems that should be investigated. A few failed or error deployments will occasionally be there because of the way that these systems are being used by the team managing the work loads. Confirm all is well with the team working on these projects. Validate Shared Services Workload Console > Instances > Shared Services Validate that shared services are running for all appropriate cloud groups and appear to be in good health. Work with the local patterns administrator to confirm the list of shared services that should be running for this system. 

NOTE: It is not unusual for shared services to be stopped or not deployed for various reasons depending upon the use of the system. Check with the patterns administrator on the expected status of shared services. INTRODUCTION TO TROUBLESHOOTING 


 1.  The first action is to open up a word processing document to capture the output of any the following steps which shows the problem. Use this technote entitled Organize your problem information [http://www.ibm.com/support/docview.wss?uid=swg21654210] concerning the problem for IBM Support. Copy this template into your word processing document. 
 2.  Check System Console > System > Problems This view should be empty.  1. However, if problems have been identified by the system, they will be listed here. You can sort problems by any of the column headers but the 
         "Created on" is usually the most helpful. Investigate any problems that are reported here and when the issue has been resolved or determined to not be a real problem, delete the record. 
      2. Problems are not automatically deleted from this view. It is important to delete entries on resolution. Problems are often (but not always) associated with Events. There may be some overlap between information in this view and that presented in the Event Viewer. 
      3. Search the support portal [http://www.ibm.com/support/entry/portal/product/puresystems/pureapplication_system] [http://www.ibm.com/]to see if there is an explanation of the message in the product documentation center, a technote, or APAR document. 
      4. Otherwise, if you have problem listed which you would like IBM to investigate, export the problem. Add this to the document created with the organize your problem information [http://www.ibm.com/support/docview.wss?uid=swg21654210] technote. Check the MustGather Read first for PureApplication System [http://www.ibm.com/support/docview.wss?uid=swg21610259] for information to gather. Proceed to the Contacting IBM Support and sending in your information section of this document. 
     
     
 3.  Check the event log: System Console > System > Events  1. If Events have been identified by the system, they will be listed here. You can sort Events by any of the column headers but the "Updated on" and "Severity" are usually the most helpful. Investigate any problems. 
      2. Some of the Events are from problems experienced by the workload users and are critical to that workload but not to the entire system. Check with the workload users to see if they are testing or also finding problems. 
      3. Search the support portal [http://www.ibm.com/support/entry/portal/product/puresystems/pureapplication_system] to see [http://www.ibm.com/] if there is an explanation of the message in the product documentation, a technote, or APAR document. 
      4. Check if the Event lists a hardware component and a hardware error. Look for the failing serial number, name or rack location number.  1. This document has a list of Events which you might see. Look under "Check these known problems first" in this technote. MustGather: Chassis Management Module, IBM Storwize V7000, Top of Rack Switch, San Switch, and Management EndPoint problems for IBM PureApplication System [http://www.ibm.com/support/docview.wss?uid=swg21666454] [http://www-01.ibm.com/support/docview.wss?uid=swg21666454] 
          2. Check the status of the hardware component via: System Console > Hardware > Infrastructure Map.Try the "Switch to Tree View" option. 
          3. If the symptoms describe an Event which requires investigation by IBM, save a copy of the information and gather the information listed in MustGather: Chassis Management Module, IBM Storwize V7000, Top of Rack Switch, San Switch, and Management EndPoint problems for IBM PureApplication System [http://www.ibm.com/support/docview.wss?uid=swg21666454] [http://www-01.ibm.com/support/docview.wss?uid=swg21666454]
         
         
      5. Otherwise, if you find an Event which you would like IBM to investigate, save the Event. You can export the one event. Use the organize your problem information [http://www.ibm.com/support/docview.wss?uid=swg21654210] technote to prepare to work with IBM Support. Check the MustGather Read first for PureApplication System [http://www.ibm.com/support/docview.wss?uid=swg21610259] for information to gather. Proceed to the Contacting IBM Support and sending in your information section of this document. 
     
     
 4.  Check the job queue: System Console > System >Job Queue  1. Under "Started Jobs Queue", check that jobs are running. If not, check the "Display Internal Jobs" check box. Refresh the screen using the "double arrows' under the "Started Jobs Queue" line. Confirm that there were internal jobs running, and successfully completing. Wait 10 minutes. If there are no internal jobs running or completing, save a capture of this screen. Then obtain a system collection set for IBM Support. Use the organize your problem information [http://www.ibm.com/support/docview.wss?uid=swg21654210] technnote to prepare to work with IBM Support. Check the MustGather Read first for PureApplication System [http://www.ibm.com/support/docview.wss?uid=swg21610259] for information to gather. Proceed to the Contacting IBM Support and sending in your information section of this document. 
      2. Look at the "Status" column for failed jobs. Wait and check this page again as you want to ensure that jobs are moving in the Started Queue and that there are not a large number of failures. Failures of individual deployment jobs are usually not a concern unless all deployment jobs seem to be failing. 
      3. Consult with your team using the system to deploy business solutions to be sure these are expected failures, or that they are aware and working on the problem. 
      4. To help you debug failed deployments, gather files via the Work Load Console. Check the MustGather Read first for PureApplication System [http://www.ibm.com/support/docview.wss?uid=swg21610259] for information to gather. 
      5. A "Pending" job is not actually running -- it is just queued up to run at some point in the future. The "Internal Backup Job" is queued up to run every day. You will always see this job in the queue as "Pending". 
      6. Otherwise, if you want IBM to investigate, use the organize your problem information [http://www.ibm.com/support/docview.wss?uid=swg21654210] technote to copy the web page for IBM Support. Check the MustGather Read first for PureApplication System [http://www.ibm.com/support/docview.wss?uid=swg21610259] for information to gather. Proceed to the Contacting IBM Support and sending in your information section of this document. 
     
     
 5.  Check the storage nodes: System Console > Hardware > Storage Devices. Look at all the Storage Nodes and Storage Node Expansions to make sure the Disk Drives, LUNs, and Storage Controller Ports are all available for each node.  1. Check the "Total" and "Available" counts for Disk Drives, LUNS, and Ports. The values should be equal. 
      2. If there is a problem, click on the "Generate" button on the "Request System Logs" line on the web page. This technote describes how to save and sending the logs: in MustGather: Chassis Management Module, IBM Storwize V7000, Top of Rack Switch, San Switch, and Management EndPoint problems for IBM PureApplication System. [http://www.ibm.com/support/docview.wss?uid=swg21666454] Proceed to the Contacting IBM Support and sending in your information section of this document.
     
     
 6.  Check the Management Nodes: System Console > Hardware > Management Nodes. Look at the status of elements and events associated with those elements.  1. There are two PureSystems Manage nodes and the two virtualization manager nodes. One of the PureSystem Managers should be marked as the leader with the word "Primary" in the "Type" field. Look for any errors that are listed for any management nodes as they can indicate significant problems. 
      2. If you do see a problem for IBM to investigate, click on the "Generate" button on the "Request System Logs" line on the web page. This technote describes how to save and sending the logs: MustGather: Chassis Management Module, IBM Storwize V7000, Top of Rack Switch, San Switch, and Management EndPoint problems for IBM PureApplication System. [http://www.ibm.com/support/docview.wss?uid=swg21666454] Proceed to the Contacting IBM Support and sending in your information section of this document
     
     
 7.  Check the High Availability Status of the system: System Console > Troubleshooting, then expand the High Availability section.  1. In the table, there are service code and status column. The status column should show a status of "Online" when this feature is working properly. 
      2. If the status is not "Online", paste this RESTAPI into the browser URL and save the output: https://<rack_ip>/admin/resources/service_status Copy this information into the word processing document and Proceed to the Contacting IBM Support and sending in your information section of this document 
     
     
 8.  Check Chassis temperature: System Console > Hardware> Flex Chassis. Open the selection. There are 2 values: "Ambient Temperature" and "Maximum Ambient Temperature". Hover over the 'temperature icons' for these field to make sure the temperature is within range. If there is a red "!" , in the picture of the rack, click on the icon. The "Default section" will be presented with more information.  1. This document has a list of Events which you might see. One is concerning temperature. Look under "Check these known problems first" in this technote: MustGather: Chassis Management Module, IBM Storwize V7000, Top of Rack Switch, San Switch, and Management EndPoint problems for IBM PureApplication System. [http://www.ibm.com/support/docview.wss?uid=swg21666454] 
      2.  If there are numbers after the red "!", click on the number and review the events posted. Check the "Updated on" column to see when the event was posted. 
      3. If you do see a problem for IBM to investigate, click on the "Generate" button on the "Request System Logs" line on the web page. This technote describes how to save and send in the logs: MustGather: Chassis Management Module, IBM Storwize V7000, Top of Rack Switch, San Switch, and Management EndPoint problems for IBM PureApplication System [http://www.ibm.com/support/docview.wss?uid=swg21666454]. Proceed to the Contacting IBM Support and sending in your information section of this document
     
     
 9.  Check the Network Switches: System Console >Hardware-> Networking devices. Click on each switch.  1. Look on the Events line for the red "!". If there is a number, click on the number to investigate the Event posted by the system. 
      2. Confirm the switch shows the "Power Status" and "Status" as "Available". 
      3. If you do see a problem for IBM to investigate, click on the "Generate" button on the "Request System Logs" line on the web page. This technote describes how to save and send in the logs: MustGather: Chassis Management Module, IBM Storwize V7000, Top of Rack Switch, San Switch, and Management EndPoint problems for IBM PureApplication System [http://www.ibm.com/support/docview.wss?uid=swg21666454]. Proceed to the Contacting IBM Support and sending in your information section of this document
     
     
 10. Check the DNS connectivity: System Console > Settings. Expand Domain Service (DNS) and use the "Lookup host name or IP address". Confirm you can connect to the expected IP addresses by IP and hostname. Consult with your team which manages the DNS sever for advice. 
 11. Check for failed deployments: Workload Console > Instances > All Instances.  1. A few failed or error deployments will occasionally be there because of the way that these systems are being used and should not be of concern. Confirm all is well with the team working on these projects. 
      2. A large number of deployments in these failed or stalled states can indicate system wide problems which should be investigated. Check the MustGather Read first for PureApplication System [http://www.ibm.com/support/docview.wss?uid=swg21610259] for information to gather. Proceed to the Contacting IBM Support and sending in your information section of this document. 
     
     
 12. The Troubleshooting chapters in the PureApplication System product documentation has more troubleshooting topics. 
 13. Otherwise, if you see other problems which you would like IBM to investigate, use the organize your problem information [http://www.ibm.com/support/docview.wss?uid=swg21654210] technote to copy the web pages into a document to show the problem to IBM Support. [http://www.ibm.com/support/docview.wss?uid=swg21610259] Check the MustGather Read first for PureApplication System [http://www.ibm.com/support/docview.wss?uid=swg21610259] for information to gather. Proceed to the Contacting IBM Support and sending in your information section of this document. 


 

CONTACTING IBM SUPPORT AND SENDING IN YOUR INFORMATION


Use the following technotes to contact IBM Support and send in the MustGather information: 

 * Contacting IBM PureApplication System Support [http://www.ibm.com/support/docview.wss?uid=swg21609051] to open a PMR. 
 * Use the IBM Secure Diagnostic Data Upload Utility [http://www.ibm.com/de/support/ecurep/send_java-tool.html] also called the Java Utility to upload the PureApplication System [http://www.ibm.com/de/support/ecurep/send.html]collections to greatly reduce the time to upload large files. 
 * Exchanging information with IBM PureApplication System Technical Support for problem determination. [http://www.ibm.com/support/docview.wss?uid=swg21596913] [http://www.ibm.com/support/docview.wss?uid=swg21596913] 
 * Review the PureApplication System Customer Support Plan [http://www-01.ibm.com/support/docview.wss?uid=swg21613951] 
 * Do not send any confidential information from your company.

Question: How can I check the health of my IBM PureApplication System? How can I check the health of my IBM PureApplication System?
Are there steps to verify that my system is healthy? Is this something I can do, or does IBM have to verify the health of my system? 

Answer: Consider forwarding the PureApplication System Events to an external monitoring system, such as IBM Netcool/OMIbus. and enabling the call home feature. 

 

With PureApplication System firmware versions 2.1.0.1 or newer you can use the System Health check report to see a consolidated view of some of the PureApplication System UI pages. 

System > System Troubleshooting, click on System Health Check.