MegaCli

MegaCli is LSI's command line interface to their MegaRAID controller family. Dell uses them in their servers as e.g. "PERC5/i" or similar. Our dataset1 server uses this RAID controller.
The tool can be downloaded from the LSI website pages on broadcom.com, e.g. here 
Version 8. The ZIP file contains a ZIP file which contains two RPM files.
NOTE that this tool is generally installed on all hosts with lsi raid from a debian package now, and is called megacli (note the lower case).
==Installation from the ZIP file==
 apt-get install unzip
 wget http://www.lsi.com/DistributionSystem/AssetDocument/8.00.11_Linux_MegaCLI.zip 
 unzip 8.00.11_Linux_MegaCLI.zip MegaCliLin.zip
 unzip MegaCliLin.zip MegaCli-8.00.11-1.i386.rpm
 
 apt-get install rpm
 cd /
 rpm2cpio ~-/MegaCli-8.00.11-1.i386.rpm | cpio -div
 apt-get install libsysfs2
 ln -s /opt/MegaRAID/MegaCli/MegaCli64 /usr/sbin/MegaCli
==Common Parameters==
Many of the commands of MegaCli make use of the following parameters:
;-aN: Specifies the adapter. Use -a0 for the first adapter, -a1 for the second, or -aALL for all adapters.
;-PhysDrv [E:S]: Specifies a physical drive. E is the enclosure ID, as returned by MegaCli -EncInfo -aALL. For dataset1, the possible enclosure IDs are 16 and 41. If more than one drive has to be specified, the drives are written in the form [E:S, E:S, ...].
;-Lx: Specifies a virtual drive (aka RAID array) (where x is a number starting with zero or the string all).
==Monitoring==
The following commands can be used to check the status of the arrays.
===Information gathering===
The following script can be used to gather useful information about a failed disk for dc-ops
sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli
===Battery status===
 megacli -AdpBbuCmd -aALL
 megacli -AdpBbuCmd -GetBbuStatus -a0 | grep -e '^isSOHGood' -e '^Charger Status' -e '^Remaining Capacity' -e 'Charging'
What is the Default and the Current policy of the RAID controller depending on the BBU status
 megacli -LDInfo -LAll -aAll | grep "Cache Policy:"
Force WriteBack if needed (this should only be done if you know what you are doing (e.g. miss-configuration)- normally WriteThrough is forced when there is a faulty battery/it is discharged. Checking the health/repacing the battery should be the main way to solve this, otherwise an unsafe configuration could cause data loss!):
 megacli -LDSetProp -ForcedWB -Immediate -Lall -aAll
Force the BBU to relearn cycle
 
 megacli -AdpBbuCmd -BbuLearn -aAll
===Array status===
To check the whole array status (ie: we have a RAID10 and we want to check the RAID status we can run)
 megacli -LDInfo -L0 -a0
And that would let us know the RAID status and the information about it (and its SPANs)
 Adapter 0 -- Virtual Drive Information:
 Virtual Drive: 0 (Target Id: 0)
 Name :
 RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0
 Size : 1.633 TB
 Sector Size : 512
 Mirror Data : 1.633 TB
 State : Optimal
 Strip Size : 256 KB
 Number Of Drives per span:2
 Span Depth : 6
 Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
 Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
 Default Access Policy: Read/Write
 Current Access Policy: Read/Write
 Disk Cache Policy : Disk's Default
 Encryption Type : None
 Bad Blocks Exist: Yes
 Is VD Cached: Yes
 Cache Cade Type : Read Only
As we can see in that RAID we have 6 SPANs and it has 2 disks per SPAN, if we are interested in checking the SPAN status (to see which of the SPANs is degraded)
 megacli -CfgDsply -a0
There we can see the different SPANs, the number of disks, the status of the SPANs and the information about the disks that belong to each SPAN.
(in this case we have two as it says on PDs: 2 and those disks are Phisicaly Disk 0 and Phisical Disk 1)
 SPAN: 5
 Span Reference: 0x05
 Number of PDs: 2
 Number of VDs: 1
 Number of dedicated Hotspares: 0
 Virtual Drive Information:
 Virtual Drive: 0 (Target Id: 0)
 Name :
 RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0
 Size : 1.633 TB
 Sector Size : 512
 Mirror Data : 1.633 TB
 State : Optimal
 Strip Size : 256 KB
 Number Of Drives per span:2
 Span Depth : 6
 Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
 Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
 Default Access Policy: Read/Write
 Current Access Policy: Read/Write
 Disk Cache Policy : Disk's Default
 Encryption Type : None
 Bad Blocks Exist: Yes
 Is VD Cached: Yes
 Cache Cade Type : Read Only
 Physical Disk Information:
 Physical Disk: 0
 Enclosure Device ID: 32
 Slot Number: 10
 Drive's position: DiskGroup: 0, Span: 5, Arm: 0
 Enclosure position: N/A
 Device Id: 10
 WWN: 5000C50043A97C34
 Sequence Number: 2
 Media Error Count: 0
 Other Error Count: 0
 Predictive Failure Count: 0
 Last Predictive Failure Event Seq Number: 0
 PD Type: SAS
 
 Raw Size: 279.396 GB [0x22ecb25c Sectors]
 Non Coerced Size: 278.896 GB [0x22dcb25c Sectors]
 Coerced Size: 278.875 GB [0x22dc0000 Sectors]
 Sector Size: 0
 Firmware state: Online, Spun Up
 Device Firmware Level: ES64
 Shield Counter: 0
 Successful diagnostics completion on : N/A
 SAS Address(0): 0x5000c50043a97c35
 SAS Address(1): 0x0
 Connected Port Number: 0(path0)
 Inquiry Data: SEAGATE ST3300657SS ES646SJ3NDEM
 FDE Capable: Not Capable
 FDE Enable: Disable
 Secured: Unsecured
 Locked: Unlocked
 Needs EKM Attention: No
 Foreign State: None
 Device Speed: 6.0Gb/s
 Link Speed: 6.0Gb/s
 Media Type: Hard Disk Device
 Drive Temperature :38C (100.40 F)
 PI Eligibility: No
 Drive is formatted for PI information: No
 PI: No PI
 Port-0 :
 Port status: Active
 Port's Linkspeed: 6.0Gb/s
 Port-1 :
 Port status: Active
 Port's Linkspeed: Unknown
 Drive has flagged a S.M.A.R.T alert : No
 
 
 
 Physical Disk: 1
 Enclosure Device ID: 32
 Slot Number: 11
 Drive's position: DiskGroup: 0, Span: 5, Arm: 1
 Enclosure position: N/A
 Device Id: 11
 WWN: 5000C50054EA3600
 Sequence Number: 4
 Media Error Count: 0
 Other Error Count: 0
 Predictive Failure Count: 8
 Last Predictive Failure Event Seq Number: 37843
 PD Type: SAS
 
 Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
 Non Coerced Size: 558.411 GB [0x45cd2fb0 Sectors]
 Coerced Size: 558.375 GB [0x45cc0000 Sectors]
 Sector Size: 0
 Firmware state: Online, Spun Up
 Device Firmware Level: ES65
 Shield Counter: 0
 Successful diagnostics completion on : N/A
 SAS Address(0): 0x5000c50054ea3601
 SAS Address(1): 0x0
 Connected Port Number: 0(path0)
 Inquiry Data: SEAGATE ST3600057SS ES656SL4GP4B
 FDE Capable: Not Capable
 FDE Enable: Disable
 Secured: Unsecured
 Locked: Unlocked
 Needs EKM Attention: No
 Foreign State: None
 Device Speed: 6.0Gb/s
 Link Speed: 6.0Gb/s
 Media Type: Hard Disk Device
 Drive Temperature :40C (104.00 F)
 PI Eligibility: No
 Drive is formatted for PI information: No
 PI: No PI
 Port-0 :
 Port status: Active
 Port's Linkspeed: 6.0Gb/s
 Port-1 :
 Port status: Active
 Port's Linkspeed: Unknown
 Drive has flagged a S.M.A.R.T alert : Yes
===Disk status===
To check all the disks we can use:
 megacli -PDList -aall
Probably, the most wanted fields we are looking at the output from this commands are:
Slot number and device ID
Firmware status (a failed disk will show Failed)
Foreign state: normally it should have "None", unless we insert a disk which has belonged to a different RAID. In that case we have to clear the previous configuration.
===Smart status===
See SMART#SMART_not_healthy
==== Controller log output to a file ====
 megacli -AdpEventLog -IncludeDeleted -f log.txt -aALL
==Replacing Disks==
=== Automatic array rebuild with a new disk ===
Normally when a disk fails, and the controller detects it, it gets marked as a bad disks (Firmware status: Failed)
It needs to be replaced, and if it is a brand new disk, as soon as it gets replaced, the controller will start to rebuild the RAID.
By using the command 
 megacli -PDList -aall
We can identify the disk that failed (look for the one that has Firmware status: Failed) and by looking at its Enclosure devide ID and Decide ID we can form the following command to check the % on the RAID rebuild:
 megacli -PDRbld -ShowProg -PhysDrv [32:4] -aALL
 
 Rebuild Progress on Device at Enclosure 32, Slot 4 Completed 2% in 5 Minutes.
 
 Exit Code: 0x00
Once it is finished we will be able to see that the array is fine again by running:
 megacli -AdpAllInfo -aALL
 Device Present
 ================
 Virtual Drives : 1
 Degraded : 0
 Offline : 0
 Physical Devices : 14
 Disks : 12
 Critical Disks : 0
 Failed Disks : 0
=== Manually array rebuild with an used disk ===
If we use a non new disk, which has been part of an array in the past, it will contain metadata and the controller will not start to rebuild the array until we have cleaned that configuration, as a security precaution. 
First of all we need to make sure that indeed, the RAID is not rebuilding because there is a foreign configuration on a disk.
We can scan for old configurations:
 megacli -CfgForeign -Scan -aALL
 
 There are 1 foreign configuration(s) on controller 0.
In that case we can verify that if it is indeed our new disk:
 megacli -PDList -aall
When looking for our new disk we will realise it will have these fields:
 Enclosure Device ID: 32
 Device Id: 4
 Firmware state: Unconfigured(good), Spun Up
 Foreign State: Foreign
So we know that the disk 32:4 is in a good state but it was part of an array before and needs to be cleaned up.
 megacli -CfgForeign -Clear -aALL
 
 Foreign configuration 0 is cleared on controller 0.
 
 Exit Code: 0x00
Now we can check if we are good to go:
 
 megacli -CfgForeign -Scan -aALL
 
 There is no foreign configuration on controller 0.
 
 Exit Code: 0x00
We are ready to replace the disk.
First of all we need to identify in which SPAN the old disk was and in which position.
We firstly need to go thru all the SPANs:
 megacli -CfgDsply -a0 | grep "Number of Spans"
 Number of Spans: 6
And we need to know how many disks per SPAN we should have:
 megacli -CfgDsply -a0 | grep "Number of PDs" -m1
 Number of PDs: 2
There must be 2 disks per SPAN, so we need to scan all the SPANs to see which one has a disk with no information.
Normally a SPAN will show the Physical Disk information empty if a disk is missing.
In this case we stop at SPAN 2 because the information for Physical Disk 0 (in bold) is missing (compare it with the information it shows for Physical Disk 1).
And the SPAN also shows: State: Degraded
 SPAN: 2
 Span Reference: 0x02
 Number of PDs: 2
 Number of VDs: 1
 Number of dedicated Hotspares: 0
 Virtual Drive Information:
 Virtual Drive: 0 (Target Id: 0)
 Name :
 RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0
 Size : 1.633 TB
 Sector Size : 512
 Mirror Data : 1.633 TB
 State : Degraded
 Strip Size : 256 KB
 Number Of Drives per span:2
 Span Depth : 6
 Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
 Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
 Default Access Policy: Read/Write
 Current Access Policy: Read/Write
 Disk Cache Policy : Disk's Default
 Encryption Type : None
 Bad Blocks Exist: Yes
 Is VD Cached: Yes
 Cache Cade Type : Read Only
 Physical Disk Information:
 Physical Disk: 0
 
 
 
 Physical Disk: 1
 Enclosure Device ID: 32
 Slot Number: 5
 Drive's position: DiskGroup: 0, Span: 2, Arm: 1
 Enclosure position: N/A
 Device Id: 5
 WWN: 5000C50043A74B14
 Sequence Number: 2
 Media Error Count: 12
 Other Error Count: 0
 Predictive Failure Count: 0
 Last Predictive Failure Event Seq Number: 0
 PD Type: SAS
 
 Raw Size: 279.396 GB [0x22ecb25c Sectors]
 Non Coerced Size: 278.896 GB [0x22dcb25c Sectors]
 Coerced Size: 278.875 GB [0x22dc0000 Sectors]
 Sector Size: 0
 Firmware state: Online, Spun Up
 Device Firmware Level: ES64
 Shield Counter: 0
 Successful diagnostics completion on : N/A
 SAS Address(0): 0x5000c50043a74b15
 SAS Address(1): 0x0
 Connected Port Number: 0(path0)
 Inquiry Data: SEAGATE ST3300657SS ES646SJ3MN6X
 FDE Capable: Not Capable
 FDE Enable: Disable
 Secured: Unsecured
 Locked: Unlocked
 Needs EKM Attention: No
 Foreign State: None
 Device Speed: 6.0Gb/s
 Link Speed: 6.0Gb/s
 Media Type: Hard Disk Device
 Drive Temperature :38C (100.40 F)
 PI Eligibility: No
 Drive is formatted for PI information: No
 PI: No PI
 Port-0 :
 Port status: Active
 Port's Linkspeed: 6.0Gb/s
 Port-1 :
 Port status: Active
 Port's Linkspeed: Unknown
 Drive has flagged a S.M.A.R.T alert : No
So our new disk 32:4 needs to go to SPAN2 and and become Physical Disk 0 so for megacli that means row0 of the SPAN.
 megacli -PdReplaceMissing -PhysDrv[32:4] -array2 -row0 -a0
 
 Adapter: 0: Missing PD at Array 2, Row 0 is replaced.
Now we are ready to rebuild the ARRAY.
 megacli -PDRbld -Start -PhysDrv[32:4] -a0
 
 Started rebuild progress on device(Encl-32 Slot-4)
And we can check the progress
 megacli -PDRbld -ShowProg -PhysDrv [32:4] -aALL
 
 Rebuild Progress on Device at Enclosure 32, Slot 4 Completed 9% in 2 Minutes.
Note that the ARRAY and the SPAN will not be set to Optimal until the disk has finished. Once it has finished.
=== Check and clear Preserved Cache ===
In cases where a disk fails and it is replaced it might happen that the controller still has cached data for said disk. Note that when in this state the controller requires manual intervention to boot and it will prompt for "preserved data cache present".
 FW error description: 
 The current operation is not allowed because the controller has data in cache for offline or missing virtual drives. 
If that's the case it is possible to list all disks with preserved cache:
 # megacli -GetPreservedCacheList -a0
 Adapter #0
 Virtual Drive(Target ID 09): Missing.
 Exit Code: 0x00
And clear the preserved cache for the drive in question:
 # megacli -DiscardPreservedCache -L09 -a0 
 Adapter #0
 Virtual Drive(Target ID 09): Preserved Cache Data Cleared.
 Exit Code: 0x00
=== Manually replace multiple disks ===
The steps to replace multiple failed disks (of different SPANs) are the same as for MegaCli#Manually array rebuild with an used disk however, we need to make sure we do one disk at the time.
Please read the MegaCli#Manually array rebuild with an used disk before continuing here, so you understand what is going to be done now:
Once we have identified the SPANs and disks that need replace, execute the Replace+Rebuild commands one by one, and do not start all of them at the same time.
Wait for each SPAN to be finished before starting with the new one.
Note: the RAID and the SPANs will not be set to Optimal until all the degraded SPANs have been fixed. 
There is an example of how to operate multiple failed disks here:
https://phabricator.wikimedia.org/T149099#2751204 and this is how to start replacing each disk: https://phabricator.wikimedia.org/T149099#2762624
=== Replace individual disks in JBOD ===
Some hosts (e.g. ms-be) have a raid controller and use all disks in each own raid 0 virtual drive (essentially JBOD). It might happen that the disk shows errors to the OS and the controller thinks the disk itself is fine (i.e. doesn't mark the disk as failed). In these cases we need to fail the disk manually as follows:
 # Identify the Physical Drive ID
 /usr/local/lib/nagios/plugins/get-raid-status-megacli
 === RaidStatus (does not include components in optimal state)
 name: Adapter #0
 Virtual Drive: 16 (Target Id: 16)
 RAID Level: Primary-0, Secondary-0, RAID Level Qualifier-0
 State: Optimal
 Number Of Drives: 1
 Number of Spans: 1
 Current Cache Policy: WriteBack, ReadAheadNone, Cached, No Write Cache if Bad BBU
 Span: 0 - Number of PDs: 1
 PD: 0 Information
 Enclosure Device ID: 32
 Slot Number: 14
 Drive's position: DiskGroup: 16, Span: 0, Arm: 0
 Media Error Count: 2360
 Other Error Count: 1
 Predictive Failure Count: =====> 1 <=====
 Last Predictive Failure Event Seq Number: 4812
 Raw Size: 7.277 TB [0x3a3812ab0 Sectors]
 Firmware state: Online, Spun Up
 Media Type: Hard Disk Device
 Drive Temperature: 41C (105.80 F)
The disk is PD slot 14 and enclosure 32 in adapter 0, therefore:
 # Mark as offline
 megacli -PDOffline -PhysDrv '[32:14]' -a0
 # Locate the drive (blink the led)
 megacli -PDLocate -PhysDrv '[32:14]' -a0
These actions will trigger a task in Phabricator. After replacement the disk can be put back (and clearing the preserved cache first if needed)
 megacli -GetPreservedCacheList -a0
 megacli -DiscardPreservedCache -L'disk_number' -a0
 megacli -CfgEachDskRaid0 WB RA Direct CachedBadBBU -a0
== See also ==
* Perccli64
* HP Hardware Raid Information Gathering
==External links==
* Cheat Sheet
* Official documentation (PDF)
* https://hwraid.le-vert.net/wiki/DebianPackages