Lustre at China National GeneBank
Liping Yang

Part One
Introduction to China National GeneBank (CNGB)    l Non-Profit Institutel Funding by:– China NDRC (National Development and Reform Commission)– Shenzhen local government– BGI self-fundingl Operated by BGIl Three Banks & Two Platforms
ReadWrite
3 Banks & 2 Platforms
Living Biobank
Bio-informatics Data Center
Biorepository

CNGB Bio-Informatics
Data Center• 65,072 CPU Cores• 258.36 TB Memory• 119.41 PB Storage• 917.23 T Flops   

Part TwoThe Road to Large Scale Lustre File System  l Configuration:– Sun Fire x4500 (500GB hard drive * 48) * 8– Soft RAID. No HA.
– Lustre 1.6

l Problems:– The reseller failed to deploy the system.
– Lustre engineer from Sun help us deploy it, andthe benchmark result is pretty good.

l Resolution:– We break the system into NAS

Source: https://www.youtube.com/watch?v=lwT3Hrk4BSol Lessons learned:– Proof of concept is not optional.


   l We buy commercial parallel storage systems like Isilon and Panasas.
l We tried many commercial and open source parallel file systems like ParaStor, BWFS
Blue Whale File System, glusterfs and Lustre.
l We use Glusterfs and Lustre is our testing environment      l Background:– Two copies. One of them in Lustre File System.

l Configuration:– Supermicro 4U 90Bay JBOD + 1U OSS Server– IEEL 2.x ~ IEEL 3.x

l Results:– No vendor lock-in– Stronger negotiating position– High density space saving– Gain Lustre production system experience       l Configuration:– Dell MD3460 + R630Supermicro 4U 90Bay JBOD + 1U OSS Server– IEEL 2.x ~ IEEL 3.x

l Compare to other commercial solutions– No vendor lock-in– 10x faster– Half-price

     l  '#
l #  !""! $&ll !#%##$"$ 

Source: http://wiki.lustre.org





Part Three
Run and Manage Lustre    l '#&( – %% – #!  l $%&%(%& l $"$ !– !%(%&%– !  %   l &"$$)$(!"$%– '#$ $"

 l (%"&)$%'##"$&$"  "'

l !' "($ 

l    !# l "    !#   l!    !! # l  



l ## '#"  'l ('%'%"&#%!#  &'%%#%#&&#%l '#%  &'&%"%$'l &#% ""%"  l l  !#–  "  

Part FourIssues and Lessons learned     

l Problems– Firmware (HBA, Hard Drive etc.)– Defective modules or cables– Network adapter and network switch cooperation issuel Resolutions– POC and benchmarking– End-to-end quality control process– Working on in-house automated test systeml Pros
– Open Source– Unified RAID administration for different hardware (no 3rd party RAID tools)– End-to-end data integrity– Scalable, online filesystem check/scrub/repair– Lower hardware costsl Cons– Slightly lower metadata performance, especially for MDT

l Considerations– zfs 0.7.0 or above for MMP. Keep your eyes on bugfix and new feature in Github.

l Future works– QAT hardware-assisted checksums, compression and encryption


l Timeoutl Adaptive Timeouts– at_min– at_maxl LDLM Timeouts– ldlm_enqueue_min– ldlm_timeoutl Debug - dump_on_timeout, dump_on_evictionl ARP issue in our environment

Reference: Lustre Resiliency: Understanding Lustre Message Loss and Tuning for Resiliency        l Background:– Tight Deadline– POSIX compliant file system storage– A fixed budget, with prioritization on storage capacityl Solution:– Intel Enterprise Edition Lustre– Intel Lustre reference architecture hardware without POC



          l Issue 1 – Network sluggish•LustreError: 51342:0:(import.c:372:ptlrpc_invalidate_import()) zfs4-OST0005_UUID: RPCs in "Unregistering"phase found (1). Network is sluggish? Waiting them to error out.

•Network interface is flapping up and down a few times per dayl Root Cause– Continuous Time Linear Equalizer (CTLE) setting incompatible between network adapter andnetwork switchl Resolution– Apply a patch to the network switch          l Issue 2 – Some compute nodes network interface dropping packets•

cngb-compute-e01-4: err: 109 drop: 1438345 frame_error: 109 nfs_err_num:8•cngb-compute-e03-5: err: 0 drop: 65514579 frame_error: 0 nfs_err_num: 0•/opt/gridengine/default/spool/cngb-compute-f22-5/job_scripts/5054771: line 2: echo: write error: Input/outputerror ln: accessing `/zfs2/ntdb/group5_20/result/process/CNV_CNVnator/101608698/chr14/101608698.bam':Cannot send after transport endpoint shutdownl Root Cause– MTU size mismatchl Resolution– Set and verify MTU setting for all network adapter and network switch          l Issue 3 – 25G NICs bonding showing only half the throughputl Root Cause– Network adapter was put in a low speed PCI Express x4 slotl Resolution– Relocate the network adapter to a PCI Express x8 slot

          l Issue 4 – Cross data center network interruption•

00000800:00000100:57.0:1528296092.423675:0:9079:0:(socklnd_cb.c:2276:ksocknal_find_timed_out_conn())
A connection with 12345-10.xxx.xxx.xx@tcp (10.xxx.xxx.xxx:988) timed out; the network or node may be down.

•

00000100:00000400:31.0:1528296182.884581:0:108584:0:(client.c:2114:ptlrpc_expire_one_request()) @@@Request sent has timed out for slow reply: [sent 1528296065/real 1528296065] req@ffff8859b92123c0x1596679597925056/t0(0) o41->lustre-MDT0000-mdc-ffff8840501ca800@10.xxx.xxx.xxx@tcp:12/10 lens
224/368 e 0 to 1 dl 1528296182 ref 2 fl Rpc:X/0/ffffffff rc 0/-1•

00000100:02000400:31.0:1528296182.884615:0:108584:0:(import.c:178:ptlrpc_set_import_discon()) szebraMDT0000-mdc-ffff8840501ca800: Connection to lustre-MDT0000 (at 10.xxx.xxx.xxx@tcp) was lost; in progress
operations using this service will wait for recovery to completel Root Cause– ARP proxy occasionally not respondingl Resolution– Use router instead of ARP proxy

          l Issue 5 – HBA reset cause ZFS hang•

May 31 02:43:11 cngb-oss-a2-2 kernel: mpt3sas_cm3: fault_state(0x5853)!

•May 31 02:43:11 cngb-oss-a2-2 kernel: mpt3sas_cm3: sending diag reset !!

•May 31 02:43:12 cngb-oss-a2-2 kernel: mpt3sas_cm3: diag reset: SUCCESSl Root Cause– Firmware version phase15IT/22.00.00.00 of HBA is not compatible with RHEL 7.4

l Resolution– Upgrade HBA firmware

          l Other issues•

Fiber optic cable defects. cngb-compute-e03-8: err: 1000601 drop: 913804 frame_error: 1000601 nfs_err_num:

1377•CRC error increasing in 25G NIC•Hybrid port VLAN tagging disorderedl Resolution– Replace fiber optical cables– Configure FEC in Optical module, and replace some faulty optical module– Apply a patch to switch

l Lessons learned:– Try before you buy– Software version do matters– Burn-In Test is not nice to have

Part Five
Summary
l Lustre is super fast and stable enough, but it’s still complex and fragile, IMHO.
– Technical support is very important.

l POC and benchmarking for systems you want to buy.
– Commercial product like DDN is good
– If you want to build your own Lustre system, do extensive benchmarking.

l It’s worth taking a hard look at ZFS.
l Ethernet is not the best option for Lustre, but it works.
– Carefully tune the timeout parameters.

l You will encounter Lustre Bug one day.
– Prepare for it. Set up Core Dump. Monitor any metrics related. Collect logs.

