# Presentation: 016808

## Moving to HPSS 5.1Lessons Learned

**Dennis Tran**
**June 2005**

## Moving to HPSS 5.1

**Agenda**
**HPSS 5.1 Status**
**Conversion from HPSS 4.5 to 5.1 – A quick overview**
**Lessons learned**

## Moving to HPSS 5.1

**Old news:**
**HPSS 5.1 has been available since 11/2003**
**Six sites already converted**
- ECMWF/UK
- LANL
- LLNL
- SNL
- IN2P3/FR
- SDSC

**Notes:**

Encina EOS dates:
 5.0	12/2005
5.1	12/2007
Follow-on release at least 2009

DCE EOS date: 4/2006

## Upgrading from HPSS 4.5 to 5.1

**Two major tasks**
**Upgrade HPSS software**
**Replace SFS with DB2 or “Metadata Conversion”**
**Available information**
**HPSS Installation Guide, Chapter 14 – from HPSS web site**
**DB2_Tuning_Guide.doc – from HPSS support representative**
**	****(rosebud:/hpss/doc/DB2_info/ DB2_Tuning_Guide.doc)**

## General Procedure for Metadata Conversion

**Prepare for conversion**
- Install DB2
- Install HPSS 5.1
- Configure DB2 using mkhpss
**DB2 tuning **
**Run the conversion scripts**
- db_convert_collect_info
- db_config_convert
- db_subsys_convert
- db_convert_bftapeseg
- db_convert_bitfile
- db_convert_nsobject
- db_convert_nstext
- db_convert_storagesegdisk
- db_convert_storagesegtape

- The metadata conversion is finished !

**Note:**
**Scripts 1- 3 must be invoked in the correct order**
**Scripts 4 - 9 can be in any order**

## General Procedure for Metadata Conversion (concluded)

**Run the verification utilities:**
- db_convert_size_check
- db_convert_ns_check
- db_convert_address_check
- db_convert_disk_allocate_check
**Before starting HPSS 5.1 servers:**
- db_convert_dce_cds
- settapestats
**DB2 tuning **
**Set up metadata backup**
**Start HPSS **
**Cleanup**

- HPSS 5.1 is up and running !

## An actual preparation for a conversion

**Before the conversion day:**
- Prerequisite software: DB2, Java
**DB2 with the needed fixpack, license key**
**Install and configure**
**Create DB2 tables**
**Set up DB2 backup**
**Tune DB2**
- HPSS images
**Install on a separate directory**
- Change migration and purge policies to purge all files from disks
- Make full SFS backup
**Just before conversion:**
- Ensure SFS backup completed
- Disable user access to HPSS
- Save lshpss output
- Save HPSS 4.5 source tree
- Dump tape sspvs to make sure “read-only” are set for tapes
- Run settapestats and save the output for post conversion verification
- Run accounting and save the output for post conversion verification

## Timeline of an actual conversion for 30 million files

**C-Day + 2**

- Run the last verification (4 hrs)

- Install DB2 backup
- Rebuild HPSS
- Bring up HPSS and release to the test team

**C-Day + 3**

- Check DB2 logging and backups

- Team meeting for decision to release the system to the users

**Notes:**
 LLNL’s  experiences
AIX p660 6M1
FAStT900
Open Computing facility
12.5 million files: < 20 hrs
Full verification on a separate machine: ~ one week
Secure Computing Facility
24 million files: 30+ hrs

ECMWF
1.3 million files: ~ 90 minutes
Full verification on a separate machine: ~ 16 hrs

## Lessons learned

**DB2 installation and configuration**
- Need to have DB2 DBA skills
**Metadata conversion**
- Test/practice conversions using the production metadata
**To determine if there are anomalies in the existing SFS data that need to be corrected before the actual conversion**
**To find an optimal way to run the conversion scripts to minimize the total run time**
**Test system can be used 24x7 without impacting the production**

## Lessons learned

**Set up and use a test system as soon as practical **

## Lessons learned (cont’d)

**     ****Set up the test system**
**SFS data volumes and log volume must have the same names and sizes as the production system’s**
**Either “dd” copy or TRB and LA files can be used to populate the test system’s SFS database.  **
**SFS TRB files must be smaller than 2GB**
**SFS files cannot be transferred via NFS**
**Checks before & after SFS data transferred to the test system:**
**Checksum**
**tkadmin to check the size of SFS data volumes and log volume**
**sfsadmin to check the number of records in each SFS file**

## Lessons learned (cont’d)

**Set up  a test system (concluded)**
**Configure and tune DB2 ***(see*** ***DB2_Tuning_Guide.doc*)
**NUM_IOSERVERS**
**NUM_IOCLEANERS**
**CHNGPGS_THRESH**
**LOGFILSIZ**
**LOGPRIMARY and LOGSECOND**
**NEWLOGPATH**
**LOCKTIMEOUT**
**LOCKLIST**

## Lessons learned (cont’d)

**Run the metadata conversion**
- Both SFS and DB2 must run simultaneously
- Time required to complete metadata conversion is SFS data size and hardware dependent
**Below are the conversion results from a test system with 3 millions files:**

| Conversion script | Minutes per one million files on an AIX p44-270 |
| --- | --- |
| db_convert_bftapeseg | 24 |
| db_convert_bitfile | 20 |
| db_convert_nsobject  | 38 |
| db_convert_nstext | 16 |
| db_convert_storagesegdisk  | 48 |
| db_convert_storagesegtape  | 48 |

## Lessons learned (cont’d)

**Run the metadata conversion **
- Actual times from a practice conversion:
- Combine conversion scripts into groups and stage the runs to minimize the run time.
**db_convert_nsobject**
**	****db_convert_storageseg_disk**
**	****db_convert_nstext**
**db_convert_storagesegtape**
**db_convert_bitfile**
**db_convert_bftapeseg**

## Lessons learned (cont’d)

**When having problems running the metadata conversion :**
- Review the outputs for warnings and errors
**HPSS Installation Guide, Section 7.4**
**DB2 UDB Message Reference Volume 2 for Version 8**
- Use load_terminate_*    utilities before restarting a conversion script
- Contact HPSS support

## Lessons learned (cont’d)

**Metadata conversion verifications**
- Verifications will take much longer than the actual conversion
**Full verifications**
**Partial verifications**
**db_convert_ns_check: *****can check first M records and last N records***
**db_convert_address_check: *****can skip through the file by using –e N number of records to check and –i M number of records to skip***
- Full conversion verification times from the previous example (p. 14)

## Lessons learned (cont’d)

**Conversion-Day – x weeks**

**Production System**

**Test System**

**DD/TRBs**

**Conversion-Day**

- Setup test system
- Conversion
- Full verification
- Refine the conversion steps
- Repeat the practice conversionas many time as needed to have aclean conversion

**DD/TRBs**

- - Conversion
-  Full verification

- - Conversion
- Partial verification

**New DD if needed**

## Lessons learned (concluded)

**In a nutshell**
- Requires detailed planning
- Not a one person task
- Must have SFS and DB2 skills (configuration, tuning, query, backup and restore)
- Must have a test system to practice conversion ahead of the actual conversion of the production system; test system can be used to reduce the verification time for the production system
- Refine the procedure to minimize production downtime