# Presentation: 813049

## Site-Wide BackupBriefing

- Ray Pasetes
- Core Support Services
- April 16, 2004

## Agenda

- Progress Update
- Proposed Model
- Proposed Rollout
- Initial Costs
- Estimated 6-year TCO
- Needed Decisions
- Critical Timeline

## Progress Update

## Progress Update	

- 12/03: Reduce scope to division-wide
- Site-wide requirements too diverse
- Start small, let others see it work, then expand
- 1/04: Interview groups
- 6 Groups
- ClueD0, D0-offline, CDF, CMS, FTP, ISA
- SCS (represents 11 clusters/groups)
- Along with CSI, 22 total clusters/groups  represented

## Progress Update	

- 2/1: Interviews finished
- 2/06: CSS department review of data
- Most accepted 8to17 x5 service
- Most accepted same day or NBD restore
- Investigate using existing resources
- Investigate costs for deployment
- Initial deployment will backup 7+TB of data
- Start pilot small

## Proposed Model

## Goals

- Provide a reliable data backup service
- Reduce redundant effort, allowing division to be more productive
- Long-term goal: reduce overall division spending on data backups via consolidation
- Long term goal: service accessible across entire site, desktops included

## Service Model

- Use farms model –> backup blocks
- Backup blocks == 1 server + 4 or more tape drives
- Smaller customers share the same backup block
- Larger customers would have their own backup block(s)

**Notes:**
The backup service would be operated similar to the farms.  The backup service would consist of backup blocks. Each block would contain (1) backup server, some caching disk and at least (4) tape drives.  Customers with small to medium data sets would share the same backup block, similar to the FT farms.  These customers would have less freedom in determining their backup rotation and tape retention schemes.

Customers with large data sets or unusual requirements (CMS scenario, 3TB delta daily) would have their own backup block or blocks.  This is similar to the CDF and D0 farms.  These customers would have more say as to the tape rotation, retention, etc.

## Costing Model		

- Charge based on total GB backed up
- Cost should completely cover tape costs
- Cost should cover hardware and software maintenance costs
- Cost should cover hardware costs
- “Profit” will be used to expand and enhance the system

**Notes:**
How do we get profit if we are only covering costs?  Our costs are based on a very conservative estimate. If we realize any compression of data, we will get a profit.  In addition, for every year we do not need to purchase another backup block, we will realize a profit.

## Costing Model - Example

- Year 1 customer: $1.15/GB on tape/yr or $34.50/GB of data to backup/year
- Covers hardware, tape and maintenance costs
- Year 2+ customer:  $0.33/GB on tape/yr or $9.82/GB of data to backup/year
- Covers maintenance, tapes, additional slots, etc.
- No charge for existing hardware
- No connection or per client fees
- No restore fees

**Notes:**
Costs based on using the D0 ADIC robot and current pricing.  Pricing subject to change without notice.
The costs for year 2+ customers assumes growth (up to a 100% increase) in data set size.

Using the same costing model for D0 offline, their initial cost would have been  $89.89/GB(?) in year1
($37.89 w/o library cost)  and  $11.20/GB year2+.   The year 2 cost does not take into account any growth rate (extra tape purchases). Costing model tries to recoup hw costs over 1 year.

## Proposed Rollout

## Proposed Rollout

- Start pilot small ~ 7.1TB
- The following groups will be asked to partake in pilot project
- 	- Astro, CEPA, CMS, D0, ESH, FESS, ISA, KTEV, LSS, MINOS, NUMI, PPD, SDSS, SIDET, Theory, VMS
- Desktops NOT part of initial rollout
- Lessons learned in year 1 determine year 2 growth

**Notes:**
In addition, service will cover AFS, Imap, kits software distribution, Linux dist servers, mail gateways, CVS, ngop, FNALU, news servers, central webservers, license servers, farms, print servers.

## Proposed Rollout – Unknowns

- Limit of each backup block
- Highly dependent on tape rotation and daily delta
- A single server at CMU can handle 15TB of data
- Compressibility of data
- Affects cost of service
- Affects performance

**Notes:**
How much data a single data block can handle is dependent on several factors:
Network speed
Disk speed
Tape drive speed
Data retention and rotation schedule

## Proposed Rollout – Timeline

- 9/1 – Equipment delivered
- 9/8 – Equipment Installed
- 9/8-9/15 – Functionality test
- 9/16-10/01 – Systems testing
- 10/04 – Start rollout
- 12/04 – Complete initial rollout
- ~2FTE effort required for initial rollout

## Initial Costs

## Costs - Library

- Three Solutions
- STK and 9940B tape drives
- ADIC and LTO-2 tape drives
- SpectraLogic and SAIT-1 tape drives

## Costs – Library

| Items | STK | Spectra | D0 ADIC | CDF ADIC |
| --- | --- | --- | --- | --- |
| Library | $0 | $69,418.12 | $19,302.00 | $19,302.00+ |
| Drives Needed | 8x 9940B | 8x SAIT-1 | 8x LTO-2 | 8x LTO-2 |
| Drive Cost | $208,000 | $122,894.87 | $76,480 | $76,480 |
| Tapes Needed | 1095 | 450 | 1095 | 1095 |
| Tape Cost | $78,292
($357.50/TB) | $80,970
($359.87/TB) | $72,270
($330/TB) | $72,270
($330/TB) |
| TiBS S/W Port | $30,000+ | $0 | $30,000+ | $30,000+ |
| Maintenance | $6,960 | $14,436 | $39,276 | $48,015 |
| Total year 1 | $323,252+ | $287,719.29 | $237,328+ | $246,067+ |

**Notes:**
There is no charge for the STK library because the assumption is that we use existing resources.  Each STK silo already comes fully populated with slots.

The charge for the SpectraLogic includes the library, slots, installation and software management tools. It does not
Include tape drives and tapes (priced separately).

There is a charge for the ADIC library since there is a charge to install the LTO-2 slots.  The charge for the D0 ADIC is the cost to add onto the functional part of the robot.  

The library cost for the CDF ADIC does not include re-installation of the library.

TiBS S/W porting costs start at $8250 initially.  Once the company fully understands how long the actual port will take, they will give us a final price and allow us to either reject or accept the offer.  They estimate that 2 weeks of development time will be approximately 3 months of real time because they would need to interleave this customization with their normal duties. We estimate the true cost will be ~$30K-$50K.

STK maintenance only covers cost of drives.  We assume CCF pays for the Library maintenance.
SpectraLogic maintenance covers cost of drives and library and is an upcharge from the standard 8x5 NBD maintenance.
ADIC maintenance costs include incremental increase on the library, quadra tower (QT), second arm and the  additional 8x LTO-2 drives.

## Costs – Server

- 2x Sun V440 -- $28K
- 1x SATA RAID Disk Cache -- $ 14K

**Notes:**
The Servers are quad CPU systems.  Moving to Linux on Dell servers would be approximately the same cost.

1x SATA RAID has already been donated by CMS for their backup block.  Therefore, we only need one more.

## Costs – Software - TiBS

- No additional client costs
- Servers: $8250 each
- 2 have already been purchased
- Additional cost for OFM s/w for Windows
- Process packs
- Processes == number of parallel backups
- 45 processes have been purchased
- Maintenance – 15%
- Currently $11,590.35 annually
_***Prices change if not managed by CSS***_

**Notes:**
Processes are purchased in packs.  A 10-pack process is now $20,160.  A 25-pack process is now $40,320.  In 2001-2002 timeframe, we purchased all 45 process packs for less than the current pricing for a 25 pack.

OFM, or open file management software for Windows boxes is necessary to backup open files. There are two options for OFM s/w.  The most economical is the server based license software, OTM by Columbia products.  We are currently using this for the CSI backup server.   OTM clients are “free”, onlly the server is charged with a license fee. The other option is to use OFM from St. Bernard.  This is priced per client.

The current license with Teradactyl is a department-wide license.  As long as CSS manages the backup server, we will not break any licensing agreements with Teradactyl.  However, if other divisions wish to use the software and put up their own server, they would need to either purchase a completely new license or Fermilab could purchase a University license upgrade, $36K, which would allow any department to setup and manage their own TiBS backup server.  Process packs could also be shared among the various departments at a discounted rate.

## Possible Funding	

- CSS Backup OP: $130K
- CSS Backup EQ: $120K
- CMS Backup EQ: $100K
- Other groups?

## Estimated 6-year TCO	

- Two Growth Models
- Fermi Standard
- 10% daily delta
- Data growth doubles yearly
- Approximately 40% above industry standard
- Fermi Active
- 45% daily delta (CMS scenario)
- Data growth doubles yearly

**Notes:**
Standard industry growth is 50-60% with a daily delta of 6-7%.

## Estimated Year 2 (14.2TB)

- Fermi Standard
- Double slots
- Add caching disk
- Tape cost down 25%
- Configuration:
- 2 backup blocks
- Slots 2x year 1
- Increased caching disk

- Fermi Active
- ~Triple slots
- Double backup blocks
- Add caching disk
- Tape cost down 25%
- Configuration:
- 4 backup blocks
- Slots ~3x year 1
- Increased caching disk

**Notes:**
In Fermi Active, slots more than triple as part of a one time correction due to the amount of daily delta.  Afterwards, slot growth should be linear.  In reality, we will most likely need to start adding slots sometime during year 1 in the Fermi active scenario because the original robot would have been sized too small.

## Estimated Year 3(28.4TB)

- Fermi Standard
- Double Slots
- Add Caching disk
- Configuration:
- 2 backup blocks
- Slots 4x year 1
- Increased caching disk

- Fermi Active
- Double backup blocks
- Double slots
- Add caching disk
- Drive cost down 25%
- Configuration
- 8 backup blocks
- Slots ~6.5x year 1
- Increased caching disk

## Estimated Year 4

- Tape technology changes
- Media capacity quadruples
- Drive performance quadruples
- 10GigE standard
- Servers now equipped with 10GiGE
- Increased bus speeds
- Generally available to the lab

## Estimated Year 4	

- Roll in new servers
- Backup block == 1 server + 8 tape drives
- Migrate customers onto new servers
- Allow old tapes to migrate off via tape retention policies
- Slowly tear down old backup blocks
- Approximately 1 year migration

**Notes:**
Normal tape retention is 1 year.  So, as we roll in the new system, we allow the old tapes to roll off.  We will not need to do any tapecopies or tape conversions.

## Estimated Year 4(56.4TB)

- Fermi Standard
- 2 new backup blocks
- Increase slots by half
- Configuration:
- 2 old backup blocks
- 2 new backup blocks
- 2/3 library old slots
- 1/3 library new slots
- Slots 6x year 1

- Fermi Active
- 2 new backup blocks
- Increase slots by half
- Configuration:
- 8 old backup blocks
- 2 new backup blocks
- 2/3 library old slots
- 1/3 library new slots
- Slots ~10x year 1

## Estimated Year 5(112.8TB)

- Fermi Standard
- Add caching disk
- Convert slots
- Tape cost down 25%
- Configuration:
- 2 backup blocks
- All slots new
- 2/3 library used
- 1/3 unused
- Slots 6x year 1

- Fermi Active
- Double backup blocks
- Convert slots
- Add caching disk
- Tape cost down 25%
- Configuration:
- 4 backup blocks
- All slots new
- 2/3 library used
- 1/3 unused
- Slots ~10x year 1

## Estimated Year 6(225.6TB)

- Fermi Standard
- Add caching disk
- Increase slots by 1/3
- Configuration
- 2 backup blocks
- Slots 8x year 1

- Fermi Active
- Add caching disk
- 3 additional backup blocks
- Increase slots 1/3
- Drive cost down 25%
- Configuration:
- 7 backup blocks
- Slots ~13x year 1

## Estimated 6-year TCO

| Year | Spectra Standard | Spectra Active | ADIC Standard | ADIC Active |
| --- | --- | --- | --- | --- |
| 1 | $329,719 | $329,719 | $279,328 | $279,328 |
| 2 | $135,064 | $454,402 | $142,848 | $474,458 |
| 3 | $132,569 | $721,767 | $234,420 | $830,748 |
| 4 | $582,057 | $756,795 | $723,142 | E |
| 5 | $194,192 | $623,280 | $268,688 | E |
| 6 | $367,681 | $1.13M | $505,083 | E |
| Total | ~$1.74M | ~$4.02M | ~$2.15M | E |

**Notes:**
SpectraLogic pricing includes 24x7x4 maintenance. The ADIC quotes are based on currently negotiated pricing and includes using the second robot arm and a separate tower. ADIC maintenance contract Is pending renegotiation on 9/30/04.

At year 4, with the ADIC Active model, we exceed the capacity of the D0 ADIC.  Another quadra tower would need to be installed, space would need to be allocated or library would need to be moved, or the CDF ADIC would need to be available.

		Spectra Standard	Spectra Active			ADIC Standard		ADIC Active
Year 1	8 drives, 450 slots	8 drives, 450 slots 		8 drives, 1095 slots	8 drives 1095 slots
Year 2	8 drives, 900 slots	16 drives, 1460slots 	8 drives, 2190 slots	16 drives, 3553 slots
Year 3	8 drives, 1800 slots   32 drives, 2920 slots        8 drives, 4380 slots    32 drives, 7106 slots
Year 4      16 drives, 2700 slots  48 drives, 4380 slots        16 drives, 6570 slots  48 drives, 10659 slots
Year 5       8 drives, 2700 slots   32 drives, 4380 slots	8 drives, 6570 slots     32 drives, 10659 slots
Year 6       8 drives, 3600 slots   56 drives, 5840 slots        8 drives, 8760 slots	   56 drives, 14212 slots

## Needed Decisions

## Decisions – Library

- Share D0 ADIC robot?
- Initially, makes most sense financially
- Backup s/w will need to be ported
- Custom engineering costs
- Current s/w assumes all tapes are owned by backup service
- We will be the only deployment
- Fermi will need to provide development resources
- Hardware
- People/coordination
- Active scenario outgrows robot at year 4. This assumes _**ZERO **_growth by D0 and SDSS (current users of robot).
- Standard scenario will use 2/3 of D0 robot.

**Notes:**
Custom Engineering costs will be an issue with each OS upgrade or version of the software.

## Decisions - Library

- Purchase new robot? (SpectraLogic)
- Higher initial cost
- Lower TCO over 6 years.
- Years 2-6 lower operational costs than ADIC
- Backup software already works with it. Service can be brought up more quickly.
- SpectraLogic library running TiBS software to be deployed at MIT (*not for public disclosure*).

## Decisions - Library

- Purchase new robot (part 2)?
- Small footprint.  Fully populated library is 7 racks wide, 2 racks deep.
- Library supports SAIT, LTO, LTO-2, SDLT, DLT
- In ’05, will support disk
- Relatively new product. Higher possibility hardware costs will decrease.

## Footprint Comparison

## Critical Timeline

## Critical Timeline

- 4/21: Division request for departments to determine how budgets will be spent
- CSI/SCS robots running out of room
- Windows support 2.3TB pending decision
- D0 Legato renewal
- 6/1/04: Promised date to CMS for backup service to be online
- 6/30/04: Latest time for TiBS to begin porting effort to make FY ’04 timeline.

## Fini

- Open Discussion

## CSI Backup System - Example

- Initially deployed in 2001 for AFS backups
- Expanded to include UNIX and some windows
- 2003 – SCS added to backup system
- Data has grown >600% since 2001.
- Backup system has not grown

**Notes:**
Our initial system, a Sun E250 with 2x 400MHz processors and 4 AIT-3 tape drives, had an initial requirement that it must be able to back up a data set of 1TB within a 24 hour window.  Our initial rollout in July-August of 2001 had 35 clients and approximately 400GB of data.  Since that time, a single full backup is over 2.6TB with ~186 clients.  We had predicted that by this time, we would have at least increased the drive count from 4 to 6, moved onto AIT-4 drives, or both.  We’ve had to do neither.

We can continue to increase the lifespan of the current hardware by changing tape retention policies and tape rotations. This would be at the cost of slightly less protection.

## CSI Backup System - Example

**Notes:**
Even though total data backed up and the number of systems backed up have grown significantly, we have only made minimal increases in hardware.  In fact, the only increase was the addition of extra caching disk to handle the very large CMS filesystems.  The CSI backup system continues to perform well because the amount of daily delta, ~100+GB is manageable and the size of an individual file system ~<200GB is also easily manageable. Currently, approximately 1/3 to half of the total backup time is spent on generating tapes versus taking backups over the network.

However, as we expand this service to other groups, huge filesystems, 2 or more TB is size, will become common and can severely impact the performance of the server while generating new full volumes.  In addition, these huge filesystem may have an abnormal daily delta (CMS claims up to 3TB) which also can adversely affect the performance of the backup system.

These two factors, along with network performance, would be the biggest cause to requiring hardware additions/upgrades. Once these two issues are better understood, we will be better able to predict future hardware growth, requirements and costs.