# Presentation: 571137

## Tank S-102 Waste Spill

- Shirley Olinger
- Manager

- Jerry Long
- Chief Operating Officer

- <date/time>

- <footer>

***  ******        ******safety     ********* *****     performance       **********      cleanup      *** *******       closure***

**Notes:**
Shirley welcomes and make introductions.

## S-102 Spill Area – Friday, July 27, 2007About 85 gallons of mixed tank waste spilled

*S-102 Spill Area – Friday, July 27, 2007**About 85 gallons of mixed tank waste spilled*

**Notes:**
(Jerry)
I’m going to give you the details of an event that occurred on July 27 of this year. 

Before I go into specifics about the cause of the event, let me show you an 3-D animation that gives a clear account of the events of that day.

## S-102 Detail

*S-102 Detail*

***Not Shown to Scale***

***Ground Level***

***Pump Inlet***

***12” Riser***

***Tank Bottom***

***Tank Waste***

***Pump Motor***

***Pump Riser Extension Box (1) ***

***Dilution Water Hose In-Box (1b)***

***Hose-in-Hose Transfer Line (1a)***

**37’**

**75’**

***Sparge Water Hose In-Box (1c)***

***Cover Block***

**Sparge Water Hose Outside Box (3)**

***Relief Valve***

***Leak Detector***

**Dilution Water Hose Outside Box (2)**

***Pit Drain***

***Pump Seal (1d)***

- Potential Spill Sources

**Pump Riser Extension Box or Transfer Pit**
- Transfer Line
- Dilution Water Hose In-Box
- Sparge Water Hose In-Box
- Pump Motor Seal
**Sparge Water Hose Outside Box**
**Dilution Water Hose Outside Box**

***Transfer Pit***

**Notes:**
(Jerry)

Events as they occurred:
As procedures, workers reversed the pump and pump shut down
After several minutes tried “pump with auto reverse”
(Cleans the pump and clears the line)
Pump shuts down a second time
Workers fix the VFD and hit reverse button again
Amp goes up to 75amps and shuts down
Finally get it to pump  in reverse

## Scenario

- With the pump suction “strainer plate” within 1⁄2 - 1” from the tank floor, the high density waste heel was sufficient to block the free flow of relatively low density (Specific Gravity of about 1.1) of transfer waste during reverse rotation and subsequently pressurized the suction cavity of the pump, forcing water up the dilution line and/or sparge line.

***Tank Bottom***

**Dilution Line**
**Discharge**

***Bottom and Side-View Photo of Pump Inlet***

**18 in.**

***Hard Waste Heel***

**Sparge Line**
**Discharge**

*Scenario*

** Reverse rotation for 105 seconds @ 45Hz was performed 3 times*

**Notes:**
(Jerry)

## S-102 Event Summary – Dilution Hose Source of Spill

- Spill occurred between 2:05 a.m. and 2:20 a.m. while pump was operated in reverse direction
- No personnel in the tank farm
- Abnormal radiation dose rate readings identified at 2:20 a.m.
- Splash pattern consistent with release near dilution hose at NW corner of pump pit
- Spill pool consistent with near surface release
- Bathtub ring along north side of pump pit between pit and dilution hose
- No discoloration near potential release points from pit or pump box
- Transfer pit and pump riser extension box unlikely sources of spill
- Pump box drains into transfer pit OR into the tank
- Leak detector in transfer pit
- Transfer pit drains to tank
- Transfer pit is below grade with a void volume of over 1,500 gallons
- Drainage collected from dilution hose resulted in high dose rates confirming waste was in dilution hose

**Notes:**
(Jerry)

The dilution hose was the most probable scenario for the source of the spill.
Splash patterns appeared on jersey barriers, back side of chair, scaffolding and stairway. 
No splash pattern was located on the wood platform, top of pump box, and  top of pump pit.
The Sparge hose was not in the vicinity of the spray pattern. 
The Leak detector in transfer pit had no indication of leak. If it came from the pit, it would have had to have had 1,500 gallons of waste before it could leak out of pit to ground.

## S-102 Event Investigations

- Several Investigations
- DOE/ORP Reviews
- DOE Type A – Complete 09/19/07
- DOE EM-60 – Complete 09/27/07
- DOE/ORP Reviews:
- ORP Response – Complete 8/29/07
- HPI – Complete 10/23/07
- Assessment of ORP Management Systems – Complete 11/09/07
- CH2M HILL Internal Investigation
- Event Investigation/Root Cause Analysis – Completed 9/17/07
- Emergency Response Investigation – Completed 8/27/07
- Health Effects Investigation – Completed 10/4/07
- Engineering Design Program Review – Completed 10/9/07
- Direct Cause: Over-pressurization and rupture of the dilution water supply hose as a result of a plugged pump suction during reverse pump operation

**Notes:**
DOE/ORP Reviews:
Requested Type A Investigation – Complete 9/19/07
Conducted Investigation of ORP Actions Related to the Tank S-102 Spill – Complete 8/29/07
Conducted Human Performance Improvements Review – Complete 10/23/07
Assessment of ORP Management Systems – draft issued 10/31/07 ECD 11/9

CH2M HILL Internal Investigation:
Event Investigation/Root Cause Analysis – Completed 9/17/07
Emergency Response Investigation – Completed 8/27/07
Health Effects Investigation – Completed 10/4/07
Engineering Design Program Review – Completed 10/9/07

## CH2M HILL Root Cause and Contributors by ISMS Core Functions

**  ****Root Causes:**
**  ****Engineering Design **
**   ****Less than Adequate**
**  ****Change Control Process**
**   ****Less than Adequate**

**Notes:**
(Shirley)
Direct Cause
Over-pressurization and rupture of the dilution water supply hose as a result of a plugged pump suction during reverse pump operation
Root Causes
Engineering Design Less Than Adequate (LTA) – The design review and hazards evaluation for the S-102 retrieval system did not consider the raw water supply system as a potential waste transfer misroute path (physically connected), despite requirements contained in safety basis
Inadequate change management process for process safety management – S-102 retrieval system software, hardware and process control changes were not adequately evaluated against the safety basis accident analyses for local impacts
Contributing Causes
Formality of Operations LTA
Radiological Controls Survey Techniques LTA
Tank Farm Lighting LTA
Abnormal Operating Procedures (AOP) LTA
Process Controls LTA
Training LTA

## S-102 Corrective Actions 

- Comprehensive Corrective Action Plan
- Type A - Justification of Needs (JONs)
- CH2M HILL Investigation Findings
- EM-60 Input
- DOE ORP Internal Reviews
- Coordination of CAP Critical
- 2 Field Offices:  ORP and RL
- 3 Contractors
- CH2M HILL – Event Owner
- Fluor Hanford – Emergency Management
- Advance Medicine Hanford – Medical Evaluations/Communication

**Notes:**
(Shirley)
CAP includes Corrective Actions (CA) from 3 Prime Contractors and 2 Field Offices
Provided for review and comment to EM (and HSS) on 10/31/07
CAP to be approved by EM-1 by 11/17/07
CAP for Type A Judgments of Need
CH2M HILL – 50 CAs
Fluor Hanford – 23 CAs
Advance Medicine Hanford – 7 CAs 
ORP – 13 CAs
RL – 2 CAs

In addition, the other reviews have generated:
- DOE ORP CAs from internal reviews:  11 corrective actions plus corrective actions from ORP Oversight Assessment
- CH2M HILL CAs: 53 corrective actions:
  event root cause analysis – 47
  event response investigation – 2+
  health effects investigation – 4
  engineering design review – CAs included in other CAPs

## S-102 Corrective Actions (cont.)

- Focus areas of CAP for ORP
- Engineering and Quality Oversight
- Heavily focused on Authorization Basis document review and  approval and SSO oversight
- Need for increased oversight of process/system engineering recognized
- Need for internal assessments of ORP
- Conduct of Operations
- Although routine operations observed, emphasis has been on known higher risk/higher consequence evolutions
- Increase oversight of more routine operational activities, including backshift operations (i.e., tank retrieval operations)
- Heighten alertness when activities not occurring as planned (pump stuck or need for reverse operation)
- Radiological and Industrial Hygiene Coverage
- Programmatic reviews conducted
- Improve field operations coverage
- Factor in lessons learned
- ORP Response/Human Factors Investigations
- Strengthen the DOE Facility Representative’s ability to properly respond to abnormal and emergency events

**Notes:**
(Shirley)
Engineering and Quality Oversight:
Re-organization with focus on Engineering and QA
Chief Engineer reporting directly to site manager
Organization would include existing authorization basis engineers
New positions with focused responsibility on process/system engineering; more robust Hazops Analysis for operational upsets
Evaluate/verify processes for flow-down of technical/operational requirements
Evaluate SS classification for waste transfer systems
Conduct of Operations
Period of improving performance indicators observed over 3 year period (i.e., frequency of occurrences, consequence of occurrences, traditional safety indicators, etc.)
Most significant recent occurrence was Tank C-202 work in September 2005
CH2 executives continued to appropriately demonstrate a high level of attention to small operational issues
Evaluating best way to refresh technical oversight staff (FRs, SMEs)
Evaluating how to best assess the discipline and rigor of operations
Radiological and IH Coverage 
Integration of actions with procedural requirements
Communications with field operations personnel (S/M, pic, etc.)
Command and control of field activity (Authority)
Instrument use
Focus directly on specific actions of Health Physics Technicians
Seeking to hire field knowledgeable radiological control experts (Nuclear Navy/Shipyards)
ORP Response/Human Factors Investigations
Increase engineering staffing, reviews and oversight 
Evaluate Contractor’s engineering organization, including subcontractors
Conduct investigation into the human factors of the ORP response
Strengthen the DOE Facility Representative’s ability to properly respond to abnormal and emergency events
Clarify TSR Definition for PHYSICALLY CONNECTED
Improve processes to prepare Facility Representatives to perform on-call duties
Improve processes to prepare personnel for management duties for event response
Independent Investigation of ORP Oversight
Integrate and strengthen the ORP management systems, including the assessment program
Strengthen the ORP engineering oversight role – increase staff, supplement with support contractors; increase assessment activities; resist collateral duties; evaluate the contractor’s engineering processes
Increase ORP quality assurance staffing;
Perform internal assessments of ORP (QA group to perform)
Improve ORP oversight program – expand the use of the Operational Awareness database

## CH2M HILL Corrective Actions

- Engineering
- Improve engineering design review and hazards analysis/evaluation process
- Review strategies for controlling waste leaks – prevention instead of mitigation
- Management System
- Analyze, develop and deliver conduct of operations training
- Review effectiveness of causal analyses and corrective actions for significant events since January 1, 2003
- Strengthen management oversight plans for waste retrieval activities
- Refine work control process
- Safety and Health
- Ensure potentially exposed personnel are examined and documented by on-site medical provider
- Develop consistent notifications to patients regarding test results
- Strengthen post-exposure medical monitoring process
- Continue medical monitoring and surveillance of potentially affected workers

**Notes:**
(Jerry)
Engineering -
Organization - strengthen the Design Authority function
Design Review/Approval - increase independence and quality
Process Hazards Analysis - including high probability/low consequence events
Process Control Plan quality and flow down to operating procedures
Management of Change process implementation similar to 29 CFR 119
Control strategy to prevent or mitigate potential waste leaks, including the identification of a safety significant boundary for all above grade waste transfer components.  Safety Basis document revisions to incorporate.

Management Systems- 
Reinvigorate conduct of operations by training employees and follow-up with assessments
Improve oversight of transfers and retrievals
Revise transfer and retrieval procedures to incorporate S-102 Lessons Learned
Work Control-
- Better define troubleshooting controls
- Improve compliance with existing requirements for minor work
- Require compliance with minimum lighting requirements of HAZWOPER
- Develop and implement technologies for small-quantity waste spill detection

Safety & Health- 
Revise Abnormal Operating Procedures (AOP) to ensure that chemical monitoring is done in conjunction with radiological evaluations.
Provide additional training and information for shift managers, operations managers, industrial hygienists, and industrial hygiene technicians on proper response to abnormal events.
Implement a process for identifying potentially affected workers after an event.
Collect additional chemical analytical data from S-102.
Continued monitoring and surveillance of potentially affected workers from S-102.
Improve medical monitoring documentation and accountability of individuals with health concerns following an event.
Improve job hazard analysis to identify the appropriate work controls.
Identify jobs and operations that require continuous industrial hygiene monitoring, train industrial hygienists on continuous monitoring methods, and implement this monitoring.
Establish lighting standards for tank farms and improve lighting where needed.

## CH2M HILL Corrective Actions (cont.)

- Emergency Management
- Evaluate low consequence, high probability hazards for emergency response
- Evaluate technologies for identifying small waste leaks
- Revise/clarify procedures for activating response teams for non-emergency events
- Work Control
- Eliminate use of verbal work packages for activities comparable to manual pump rotation which required application of specific torque limits provided by vendor
- Industrial Hygiene
- Implement methods for continuous monitoring of vapors during waste transfers
- Integrate Industrial Hygiene response to abnormal events that may involve chemical release into abnormal response procedures.  Sampling requirements should be specified
- Radiological Protection
- Clarify radiological monitoring requirements associated with TSR and leak detection

**Notes:**
(Jerry)
Emergency Management- 
Revise Abnormal Operating Procedures (AOPs)to incorporate S-102 Lessons Learned
Training and drills.  Increase number of drills to assure understanding of AOPs
Improve accountability of personnel affected by events

Work Control- 
Better define troubleshooting controls
Improve compliance with existing requirements for minor work
Require compliance with minimum lighting requirements of HAZWOPER
Develop and implement technologies for small-quantity waste spill detection 

Industrial Hygiene-
Refine methods for continuous industrial hygiene monitoring during waste disturbing operations
Incorporate IH monitoring requirements in abnormal and emergency operating procedures
Improve IH specific procedures to compliment abnormal operating procedures

Radiological Protection
Clarify radiological monitoring requirements associated with TSR and leak detection
Provide training on procedures and expectations for abnormal and emergency response radiation monitoring methods

## CH2M HILL Lessons Learned

- “Worst-to-Best” DOE Complex Safety Statistics
- From 2005 to 2007:
- Total Recordables dropped 71%
- DART Rates dropped 82%
- Lost Workday Case Rate dropped 61%
- Stop Work process dramatically dropped 100%
- Occurrence Reports dropped 71%
- Skin and Clothing Contamination Reports (ORPS Reportables) dropped 100%
- Employee Concerns dropped by 62%

- Trend data indicated safe operations . . .

- 2007 America’s Safest Companies

**Notes:**
(Jerry)
Our safety stats are good, our stop works have reduced – even though employees continue to have the authority, they work through the issues with their management team

## Lessons Learned

- CH2M HILL
- Engineering Design / Hazards Analysis
- Formalize design review process and evaluate some total of all modifications.
- Conduct of Operations
- Complacency:  lighting, notifications, radiation readings, etc.
- More focus on ConOps training and implementation.
- Emergency Response
- Drills:  practice is important
- Focus on chemical hazards as well as radiological hazards

**Notes:**
Engineering Design & Hazards Analysis
Configuration of retrieval pump dilution line not considered “physically connected” to the waste transfer system when originally designed in 2002
DSA analyzed waste leak accidents from raw water supply lines connected to pumping systems, such as dilution lines
Dilution line terminated inside a strainer basket external to pump suction
Mindset of engineers analyzing S-102 design -- dilution line was not connected, therefore backflow prevention TSR requirements were not applicable
Design review comment regarding potential to pressurize dilution line during reverse flow was not addressed
Pump modifications in 2007 did not prompt reevaluation
Dilution line supply location moved - integral to pump suction cavity
Suction screen replaced with “strainer plate” that reduced exit area by more than 50% 
Reverse pump speed tripled, increasing pressure in suction cavity 9X
Conduct of Operations
Tank farm lighting did not meet HAZWOPER standards
Radiological control technicians did not recognize importance of open window beta readings during response to unexpected high radiation levels
Operators unaware of design basis radiation levels
Operating and response procedures did not provide sufficient direction for response to upset conditions
Operations support personnel not trained on system operation
Some notifications delayed or incomplete
Work control deficiencies
Emergency Response
Response limited to radiological hazard - Insufficient attention to potential chemical hazard
Potential health effects of chemical exposure not adequately evaluated
Affected personnel not identified in a timely manner
Alert levels did not include “high probability – low consequence” events such as small quantity waste spill
  Lessons Learned
Develop alert levels for local impact events below DSA accident analysis thresholds
Conduct small-scale local-impact drills to evaluate abnormal/emergency response

**Notes:**
(Shirley)
ORPS Recurring Events:  1Q FY07 – Inadvertent forklift impacts

Electrical Safety: 4Q FY06 – Energized wire found while performing safe-to-work check of field electrical skid

Environmental Permits:  3Q FY07 – Temporary transfer line management plan NOV. 

**Notes:**
(Shirley)
TSRs: 1Q FY07 - Violation of Technical Safety Requirement during a Waste Transfer; 2Q FY 2007, where a Caustic Flush of 702-AZ Ventilation Condensate Drain System Causes AZ-301 Waste to Fall Outside of Waste Category L Criteria.

## Lessons Learned

- Good safety performance metrics can give a false sense of security
- Key performance indicators – TRC/DART, Radiological, TSR, Electrical Safety, ORPs, PERs, etc.
- ISMS performance measures
- Safety culture indicators – VPP, Safe Work Environment personnel survey results, Hanford Concerns Council
- Reduction in employee concerns
- ORP assessments – more is not necessarily better
- ORP completed 38 assessments of the TFC in FY2007
- Facility Representative oversight of retrieval operations: no procedure compliance problems
- Quarterly Assessment Program evaluations, Quarterly recurring events evaluations did not indicate negative trend in operational performance
- Facility Representative observations of S-102 retrieval operations 1 and 2 days prior to the event indicated good conduct of operations
- Past successes do not guarantee future success
- Successfully completed 7 tank retrievals
- Successfully deployed similar retrieval system at tank S-112

**Notes:**
(Shirley)
CH2 went from one of worst in 2002/3 to one of best in 2006/7. Recently voted as one of safety companies in nation
TRC = 0.32; DART = 0.32
  - lowest rate since January 1994
  - DOE 2006 TRC = 1.6
  - TFC has reached over one million hours worked without a lost time injury 3 times since January 2006
  Most all metrics on dashboard green, especially in the last 4 qtrs
  Safety Culture good (VPP, SWE, HCC, ECP, workers and mgmt working together).  They seemed to get it, good safety results in good work and $$ to company.  
Decrease in ORPs; Recurring Events
38 assessments of the TFC conducted in FY07: 
6 assessments on the Safety Basis; 5 SSO/Engineering assessments; 13 operations/Integrated Safety Management; 2 Quality Assurance; 5 Radiological Controls 
The last TF spill of rad/chemical tank waste was January 6, 2000, during saltwell pumping of tank S-103. 

## Lessons Learned (cont.)

- Hind-sight indicators provide vital information
- Several contractor lay-offs due to funding cuts
- Contractor senior manager for Closure Operations distracted due to medical and other reasons
- S Tank Farm housekeeping less than adequate
- Vacant position for Retrieval FPD, rad engineer, industrial hygiene for several months
- Inadequate DOE oversight of contractor engineering programs
- Not enough FTE’s to oversee 2 major projects

**Notes:**
(Shirley)
Contractor Lay-offs/Unengaged Closure Ops VP should have alerted us to watch more closely and insist on more indep oversight by CH2
Last 2-3 years cleaning up all farms but S-Farm (AP, AN, C)
ORP STAFF ISSUES:
Managed contract not contractor was basis for reducing the FTE ceiling at ORP from 150 need in Boston’s regime to less than 100 in Schepen’s regime.
I believe we need closer to the 150 FTE ceiling to adequately be cognizant of all the technical aspects of these 2 major projects where we aren’t reducing the footprints but rather significantly increasing the footprints.
Don’t have flexibility in TF funding to hire GSSCs to supplement fed oversight like we have w/WTP 
ACTIONS:
Increase Engineering staff and perform more formal assessments of TFC engineering processes (not just DSA)
Establish Internal Assessment program to look at us
Routinely conduct FRs surprise/backshift area surveillances
Designate an Honest Broker  when evaluate data.  
Establish coach for FRs to sharpen their skills
Establish a “minimum cadre” (includes experience not just #s) using rules of thumb

## Next Steps

- Issue Independent ORP Oversight Assessment Report and develop Corrective Actions
- EM-1 Approve Type A CAP
- Complete ISMS Declaration
- Conduct Contractor Readiness Assessment to resume retrieval of tank C-109; ORP Manager start-up approval authority

**Notes:**
(Shirley)