# Presentation: 517678

## Fermi Computer Incident Response

- Michael Diesburg
- Fermi Computer Incident Response Team
- (FCIRT)
- March 22, 2005

## Computer Security Plan

- FCIRT operating rules are laid out in CSPP
- CSPP defines 3 levels of incidents:
- Non-issue
- SMOKE: _S_ystem _M_anagers _OK_urance _E_valuation
- FIRE: _F_CIRT _I_ncident _R_esponse _E_mergency
- CSPP defines reporting procedures to keep line management informed during serious incidents (FIRES)
- Defines authority of FCIRT to take control of resources and manpower during FIREs

## FCIRT Operation

- Response consists of several parts:
- Containment
- Removal from network, network blocks
- Classification
- No incident, SMOKE, FIRE
- Investigation
- Determine root cause that allowed breakin
- Restoration to service
- Define action necessary to return to service
- Assessment
- Evaluate procedures with CST, relevant parties

## Incident Classification

- Non-Incident
- No actual compromise or significant threat to any Fermi resources
- SPAM from outside FNAL
- Non active virus laden files reported by AV
- “Normal”, ineffective probing from off site
- Instances of inappropriate use that are actually line management issues (and not visible from outside).   I.e. when the computer is only an incidental tool to the actual infraction

## Incident Classification

- SMOKE
- Security issues exist which need to be addressed but are limited in scope and threat level
- Known viruses confined to single system and not in a Critical System (Major Application)
- Configuration deficiencies which require attention of management personnel
- Suspicious but not obviously threatening activity which requires further investigation by system managers or FCIRT

## Incident Classification

- FIRE
- Defined by scope and visibility
- Any computer activity which threatens the operation of the laboratory
- Threat to Critical System (Major Application)
- Threat of data loss
- Possibility of public embarrassment (activity noted by outsiders)
- Disgruntled employees (TEPCSR)
- Need to invoke authority
- Illegal activity or involvement of outside agencies

## Serious Incidents

- Rate of FIREs ~ 1/month
- 2001	18
- 2002	  7
- 2003	  9 (+ 1 TEPCSR)
- 2004	12 (+ 3 TEPCSR)
- 2005	 None to date (1)

## Overview of Incidents - 2003

- 01-17 csdserver1
- Problem:  Web defacement
- Alert: CIAC, published on zone-h.org
- Root Cause: Unpatched application (Frontpage)
- 01-25 SQL Server
- Problem: root compromise
- Alert: mail to _[abuse@fnal.gov](mailto:abuse@fnal.gov)_
- Root Cause: Unpatched application (MS SQL)
- 04-15 Samba Server
- Problem: root compromise
- Alert: Autoblocker, FBI
- Root Cause: Unpatched application (samba).  Update failed. Threat noted on 7th, patched on 8th , entered on 13th
- 05-13 beamssrv1
- Problem: root compromise
- Alert: Internal user, drives not available
- Root Cause: weak password/TS/stolen laptop.  Not resolved

## Overview of Incidents - 2003

- 07-07 DOE-hep.hep.net
- Problem: Web defacement
- Alert: Internal user
- Root Cause: Inappropriate configuration/pilot error
- 07-18 Cisco IOS router exploit
- Problem: Significant vulnerability (no compromise)
- Alert: CIAC
- Root Cause:  NA
- 07-23 Unauthorized file distribution
- Problem: Unauthorized distribution of IP
- Alert: External, MPAA
- Root Cause: Unpatched application (MS SQL)
- 08-19 W32/Sobig virus
- Problem: Widespread infection
- Alert: Internal alert, CIAC
- Root Cause: Unpatched machines. Late AV signatures.  Pilot error
- 08-19 Nachi/Welchia
- Problem: Widespread infection
- Alert: Internal
- Root Cause: Unpatched machines. Late AV signatures. Pilot error

## Overview of Incidents - 2004

- 02-15 Infected systems (Netsky)
- Problem: Widespread infection
- Alert: Internal
- Root Cause: Unpatched machines. Late AV signatures. Overloaded mail gw not scannin plain/txt files.  Missing some zip files. Pilot error
- 03-20 Black Ice
- Problem: Widespread compromise
- Alert: Autoblocker
- Root Cause: Unpatched application (black Ice)
- 04-08 Web site hacked
- Problem: Web site defacement
- Alert: Internal
- Root Cause: Unpatched application (Serv-U ftp)
- 04-21 bdcry026
- Problem: root compromise
- Alert: Internal, admin noted unauthorized logins
- Root Cause: Unpatched application (MS SQL)
- 05-13 Increased inbound scanning
- Problem: Off-site connectivity impacted by scans. Routers overloaded
- Alert: Autoblocker
- Root Cause: Inbound port 5554/9898 scans

## Overview of Incidents - 2004

- 08-04 fcdfdata28
- Problem: root compromise
- Alert: External via inside channels
- Root Cause: Pilot error (password stored in file on remote system)
- 08-09 tdpc355
- Problem: Widespread infection
- Alert: Internal
- Root Cause: Unpatched machine, Pilot error. Late AV signature
- 09-01lss-recfacility
- Problem: root compromise
- Alert: Internal, Autoblocker
- Root Cause: Unpatched application (MS SQL)
- 09-22 laptop-sopczak
- Problem: Widespread infection
- Alert: Autoblocker
- Root Cause: Pilot error.  Rode in on laptop
- 09-28 trojan on fcdflnx3
- Problem: User level compromise
- Alert: CIAC
- Root Cause: Remote root comrpomise + X session hijacking
- 10-21 Fermi Web site issue
- Problem: Web defacement
- Alert: External (DOE CIOs office)
- Root Cause: Pilot Error.  Inappropriately configured application

## Incident Reporting

- For all serious incidents as described above, the CSPP defines reporting procedures:
- Inform CSExec
- Inform relevant system manager(s)
- Inform relevant Division Head
- Inform Public Information office if there is significant visibility from off-site
- Inform CIAC, DOE, CI and IG if incident meets their reporting guidelines

## Summary of Incident Causes

- 2003
- Unpatched systems	5(4)
- Bad configuration		1
- Update failure		1
- Unknown			1
- 2004
- Unpatched systems	6(4)
- Bad configuration		1
- Pilot error		3
- Other			1
- First two categories in above list should be preventable.  Possibly only 4 in first category for each year.

## Preventative Actions

- Unpatched vulnerabilities are a clear problem
- Two pronged approach:
- Identify and scan for critical vulnerabilities
- Ongoing problem as new holes emerge
- Time is of the essence
- See R. Reitz talk
- Patch/Configuration management
- May not cover non mainstream apps, but should cover biggest holes
- Need to identify stealth installations
- See M. Kaletka talk
- Pilot error will always be with us
- Ongoing educational efforts
- Computer Security Awareness Day
- Articles in Fermi Today
- It’s a 00.05% problem