# Presentation: 545183

## LBNL Incident Response and Forensics

**Jim Mellander**
- Incident Response Manager
- Cyber Protection Program
- Lawrence Berkeley National Laboratory
**jmellander@lbl.gov**

## Incident Response Cycle

- Detection
- Triage
- Forensic Examination
- Remediation
- Lessons Learned

## Detection

- Cornerstone of incident detection is the use of IDS software.
- LBNL uses BRO
- Tcpdump used to bulkrecord traffic for later forensic analysis.
- Central Syslog Server
- Bondo – custom rootkit detector
- Knowledge of network usage patterns and examination of anomalies.
- Download of toolkits.
- IRC – many hackers brag on IRC, or set up bouncers.
- New services and/or open ports appearing on systems.
- Scanning activity – either internal or external.

## Triage

- Block Internet Access
- Automatically by BRO, if threat profile high enough.
- Manually, when detected compromised.
- Block Internal Access
- DHCP Hosts – don’t give out lease
- Turn off port on switch
- Collateral damage – other hosts on switch
- Physically disconnecting from the network.
- User/Sysadmin communication
- Upon security incident, system is no longer under control of users, but security staff.
- Don’t do anything on or to system, without explicit instructions from security staff.

## Forensic Examination 1

- Considerations
- Legal – Evidence Preservation, Chain of Custody.
- Preservation of evidence on running system.
- Accurate forensic images.
- Getting system back up & running.
- Examine system before shutting down
- Dangerous in some cases, as it can damage information.
- If system has robust filesystems, often best to just pull the plug.

## Forensic Examination 2

- Forensic Imaging
- Commercial Toolkits – vetted for evidence preservation.
- Non-commercial/Freeware – often easier/faster.
*dd* – Bit Image Copy
*buffer* – my favorite: superfast due to double buffered operation.  Source code available from http://www.hello-penguin.com/software.htm or http://packages.debian.org/stable/source/buffer

## Forensic Examination 3

- Use Bootable CD to forestall any possibility of contamination during forensics
- F.I.R.E. is one such Linux environment
- System Data Examination
- Changed files with timestamp >= compromise time.
- Curiously enough, hackers rarely change timestamps, although it would be easy to do.
- Directories with weird names: ... or ‘ ‘ (space)
- Logfiles
- Again, many hackers do not change logs
- Central syslog server preserves data, even if host logfiles damaged/deleted.

## Forensic Examination 4

- Logfile Data Examination
- Determine offset of system clock from actual time by correlating system logfiles with known network event timestamps.
- Reconstruct session data from bulk recorded tcpdump.
- Develop timeline of events
- Determine further extent of compromise
- Hacker tool download
- Following path of hackers in downloading tools provides insight into methods
- Usually performed from offsite address to avoid correlation by hackers with LBL.

## Remediation

- If system state can be reliably returned to pre-compromise condition, and compromise vector plugged...
- Remove all vestiges of hacker activity, or
- Restore from backup
- ... then system does not need to be rebuilt
- Otherwise,
- If hackers were able to perform unscrutinized activity...
- Encrypted sessions,
- Toolkit action unclear
- ... then system need to be rebuilt from media, with appropriate safeguards to ensure compromise doesn’t reoccur.

## Lessons Learned 1

- Examine what went wrong
- Vulnerability?
- Mechanisms?
- Shared authentication often extends compromise to multiple systems
- Procedures?
- Lax authentication processes
- User education?
- Lack of awareness and/or concern with security.
- Sometimes arrogance, actually – “I’m a brilliant scientist, so I can take shortcuts”...
- “It happens to other people, but not to me”

## Lessons Learned 2

- Determine what will prevent re-occurrences in the future.
- Vulnerability?
- Patching
- Scanning
- Mechanisms?
- One Time Passwords?
- Procedures?
- Tighter controls on authentication mechanisms.
- User education?
- Hot Stove is the best teacher – you only get burned once before you learn your lesson.
- Education – even brilliant scientists (or security experts) can learn more about security
- What motivates people to be involved in their own cybersecurity?

## Questions???