# Presentation: 002428

## LogJAMM: Logs Are Just Another Monitoring Mechanism

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.

**The acknowledgement statement **_**MUST**_** be used on the title slide****of all presentation material distributed outside of Sandia.**

- Creating a system to record, query and analyze real-time log data
- June 12, 2007
- Paul Sery
- System Administrator

## The Cuckoo's Egg

- Cliff Stoll's first person account of extreme system administration
- He was just starting a temporary job as a sysadmin
- Noticed a 75 cent accounting error in his logs
- Unraveled an international spy ring
- An extraordinary tale of dedication and deduction

## We don't expect computer users to read their logs
But we do expect our system administrators to do so
Do they?
We don’t really want to know the answer
Example: our logging client SSL certificate expired:
stunnel can't connect to server
it tries to reconnect every 10 seconds
thousands of log messages piled up on about 60 boxes
my email address was in each message!
I didn't get a single complaint or query

- But we do expect our system administrators to do so
- Do they?
- We don’t really want to know the answer
- Example: our logging client SSL certificate expired:
- stunnel can't connect to server
- it tries to reconnect every 10 seconds
- thousands of log messages piled up on about 60 boxes
- my email address was in each message!
- I didn't get a single complaint or query

- Is the read bit set on /var/log files?

## Is the read bit set for /var/log files?

- This is not really a surprising result
- reading logs is a boring, mind numbing process
- easy to forget
- easy to ignore
- even when we do read logs it's easy to miss important events in the clutter

## What's the solution?

- Hire Cliff Stole
- Unfortunately, he's retired from system administration
- And there’s no silver bullet so we need to do what's possible:
- gather data
- provide better tools
- encourage system administrators to read & digest their logs
- incrementally improve with time
- Thus a central logging system:
- centralize log data collection
- make it easier to read logs
- make it easier to analyze logs
- create automated analysis systems
- research advanced techniques

## LogJAMM Overview

- The remaining slides describe the
- system architecture
- server cluster configuration
- client installation and configuration
- query and analysis system
- future

## System architecture

- Client-Server
- Clients sends log data to server
- Log transmissions can be encrypted/unencrypted
- Server collects and stores data
- Analysis machine downloads, archives, crunches and makes the results available to certified system administrators

## Server Cluster Configuration

- Log servers perform single function: accept incoming log messages
- Record events to simple text files
- One text file per client per day
- Fail-over using Linux Heartbeat Project
- Simple, Round-robin IP load balancing
- Analysis performed on separate machine
- Log analysis machine provides web interface
- Raw logs downloaded daily from servers
- Raw logs homogenized and stored in database
- Further data reduction, correlation, etc.
- Reliable and secure data delivery
- Use syslog-ng instead of syslog to get TCP
- TCP ensures reliable data transmission
- UDP can lose data
- Optionally use SSL to encrypt data transmission
- prevents unauthorized viewing of data
- prevents tampering with data in transit

## LogJAMM Linux Client Installation

- Make use of the Linux Common Operating Environment (COE)
- COE provides distribution and update mechanism
- Use syslog-ng from Fedora Extras repository
- Created our own syslog-ng configuration RPM
- Quick Linux COE overview
- Linux Common Operating Environment
- Standardizes and simplifies Linux installations
- Customize Kickstart  on first RHEL iso image
- Use RHEL site license and satellite servers
- Install Red Hat and custom Sandia RPMS
- COE spearheaded by Tim Seigler tseigl@sandia.gov

## Windows and UNIX Client installation

- Use open source Snare to transmit Windows Eventlogs
- Use an installation script to install Snare
- Manually configure syslog on non-RPM Linux and UNIX

## LogJAMM Client configuration

- Duplicates traditional syslog configuration
- syslog-ng continues to log locally
- local logs store same information
- local logs use same file structure
- we don't want to interfere with existing scripts, etc.
- All events are transmitted to remote log server
- Adds facility and priority information not stored locally

## Client installation status

- Total of 1300 Linux/UNIX/Window clients (5/30/07)
- Up from about 100 last year at this time
- Only about 600-650 are logging at any time
- We have 2-3,000 Linux/UNIX machines
- Many more Windows boxes
- COE is primary Linux distribution mechanism
- Few COE installs after RHEL4 surge last year
- Expect big increase when RHEL5 introduced

## Query and Analysis System 

- Designed for System Administrators
- Administrators have lots of jobs and little time
- Our job is to make their job easier
- Help them to consistently and thoroughly review logs
- To this end:
- Remove need to log into individual machines
- Remove need to sift through repetitive log data
- Use system administrators as our de-facto IDS

## System Administrator tools

- Accessed through a web page
- https://logjamm.sandia.gov
- Access to query/analysis tools
- Requires Sandia user name and Kerberos password
- Access limited to certified system administrators
- You can access log data if NWIS says you can
- See sysadmin.sandia.gov for more information on the Sandia Certified System Administrator program

## Query Tools

- Search raw logs
- Search homogenized (normalized) logs
- Daily e-mail of homogenized & new logs (by request)
- Search using regular expressions
- Search using self-generated indexes (tokens)
- Search for new, never before seen events
- Instantaneous e-mail of OSSEC alerts
- OSSEC provides signature-based, alerts

## Raw Logs

- Use the web page to look at your raw logs
- Don’t have to interactively log into each machine,
- or write your own scripts
- Display by machine, date, facility and priority
- You can also do text searches
- One-stop-shopping makes reading logs easier

## Raw log example

- Mar 26 04:10:01 xyz crond[13283]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 04:20:01 xyz crond[13295]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 04:22:01 xyz crond[13298]: (root) CMD (run-parts /etc/cron.weekly)
- Mar 26 04:24:45 xyz anacron[1766]: Updated timestamp for job `cron.weekly' to 2006-03-26
- Mar 26 04:30:01 xyz crond[1770]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 04:40:01 xyz crond[1774]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 04:50:01 xyz crond[1778]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 05:00:01 xyz crond[1858]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 05:01:01 xyz crond[1881]: (root) CMD (run-parts /etc/cron.hourly)
- Mar 26 05:10:01 xyz crond[1993]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 05:20:01 xyz crond[1997]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 05:30:01 xyz crond[2001]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 05:40:01 xyz crond[2006]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 05:50:01 xyz crond[2010]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 06:00:01 xyz crond[2024]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 06:01:01 xyz crond[2027]: (root) CMD (run-parts /etc/cron.hourly)
- Mar 26 06:10:01 xyz crond[2030]: (root) CMD (/usr/lib/sa/sa1 1 1)
- Mar 26 06:20:02 xyz crond[2034]: (root) CMD (/usr/lib/sa/sa1 1 1)
- and on, and on, page after page

## Homogenized Logs

- Log messages contains static and variable information
- Static: program name, message header, etc.
- Variable: IP addresses, PIDS, etc.
- Stripping out – homogenizing - variable information reduces visual “noise”
- Homogenized logs are much easier to scan
- Unusual events tend to stand out
- Normal computer logs can be 10s, 100s, 1000s of lines per day
- Homogenizing drastically reduces total number of messages
- Many messages are exactly alike once variables stripped

## Homogenized Logs Example 1

- Count Homogenized message
- 1       anacron: Updated timestamp for job cron.daily to ...-...-...
- 171     crond(pam_unix)[...]: session closed for user root
- 170     crond(pam_unix)[...]: session opened for user root by (uid=...)
- 144     crond: (root) CMD (/usr/lib/sa/sa... ... ...)
- 1       crond: (root) CMD (run-parts /etc/cron.daily)
- 23      crond: (root) CMD (run-parts /etc/cron.hourly)
- 341     crond: pam_krb...[...]: ccache dir: /tmp
- 341     crond: pam_krb...[...]: configured realm dce.sandia.gov
- 170     crond: pam_krb...[...]: no v... creds for user root, skipping session cleanup
- 170     crond: pam_krb...[...]: no v... creds for user root, skipping session setup
- 170     crond: pam_krb...[...]: pam_close_session returning ... (Success)
- 170     crond: pam_krb...[...]: pam_open_session returning ... (Success)
- 341     crond: pam_krb...[...]: renewable lifetime: ...
- 341     crond: pam_krb...[...]: ticket lifetime: ...
- 341     crond: pam_krb...[...]:...: /etc/v...srvtab
- 341     crond: pam_krb...[...]:...: forwardable not proxiable
- 341     crond: pam_krb...[...]:...: Kerberos ...
- 341     crond: pam_krb...[...]:...: no ignore_afs
- 341     crond: pam_krb...[...]:...: no krb..._convert
- 341     crond: pam_krb...[...]:...: user_check
- 341     crond: pam_krb...[...]:...: validate
- 341     crond: pam_krb...[...]:...: warn
- 5       rhnsd: running program /usr/sbin/rhn_check
- 1       sendmail:...: to=root, ctladdr=root (.../...), delay=...:...:..., xdelay=...:...:..., mailer=local, pri=..., dsn=..........., stat=Sent
- 14      stunnel: ... connected from ...............:...
- 9       stunnel: VERIFY OK: depth=..., /C=US/ST=NM/L=Albuquerque/O=Sandia National Laboratories/OU=CSUSP/CN=central syslog _[admin/](mailto:admin/emailAddress%3Dpgsery@sandia.gov)__[emailAddress](mailto:admin/emailAddress%3Dpgsery@sandia.gov)__[=](mailto:admin/emailAddress%3Dpgsery@sandia.gov)__[pgsery@sandia.gov](mailto:admin/emailAddress%3Dpgsery@sandia.gov)_
- 14      syslog-ng[...]: Connection broken to AF_INET(...............:...), reopening in ... seconds

## Homogenized Logs Example 1

- Example 1 summarized over 2,000 normal log messages
- But very little action took place on the machine
- Many cron jobs ran but not much else
- Have to sift through a lot of clutter to determine that the logs aren’t interesting
- How do see a tree for the forest?

## Homogenized logs example 2

- 185	crond(pam_unix)[...]: session closed for user root
- 184	crond(pam_unix)[...]: session opened for user root by (uid=...)
- 131	crond: (root) CMD (/usr/lib/sa/sa... ... ...)
- 1	crond: (root) CMD (/usr/local/sbin/CRONJOBS/normalize_messages.pl)
- 1	crond: (root) CMD (/usr/local/sbin/CRONJOBS/xfer_logs....sh )
- 21	crond: (root) CMD (/usr/sbin/ntpdate -s ntp-srn.sandia.gov > /dev/null)
- 3	crond: (root) CMD (/usr/sbin/up...date -u > /dev/null)
- 22	crond: (root) CMD (run-parts /etc/cron.hourly)
- 150	crond: pam_krb...[...]: ccache dir: /tmp
- 149	crond: pam_krb...[...]: configured realm dce.sandia.gov
- 184	crond: pam_krb...[...]: no v... creds for user root, skipping session cleanup
- 184	crond: pam_krb...[...]: no v... creds for user root, skipping session setup
- 74	crond: pam_krb...[...]: pam_close_session returning ... (Success)
- 74	crond: pam_krb...[...]: pam_open_session returning ... (Success)
- ...
- 1	httpd: [error] [client  SRN_ADDRESS ] script not found or unable to stat: /var/www/cgi-bin/bash
- 1	httpd: [error] [client  SRN_ADDRESS ] script not found or unable to stat: /var/www/cgi-bin/csh
- 1	httpd: [error] [client  SRN_ADDRESS ] script not found or unable to stat: /var/www/cgi-bin/finger
- 1	httpd: [error] [client  SRN_ADDRESS ] script not found or unable to stat: /var/www/cgi-bin/hamcards.cgi
- 1	httpd: [error] [client  SRN_ADDRESS ] script not found or unable to stat: /var/www/cgi-bin/maillist.pl
- 1	httpd: [error] [client  SRN_ADDRESS ] script not found or unable to stat: /var/www/cgi-bin/noN-exIstAnt_scriPt.lp
- 1	httpd: [error] [client  SRN_ADDRESS ] script not found or unable to stat: /var/www/cgi-bin/nph-test-cgi
- 1	httpd: [error] [client  SRN_ADDRESS ] script not found or unable to stat: /var/www/cgi-bin/postcard.pl

## Homogenized logs example 2

- Example 2 clearly shows some interesting events
- Notice the script execution attempts!
- This is “script kiddie” stuff commonly seen by Internet-facing web servers
- But this web server isn't facing the Internet!
- Turned out to be a Security scan
- Spotting such events against the noisy background of “raw” log events is difficult

## Raw log text string searches

- You can search your logs using text strings
- Standard, but useful capability
- Similar to “grep xyz /var/log/*”
- But you don't have to login to individual machines

## Self-generated index searches

- Problem: how to find similar messages
- traditional solution: string or regex search
- but, to some extent, you need to know what you're looking for
- How do you search for “something like xyz”?
- Possible solution: use self-generated index searches
- create numeric indexes (tokens) from the log message
- convert words and phrases into numbers
- search for phrases numerically “close” to each other

## Self-generated index searches

- Convert each letter into its ASCII equivalent
- For instance, abc = 304 and xyz = 373
- “I know my abc's by heart” = 2119
- “I know my xyz's by heart” = 2188
- The sentences differ numerically by about 3%
- Use the indexes to search for similar sentences
- Using regular expressions to do this is harder
- Homogenized log messages generated by the same program tend to group together
- For instance, failed ssh logins look like:
**sshd: Failed password for bob from ::ffff: SRN_ADDRESS port ... ssh...**
- Use the index concept to search for similar events
- Varying the index by 5% reveals messages like:
**sshd: Failed password for bobz from ::ffff: SRN_ADDRESS port ... ssh...**

## Self-generated index searches

- Generate numeric indexes for the program name
- the first word in the message
-  and the remaining words
- sshd: Failed password for bob from ::ffff: SRN_ADDRESS port ... ssh...
*sshd:* is the program index
*Failed* is the first word index
*password for bob ...* is the remainder index
- Resulting indexes: 492, 581, 4813
- Using numeric indexes helps find both exact and similar events in the database
- Index searches of complex messages break down
- “I know my abc's by heart” = 2119
- “I don't know my xyz's very well” = 2835
- You might need to correlate such messages, but they differ by over 25%
- So far, it appears that 20% is the effective limit

## Self-generated index searches

- This is a beginning
- Many possible improvements:
- Perhaps split “remainder” into two+ indexes, which will provide finer grained control
- Perhaps create an index for every word in the message “sentence”

## Daily homogenized log e-mails

- Schedule daily e-mails of homogenized logs
- Have your cup of coffee while reviewing yesterday's “summarized” logs
- Quickly see what happened on your machines
- Simple, easily performed job
- I've been using it for 1⁄2 year
- I find it to be a very effective tool

## Real-time alerts

- The Simple Event Correlator (SEC) provides real-time syslog pattern matching
- Search individual events for patterns
- Search sequence of events for patterns
- Specify action to take when pattern matched
- Enter patterns for your machines via web page
- Interactively search results or receive e-mails

## OSSEC Alerts

- Has over 800 UNIX/Linux, Windows, Snort, etc. signatures
- Matches atomic and multiple events against signatures and alert when threshold is exceeded
- We've integrated into our log collection system
- System administrators select machines to receive alerts for (select a threshold) via web page
- LogJAMM instantaneously sends alerts via e-mail

## Local vs Remote Log Comparisons

- Local and remote logs should match up
- Four reasons they won't:
- Network trouble – logs don't get to the server
- Race conditions – SSL latency drops initial messages
- Server trouble – no server to save to
- Hacker wipes logs!
- The last reason is very interesting

## Simple log comparison

- Checksum yesterday's local logs
- Transmit checksum to server via syslog-ng
- Server compares to its copy vs. client checksums
- This works but #1-3 (from previous slide) can cause mismatch
- Right now too many false-positives
- Perhaps better syslog-ng configuration will reduce the number

## Direct Log Comparisons

- Better to copy yesterday's logs to server
- Server directly compares local vs remote
- Differences stored in db and mailed to sysadmin
- Small problems like SSL start-up latency are obvious
- So are big problems
- More difficult to setup than checksum method
- Need cron job or daemon to transmit logs
- Use more network resources
- Most useful for secure servers
- Voluntary for now

## New, never before seen events

- Use normalized message database
- Use self-generated indexes
- Compare today's events to all previous ones
- Record, display e-mail new events
- Hopefully, interesting trends and interesting events will be noticed

## Future

- Anomaly detection
- Integrate Sisyphus data mining analysis system
- Created to debug Red Storm
- Researching a State-full Log-based IDS (SLIDS)
- Want many more clients
- 2000 per-day is our immediate goal
- The more data, the better
- Investigate other analysis tools