# Presentation: 082762

## OpenSSH Monitoring and Analysis

**Scott Campbell**
**NERSC/LBNL**

## Overview

**What does the instrumented SSHD do?**
**Why are we doing this?**
**Overall design considerations.**
**Overview of:**
**	****sshd core and related software**
**	****transport **
**	****bro policy analysis**
**Future Modifications.**

## Functional Goals

**Record session/keystroke data.**
**Record user login information.**
**Identify and record SSH events such as channel creation and command execution.**
**Track scp and sftp data movements.**
**Integrate PSC performance mods.**
**Tunnel data across encrypted channel to be digested and analyzed by bro.**
**Analyze data stream for hostile content.**

## Stuff not done

**Data is a text stream – there is no notion of process accounting or system log level detail.**
**No cleaning of data – meta data such as colors etc still show up as junk.**
**No application reassembly – vi and emacs provide a somewhat murky picture.**

## Purpose of Project

**Gain insight into user activity: user actions are opaque – we want to get to an attacker in real time.  **
**Leverage experience and analysis from clear text era.**
**Augment process accounting which is excruciating and difficult to process in a timely manner.  Limitations such as no command arguments in AIX.**

## Design Considerations

**Focus on not impacting our users:**
**Minimize performance impact.**
**Identical user experience.**
**Minimize changes to existing software.**
**Avoid additional SSH failure modes or security exposure.**

## More Design Considerations

**Note the lack of explicit security requirements – this is a big change so by minimizing impact on users and system admins we ease transition path and get buy in.**
**Make SSHD end as bullet proof as possible: put more fragile analysis and ‘smarts’ away from users.**

## Design Overview

- SSHD

- Stunnel

- Stunnel

- Log
- Parser

- Bro

- Client System

- Analysis System

## SSHD Design Overview

**Mechanics: startup, server id and client id.**
**Instrumentation: when, where and defensive strategies for users accidents.**
**Passwords: the 98% solution.**
**Decoupling the write.**
**PSC Mods – while we were in there ...**

## Startup

**At startup several things happen to log main server process.**
**Message sent that sshd has started:**
**sshd_start 127.0.0.1:2222/tcp 4102_spud_2222**
**Creation of server session id composed of:**
**<process-id>_<hostname>_<port>**
**This id will be used to identify any further activity from this sshd instance.**

## Startup: Client Connect

**When a client connects, it is assigned a session id equal to the pid of the child process.**
**Process mechanics avoid collisions.  If one occurs (race condition, logging failure etc), bro will restart counters for client – ie not pathological.**

## Login Information

**Logging looks like:**
**1192854558.838989 #2 ssh_connection_start**
**   ****127.0.0.1:32812/tcp > 127.0.0.1:2222/tcp**
**1192854559.428210 #2 auth_ok scottc publickey**
**   ****127.0.0.1:32812/tcp > 127.0.0.1:2222/tcp**
**1192854559.481470 #2 new_session SSH2**
**1192854559.848240 #2:0 new_channel_session pty-req**
**1192854559.860270 #2:0 new_channel_session shell**

- Bro Session ID

- Channel Number

## Instrumentation

**After user logs in, we begin keystroke logging in earnest.  Most data encapsulated in *****data_server***** and *****data_client***** events.  **
**Clean any non-ascii data into the record to avoid binary interpretation issues.**
**Bro sanitizes it’s end as well.  **

## User Games

**Copying all data is expensive and provides a diminishing return.**
**Limit volume of data from client/server in a single line.**
**Limit number of lines before summary byte count.**
**Count reset by client data.**
**Observe most attacks happen in the beginning of the typing stream – ie not:**
**./././././././././././././././.[.n.]/../tmp/attack**
**On the other hand, reading the output from a three hour make or >cat big-binary-file seem within the bounds of experience.  Plan defensively.**

## Passwords!

**The decision for a site to record password information can be a protracted mess. **
** **
**Simple test for passwords/phrases – matching on “pass(word|phrase)” tags the next user response line as a credential.  Optionally log it.**
**Logging is a configure time option and is *****opt-in*****.**
**This covers both login and in line activity so additional logins to other sites can also have that data recorded as well.  User education is important!**

## Decoupling the Write

**Failure of downstream networking or software can not effect the users ssh experience.  Our design will lose security data before slowing down our users.**
**Use of non-blocking write in sshd for sending data to a local stunnel socket.**
**Stunnel socket has aggressive timeout to avoid buffering issues on the sshd side.**

## PSC Modifications

**PSC has a set of patches for OpenSSH which provide for significant performance improvements in data transfer rate.**
**Benefit from performance gain greater than increased exposure to new security vulnerabilities. **
**More info: **
**http://www.psc.edu/networking/projects/hpn-ssh/**

## Stunnel

- SSHD

- Log
- Parser

- Bro

## Notes on Stunnel

**Generic configuration.**
**Data sent to logger on server end.**
**Initial implementation tolerates snakeoil certificates – only the central collector is required to be official.**
**Cross authentication is planned for near term.**

## Log Parser

- SSHD

- Stunnel

- Stunnel

- Log
- Parser

- Bro

## Log Parser Design

**The log parser is a libBroccoli based converter for translating well structured test into bro events which can be (optionally) transmitted across the network.**
**Changes made to process long strings containing spaces and odd characters.**

## General Idea

**Takes formatted text as input:**
**event_name count=0,5 addr=0,1.2.3.4 string=3,foo**
**Converts the data types: **
**Event = event_name;**
**int tmp_1 = atoi(“5”);**
**uint32 tmp_2 = (uint32)inet_addr(“1.2.3.4”);**
**bro_string_init(“foo”);**
**Then turns it into a bro event:**
**event event_name(count i, addr a, string s)**

## History: brosocket

**Mar 10 18:39:41 2.3.4.5 Message forwarded from host609: sshd[74334]: Accepted password for user1 from 1.2.3.4 port 45674 ssh2**

- (2) Parse into structured text

- (1) Start with Syslog message

- (3) **Bro Generic Client** reads text and communicates event to Bro

- (4) Bro acts on communication

**ssh_login double=1108696916 addr=1.2.3.4 addr=2.3.4.5 string=user1 string=password**

## Design Overview

- SSHD

- Stunnel

- Stunnel

- Log
- Parser

- Bro

## Bro Overview

**Continuing the tradition of separating agnostic events and larger policy.  All security related decision making is concentrated with the bro instance.**
**General design: strong emphasis on auditing and connection details.  **
**Once you get that right, you have the scaffolding in place to make security decisions.  Without, they are ad-hoc. **

## Full login example

**#511 ssh_connection_start 128.55.19.253:34141/tcp > 0.0.0.0:22/tcp**
**#511 auth_ok scottc publickey 128.55.19.253:34141/tcp > 0.0.0.0:22/tcp**
**#511 new_session SSH2**
**#511:0 new_channel_session pty-req**
**#511:0 new_channel_session shell**
**#511:0 data_server Last login: Fri May  2 13:35:50 2008 from red.nersc.gov PDT**
**#511:0 data_server FreeBSD 6.2-RELEASE-p7 (GENERIC) #1: Thu Aug  9 22:19:43 PDT **
**#511:0 data_server ****************************************************************
**#511:0 data_server                           NOTICE TO USERS**
**#511:0 data_client ls**
**#511:0 data_server ls**
**#511:0 data_server 1****	****    bro.tar**
**#511:0 data_server NODE           development**
**#511:0 data_server bin****	****    hot.bro**
**#511:0 data_server block-server   incidents**
**#511:0 data_client exit**
**#511:0 data_server exit**
**#511:0 data_server logout**
**#511 ssh_connection_end 128.55.19.253:34141/tcp > 0.0.0.0:22/tcp**
**Note: server data truncated during user notification and during ‘ls’ output.  The cutoff value is still being explored.**

## Event Breakdown

**For a remote execution we see:**
**ssh_connection_start**
**	****auth_ok**
**	****	****new_session**
**	****	****	****new_channel_session**
**	****	****	****ssh_remote_exec_pty**
**	****	****	****ssh_remote_do_exec**
**	****	****	****channel_exit**
**ssh_connection_end**

## Interactive Sessions

**For interactive sessions the data stream is monitored by several mechanisms.**
**Watch server response to clients typing to avoid translating “ka^h^hls” into “ls”.**
**Augment login.bro for a more contemporary hostile command list.**

## General Detection Schemes 

**Patterns/command for immediate notification. Pass the server_data stream through the login analyzer.**
**In addition, look for a series of suspicious command that by themselves are not clearly hostile, but in total represent a actionable event.**

## Future Modifications

**Bi-Directional authentication.**
**Continuing work on improving detection mechanisms.**
**Upgrade to the 5.x series of OpenSSH.**
**??? – we are just beginning to look at large scale production data and do not really know where than will lead us.**

## Questions?

**Contact:**
**Scott Campbell**
**NERSC/LBNL**
**scampbell@lbl.gov**

## Design Overview

- SSHD

- Stunnel

- Stunnel

- Log
- Parser

- Bro