# Presentation: 277987

## Vulnerability Assessmentof Grid Software

**James A. Kupsch**
- Computer Sciences DepartmentUniversity of WisconsinFermi National Accelerator Laboratory
- March 20, 2007

## Security Problems Are Real

- Everyone with a computer knows this.
- 
- If you’re not seeing vulnerability reports and fixes for a piece of software, it doesn’t mean that it is secure. It probably means the opposite; they aren’t looking or aren’t telling.
- 
- The grid community has been largely lucky (security through obscurity).

## Many Avenues of Attack

- Attack web using
- www protocols

- Compromised host

- Internal bad guy

- We’re looking for attacks that exploit inherent weaknessin your system.

## Impact of Vulnerabilities

- FBI estimates computer security incidents cost U.S. businesses $67 billion in 2005 [_[CNETnews.com](http://www.cs.wisc.edu/condor)_]
- 
- Number of reported vulnerabilities each year is increasing [_[CERT stats](http://www.sdsc.edu/srb)_]

## Security Requires Independent Assessment

**Fact #1:**Software engineers have long known that testing groups must be independent of development groups
**Fact #2:**Designing for security and the use of secure practices and standards does not guarantee security
- Independent vulnerability assessment is crucial...
- ...but it’s usually not done ****

## Security Requires Independent Assessment (cont.)

- You can have the best design in the world, but can be foiled by ...
- Coding errors
- Interaction effects
- Social engineering
- Operational errors
- Configuration errors
- ...

## Project Goals

**Project Goals**

- Develop techniques, tools and procedures for vulnerability assessment focusing on Grid software
- Apply to production software
- Improve the security of this software
- Educate developers about best practices in coding and design for security
- Increase awareness in the grid and distributed systems community about the need for vulnerability assessment
- Train and build a community of security specialists

## Security Evaluation Process

- Architectural analysis
- Resource and privilege analysis
- Component analysis
- Codification of techniques anddissemination
- Overview
- Insider - full access to source, documents, developers
- Independent - no agenda, no blinders
- First principles - let the process guide what to examine

## Architectural Analysis

- Create a detailed big picture view of the system
- Document and diagram
- What executables exist and their function
- How users interact with them
- How executables interact with each other
- What privileges they have
- What resources they control and access
- Trust relationships

## Architectural Analysis (cont.)

- Created by looking at existing documentation, talking to developers, experimenting with the code and occasionally looking at the code
- Usually not well documented, or is spread across multiple documents or notes
- We’ve noted that developers often learn things when presented with just these diagrams

## Resource and Privilege Analysis

- Document and diagram
- Resources in the system such as files, users’ jobs, execution hosts, logs
- Operations allowed to be performed on a resource
- Privileges required to perform an operation on a resource

## Privileges - Root Install

## Privileges - Non-Root Install

## Component Analysis

**Audit the source code** of a component...... the audit is directed by earlier analyses
**Look for vulnerabilities** in a component

*“**A vulnerability is a defect or weakness in system security procedures, design, implementation, or internal controls that can be exercised and result in a security breach or violation of security policy.*”
- 	- Gary McGraw, *Software Security*

- i.e., **A bad thing**

## Categories of Vulnerabilities

**Design Flaws**
- Problems inherent in the design
- Hard to automate discovery
**Implementation Bugs**
- Improper use of the programming language, or of a library API
- Localized in the code
**Operational vulnerabilities**
- Configuration or environment
**Social Engineering**
- Valid users tricked into attacking

## Many Types of Vulnerabilities

- Buffer overflows
- Injection attacks
- Command injection(in a shell)
- Format string attacks(in printf/scanf)
- SQL injection
- Cross-site scripting or XSS(in HTML)
- Directory traversal
- Integer vulnerabilities

- Race conditions
- Not properly dropping privilege
- Insecure permissions
- Denial of service
- Information leaks
- Lack of integrity checks
- Lack of authentication
- Lack of authorization

## Buffer overflows

**Buffer overflows**

- Accessing locations of a buffer outside the limits of the buffer, commonly caused byC-style strings

**	****char buf[HUGE_SZ];**
**	****int isAdmin;**
**	****sprintf(buf, "****select * from T where id='****%s****'****", id);**
**If**** strlen(id) + 28 > HUGE_SZ**
**isAdmin ****overwritten****, changing control flow**
**Potential ****stack smashing**** (return address, frame pointer overwritten)**

## Injection Attacks

- A string constructed with user input, that is then interpreted by another function, where the string is not parsed as expected
- Common types include:
- Command injection (in a shell)
- Format string attacks (in printf/scanf)
- SQL injection
- Cross-site scripting or XSS (in HTML)

## SQL Injection

**SQL Injection**

- SQL injections occur when SQL statements are dynamically created using user input.

**  ****char buf[HUGE_SZ];**
**  ****sprintf(buf, "****select * from T where id='****%s****'****", id);**
**  ****h = ExecuteSQL(buf);**
**	**_**id**_**	**_**buf**_
**	****ab123****	****select * from T where id='****ab123****'**

**	****';****drop table T****--****	****select * from T where id='****';****drop table T****--****'**

## Directory Traversal

- When a user specified path, that is supposed to be restricted to a particular part of the file system, can be circumvented through the use of unexpected path components (empty, ‘.’, and ‘..’), or the use of existing symbolic links

**  ****const char safeDir = "****/var/appData****";**
**  ****sprintf(buf, "%s****/****%s", safeDir, userFile);**
**  ****h = ExecuteSQL(buf);**
**	**_**id**_**	**_**buf**_
**	****e.dat****	****/var/appData/****e.dat**

**	****../../etc/passwd****	****/var/appData/****../../etc/passwd**

## File System Race Condition

**f = fopen(fname, "w");**
**chmod(fname, 0700);**

**File System Race Condition**

- Multiple processes trying to perform non-atomic operations on a file system object.
- In the example below the program wants to write some data to a file so only it can read it.

**Attacker 1:**
**fopen(fname, "w")**
- Owns file: Can read, modify,
- delete or deny access

**Attacker 2:**
**symlink(fname, "/etc/passwd")**
**fopen**** **will now replace password
- file, if running as root

**Attacker 3:**
**fopen(fname, "r+")**
- Using **chmod** after an open allows
- a window of weak permissions

## Techniques Used to Find Vulnerabilities

- Look at suspect design flaws from architecture
- Examine and verify privilege changes
- Examine message protocols
- Verify correct authorizations for operations
- Problematic programming constructs
- APIs that are difficult to use correctly
- Arrays/strings for buffer overflows
- Look at bugs found during testing
- Source code analyzers

## Difficulties

- Need to trace function call graphs to trace data flows to determine potential values
- It is difficult in C++ to determine function call graphs using a textual analysis, due to the ambiguity of identifiers; the name alone is insufficient to determine the actual function
- The use of function pointers also complicates this analysis

## Code Browsing Tools

**Code Browsing Tools**

- cscope
- Doesn’t understand C++
- ctags
- Useful for finding definitions of global variables and functions, but not uses
- eclipse
- Doesn’t handle size of code and style well
- Hand written perl scripts to search code
- Useful, but really need to parse C/C++

## Static Code Analysis Tools

**Static Code Analysis Tools**

- Grammatech
- CodeSurfer: code browser, call chain analysis
- Path Inspector: queries over parse tree
- CodeSonar: looks for specific problems in code
- Flawfinder
- http://www.dwheeler.com/flawfinder
- RATS (Rough Auditing Tool for Security)
- http://www.securesoftware.com/rats
- ITS4
- http://www.citigal.com/its4

## Vulnerability Report

**Vulnerability Report**

- One report per vulnerability
- Provide enough information for developers to reproduce and suggest mitigations
- Written so that a few sections can be removed and the abstracted report is still useful to users without revealing too much information to easily create an attack.

## Condor Vulnerability Report

## Vulnerability Disclosure Process

- Disclose vulnerability reports to developers
- Allow developers to mitigate problems in a release
- Now here’s the really hard part:
- Publish abstract disclosures in cooperation with developers. When?
- Publish full disclosures in cooperation with developers. When?

## When a Vulnerability Is Found

**Don’t Panic!!!** Have a plan.
- Plan **what, how, when and to whom to announce**
- Plan **how to fix**, and **what versions**
- Separate security release or combine with other changes?
- When to release full details
- are details known or being exploitedexternally
- open/closed source projects
- allow time for users to upgrade

## Systems Investigated

- Univ. of Wisconsin’s **Condor Project**
- Batch queuing workload management system
- 600K lines of code, began 15 years ago
- _[http://www.cs.wisc.edu/condor](http://www.cs.wisc.edu/condor/security)_
- SDSC’s **Storage Resource Broker (SRB)**
- Distributed data store, with metadata and federation capability
- 275K lines of code, began 9 years ago
- http://www.sdsc.edu/srb
- NCSA’s **Myproxy** (just starting)

## Vanilla Universe Execution

**Shadow**

**OS Kernel**

**Vanilla Universe Execution**

**Execute Host**

**Starter**

**OS Kernel**

**User Job**

- 1. System Call

- 2. Return

## Standard Universe Execution

**Starter**

**OS Kernel**

**User Job**
**Condor C Library**

- 6. Return

- 1. System Call

**Standard Universe Execution**

- 2. RPC system call

- 5. RPC return

**Submit Host**

**Shadow**

**OS Kernel**

- 4. Return

- 3. System Call

## Checkpointing a Job

**User**

**startd**

**schedd**

**Shadow**

**Execute Host**

**startd**

**schedd**

**Checkpointing a Job**

- root

- condor

- user

**Real UIDs**

**Checkpoint Server A**

**ckpt_server**

- 1. ckpt signal

**Starter**

**User Job**
**libcondorsyscall.a**

- 2. checkpoint
- location query

- 3. checkpoint to
- Checkpoint Server A

- 4. Checkpoint File:
- Process Stack and Data Registers
- Shared Library Code and Data Mapped into address space
- State of all open files
- Signal handlers and pending signals

**Ckpt file**

## Generic Connection Brokering

**Private Network A**

**Public Network**

**Private Network B**

**Internal****Condor Host B**
- (ipX:portY)

**Internal****Condor Host A**
- (ipV:portW)

**Central Manager**
- (ipE:portF)

**GCB Server B**
- (ipB)

**GCB Server A**
- (ipA)

- Condor Traffic

- GCB-ModifiedCondor Traffic

- GCB Control Traffic

- 1. bind ipX:portYand maintain TCPconnection

- 2. bound ipB:portD

- 3. Job ClassAdipA:portC

- 4. connect to ipB:portD

- 5. use PASSIVE mode

- 6. CONTACT ipE:portF

- 7. initiate connection

- 8. notify of match with ipA:portC

- 3. Machine ClassAdipB:portD

- 9. connect to ipB:portD

- 10. use ACTIVE modeipB:portR1

- 10. CONTACT
- ipB:portR2

- 11. initiateconnection

- 11. initiate connection

- portR1

- portR2

**Generic Connection Brokering**

## Dynamic Port Forwarding

**DPF Server**

**NAT**

**Pass-Through**

**Private Network**

**Public Network**

**External****Condor Host**
- (ipC:portD)

**Internal****Condor Host**
- (ipX:portY)

**Central Manager**

- Condor Traffic

- DPF-ModifiedCondor Traffic

- DPF Control Traffic

- 1. bind ipX:portY

- 2. set ruleipA:portB -> ipX:portY

- 3. ok

- 4. bound ipA:portB

- 5. Machine ClassAdipA:portB

- 6. Job ClassAd

- 7. Notify of matchwith ipA:portB

- 7. Notify of matchwith ipC:portD

- 8. Claim resource andinitiate connection

**Dynamic Port Forwarding**

## Vulnerabilities Found

- 15 vulnerabilities in Condor documented
- 2 from Condor staff
- 1 in third-party library
- 6 vulnerabilities in SRB documented
- 1 from SRB staff
- Most of these have existed for years in shipping releases

## CONDOR-2005-0004 Vulnerability

- Condor runs as “root”
- It looks in a directory owned by the “condor” user for additional configuration
- Programs the Condor starts as root are controlled by this file
- This implies access to “condor” account gives “root” access
- Fortunately only the Condor daemons run as “condor,” so the risk should be low

## CONDOR-2005-0004 Details

- Condor has a feature to persistently change the configuration of a component at runtime, overriding main configuration files
- Persistent changes are stored in hidden local files owned by the condor user in the Condor log directory
- Most Condor daemons run as root, the Condor master uses this to determine daemons to start as root
**Insecure permissions** implies condor account is equivalent to root

**Submit Host**

**/etc/condor**

**/var/condor/log**

## CONDOR-2005-0003 Vulnerability

- Unfortunately, this vulnerability allows users to execute code as “condor”
- Email was sent by starting an external mail program as “condor”
- Due to the implementation, a carefully crafted email address could run arbitrary programs as “condor”
- These two allowed a “root” compromise of most machines in the Condor pool

## CONDOR-2005-0003 Details

- User supplied data (notify_user) is used without validation or proper quoting in **popen** resulting in a **command injection**
- Arbitrary code can then be run with the “condor” user’s privileges
- Condor administrative data and log files are owned by this user

**Submit Host**

**submit**

**schedd**

**shadow**

**Submit file:**
- ...
- notify_user = u@bad.com;/home/bad/evilProg

- /* remote job is done, send email */
- sprintf(cmd, “%s -s \“%s\” %s”, mailer, sub, email)
- AssumeCondorUserId();
- popen(cmd, “w”)

- /bin/sh -c ‘/bin/mail -s “...” u@bad.com;/home/bad/evilProg’

## Sample SRB Vulnerabilities

- Files are normally placed in the SRB’s logical namespace by uploading a file to the server
- It also allows existing physical files on an SRB server to registered to a logical name
- SRB-2006-0001 - **improper authorization**
- Ordinary users could register any file accessible to the “srb” user
- 	Sregister /etc/passwd pwScat pw	#  view /etc/passwd
- SRB-2006-0002 - **directory traversal**
- Ordinary users could circumvent SRB authorization system by using non-canonical names for files
- 	Sregister /Vault/./rsrc1/privUser.domain/secret notAnyMoreScat notAnyMore	#  view the fileSappend badData notAnyMore	#  change the fileSrm notAnyMore	#  delete the file

## Summary of Project Results

- Develop local expertise in vulnerability assessment
- Develop assessment procedures
- Produce architecture, resource and privilege documentation
- Finding real vulnerabilities and producing vulnerability reports and disclosures.
- Codifying methodology and techniques
- Disseminating methodology and techniques to others
- Train developers to prevent vulnerabilities

- For more information about Condor vulnerabilities see:
- http://www.cs.wisc.edu/condor/security

**Questions**