# Presentation: 158495

## Cyber Security at Bookhaven National Lab

**Martin Purschke, Brookhaven National Lab**

**BNL from space**

**Long Island, NY**

**Long Island, NY**

**Long Island, NY**

**The Relativistic **
**Heavy Ion Collider**
** ****RHIC**

## Cyber Security 

**Tom Schlagel, the head of the “Information Technology Dept” ITD was supposed to stand here**
**We have a Cyber Security audit going on this week (maybe it would be better to list the times when we do not have an audit going on), Tom asked if I could stand in for him**
**In a way, we are supposedly on different sides of the fence, since...**
**I am the  Chairman of BNL's Cyber Security Advisory Council “CSAC”, which represents the interests of the scientists and projects (as my day job, I work at the PHENIX experiment at RHIC)****‏**
**So maybe this will be with a slightly different perspective than Tom would have had...**

## CSAC 

**CSAC has about 25 members representing the various departments and larger groups**
**We have, by our charter, a proactive role**
**We review the key policies and decisions concerning computing infastructure and CS and evaluate their impact**
**Suggest changes, modifications, exceptions, etc**
**Has been working nicely so far (I guess ITD sees us as partners, else I would probably not be standing in today)****‏**

***“******The Cyber Security Advisory Council (CSAC) participates in formulating, establishing and reviewing BNL's security policies, plans and strategy, and advises the Information Technology Division Manager on security and other issues.“***

## Some of the Issues

**Clash of perceptions at DOE and locally-- no, we don't have people doing spreadsheets in front of their desktop windows machines (as a matter of fact, that OS is the minority)****‏**
**A general risk-aversity even for minor things – some people like the devil's advocate game**
**Local compliance with policies, enforcements (in the hallways you hear “they rendered my wonderful fast Dell essentially useless”)****‏**
**Reporting and remote management tools, clashes with online/analysis systems**
**SUI and PII, laptop encryption**
**Remote access, crypto cards, ssh, VPN, etc**
**Foreign National Access, local and remote**
**Wireless access, access for short-term visitors (meetings, conferences, guests)****‏**
**Blocking of ports**
**Blocking of machines from the network, false positives**
**...**

## General

**I was asked to speak  (and know most, obviously) about the things happening in Brookhaven, but I tried to pick the items which are of general interest beyond BNL**
**I guess you'll recognize some of the issues at home**
**I'm going to run through some facts and try to highlight what the stakes are for the UECs and NUFO**

## General Network Topology outline

**General Network Topology outline**

- RHIC & ATLAS

- General Campus and “Protected Core”

## Conceptual View of the BNL Network

**Stole this concept from proper ship design – copmpartments with hatches in beween, flood one and the damage is confined to that compartment**
**Access between those “enclaves” is generally on the same level as remote access (with some exceptions) **

**“****Enclaves”**

**(they are be a bit more fine-grained than shown here)****‏**

## Better Safety Posture

**This is a nice change from “hard on the outside, soft on the inside”**
**You put challenges/firewalls and hurdles between units which don't have to communicate routinely **
**Compromise one, you still haven't compromised the others**
**The sentry points can hold IDSes **
**Each enclave has some say on what the firewall rules are, within reason**
**And tighten down as needed (payroll, HR, administrative, research ,etc)****‏**
**A positive step away from “one size fits all”**

**We have been doing pretty good on containing intruders/viruses/ and yes, auditors just with this setup, but it takes a lot of explaining of such security-enhancing concepts to receive proper credit**

## “Know what's on your Network”

**There's some desire to know what's connected to the network**
**That makes sense up to a point – if you let guests in your home use your Internet connection, you know who they are and know that you can trust them**
**But this can also go overboard -- in experiments, things change way faster than you have a chance to document**
**Virtually everything is a “computer” these days - your scope, your Logic Analyzer, and virtually everything is configured and accessed by way of a web browser  **

**Many of the requirements that we get handed down are geared towards an environment with mostly Windows Desktops.**
**Limited applicability in the Lab**
**We struggle with this general attitude a lot.  **

## Translation into Security Policies

**There are some absolute rules that everone has to follow**
**Then there are many that you need to follow unless stated otherwise**
**If you cannot follow a rule (example: locking screensavers), there are 2 options:**

**Ask for a variance. This is **
**done on a case-by-case **
**basis for individual or few **
**machines**

**Write an Enclave Document **
**that describes how you do **
**business within your (usually **
**small) enclave**

**In both cases you have to implement and document some “mitigating controls” that minimize the risk from not complying with all the rules**

**This is far away from a well-established, smooth process yet, and we have had this for a few months only, but this is getting in the way, a lot**

## Example: My own shop, PHENIX

**The PHENIX experiment's couting house is a mid-sized computing center by some standards**
**About 580 machines in various shapes and forms, mostly rack-mounted machines from the event builder, a farm for online monitoring and processing, 2 dozen desktops, some VME processors, etc**
**Some standard rules that we can't follow and which are covered in my enclave document:**
** ****No locking screen savers by default**
** ****Group accounts (can't run the DAQ from indiviuals' accounts)****‏**
** ****A localized internal wireless network**
** ****Patching on a best-effort / best moment basis**
** ****Standard quarterly scans w/o exemptions but on my schedule**

**The cost of doing that is a firewall that shields us from most evil things**
**Some increased vigilance in monitoring bad stuff **

## Along the same lines: non-BNL computers

**Following rules for BNL-owned and inventarized machines is... not easy but manageable**
**Machines belonging to external groups are harder**
**We have short-term transients (example: Light Source, group comes, sets up shop for one day to a week or so, leaves)****‏**
**Others are non-BNL-owned but spend their entire life cycle at BNL**
**The short-term ones are hardest, limited setup time, beam is holy, get in and get out**
**Also, those have to comply with sometimes conflicting requirements from owner and BNL**
**No final solution yet, have talked about an enclave with just enough connectivity to do business**
**But in general, externally-owned machines have to comply if connected internally**

## Externally and internally connected

**Throughout the lab, there is a visitor wireless network that connects you outside the core BNL network**
**It's like logging in from home or somewhere else off-site**
**On the visitor wireless you register, but that's all that's required, mostly**
**All machines must be registered with an identifyable owner or admin**
**That satisfies the “know what's on your network” rules **
**Internal Windows machines generally join the BNL domain and run the SMS reporting service (ITD learns what your pacth levels are etc)****‏**
**Unix-like machines (Linux and Macs, etc) run a homegrown utility, ORDO, that collects some data and sends it to a central machine**
**ORDO has gained a lot of trust since it's a perl script that you can look at, also you can see what's being sent**

**I personally find myself almost permanently externally connected with my laptop and don't find it so restrictive... but perceptions vary**

## Quarterly and Continuous Scans

**Most machines are subject to a quarterly full Nessus scan and less-intrusive continuous scanning (some notorious port ranges)****‏**
**Owners, where identifyable, get mail about issues that come up**
**Findings have to get cleared out one way or another, fix it, document as false  positive, operational need, etc**
**This is a very useful tool for admins of large setups **

## SSH and VPN gateways

**All interactive remote access goes via a relatively small number  of gateways. From there one can hop to the final destination**
**Most RHIC folks that I  deal with are relatively computer-savvy and know how to use ssh tunnels – this has been a relatively small inconvenience**
**Access between enclaves typically goes via the same gateways in order to traverse the firewalls**
**The gateways plug into the user registration database and verify a user's proper status as guest or employee, else denies access**
**That implements the DOE mandate that only authorized users can access BNL's infrastructure**

**Some definition problems what constitutes “access”**
**e.g. you change your job, are no longer a guest, what happens to your email and email address? **

## Intrusion Detection

**A pretty sophisticated system that watches for suspicious traffic leaving or coming into  BNL (bittorrent movie downloads, Napster-like file sharing, virus signatures, DOS attacks, etc)****‏**
**Works nicely – I got a rack of pre-infected machines from a  vendor once, they were caught within minutes**
**Can pinpoint violations of the user agreement, actions taken when excessive**
**The biggest threat at this point is coming from inside from internally connected machines that get compromised elsewhere, Laptops, machines visiting malicious web sites, etc**
**The compartmentalization helps a lot here**

**Some fine line what is “appropriate use” **
**had some clear and systematic violations that lead to terminations**
**but it's something to watch out for, you don't want to get into the role of school libraries forced to block content**

## Remote Users/Foreign Nationals

**BNL is  a Tier-2 Lab in between Labs like LBNL, FNAL etc on one hand and LLNL, LANL, etc on the other (some non-proliferation work at BNL bumps us up to 2)****‏**
**A bit more involved to get access especially for FN's**
**Once established with a guest appointment, computer access is easy**
**The hurdles are in getting that done in a predictable and streamlined way**
**Smooth for most, but we have seen some kerfuffles now and then**

## We are truly international...

**Huge variety on citizenship  -- **

**Some nationalities are routine, others take a lot longer, we had a Cuban who gave up, eventually**

**Have to work against an attitude that “they are there to steal your data and knowledge” within DOE**
**From a (now-pulled) “Foreign National Training” course: “1 out of 3 Foreign Nationals is a professional spy”**

## How well does all this work, really?

**As far as technical things go, pretty well, actually**
**Always some issues, but by and large, we get by**
**There will always be the “I cannot print today”-style problems, etc, usually resolved quickly**

**As human interactions go, well, it's a different story**
**Ok, we have come past the “us vs them” mentality**
**Still, we see what I call “the weekly assault on the cyber security status quo”**
**People who never liked the way Science does business (open, international, few controls) feel wind under their wings, it seems**
**Some processes are simply broken. **

## So who is Boss?

**Groups talking with too many people with similar job descriptions, it seems**
**Too many overlapping audits, can we just get one comprehensive one and be done with it? **
**Deep culture clash of auditors with BNL population (“We could sit connected to the wireless network here for hours, unchallenged!!!” -- “Great, for once it worked!”)****‏**
**Some autitors shocked that they could roam unescorted, etc**
**At times little understanding of what the program is – the GRID paradigm is a particular mystery to most**
**Some real PR nightmares lately, of all things, with BNL's UEC**
**However, many processes are new and I (wearing my CSAC hat) sense a genuine willingness to straighten things out upstairs, has not always percolated to the people in the trenches yet**

## What we have been busy with lately

**Out of the blue came a steady stream of “Technical Management Requirements” that attempt to prescribe te way we do business**
**Bypasses usual procedures, DOE expected comments on a very short time scale**
**One size-fits-all for all labs **

**Our hopes that they would simply go away after the scathing comments by the Labs did not come true – a new volley of revised TMR's were just sent out yesterday (Monday)****‏**

## TMR potential Impact

**Some TMRs are just technical requirements that define and clarify things and have neglibgible impact on the normal Lab population (e.g. TMR17 - ”INFOCON” defines some DHS-style green/yellow/range threat levels and conditions)****‏**
**Others require more or less benign changes**
**Others would have severe impact on how we do business and our mission, and we need to make our point of view heard (TMR13 - “Portable Devices”, TMR18 - “P2P networking”, TMR19- ”Remote access”) which might affect things like bittorrent downloads, skype, and many other things**
**(Most of them are written with a Windows-only outlook and rarely match the world as it is, we need to comment because it's hard to see how to apply those)****‏**
**Then there are a few that would jeopardize our mission, mostly, but not only, the ones that deal with foreign nationals and seem inspired by a cold-war era view of the world **

## TMR Poison Pills (1)‏

**TMR0 says:**** **_**Users.**_
**Privileged Users.**
**Privileged User Requirements**
**Be U.S. citizens, unless otherwise approved in writing by the cognizant DAA.**
**(DAA is the Designated Approving Authority, someone from BNL's DOE Office)****‏**

**Comment – Requirement 2.j.(1)(a) that Privileged Users be U.S. citizens, unless approved in writing by the DAA will have a large impact on the BNL science mission.  The definition of “Privileged User” is broad enough that it applies to anything from a desktop computer to a domain controller, firewall or router.  There are different levels of risk associated with these types of devices, and therefore there is a different level of risk associated with the administrator of the device.  This requirement would restrict non-U.S. citizens from being the local administrator of their own laptop.  For example, BNL has as many as 400 non-U.S. citizens that could be considered privileged users under this definition.**

## TMR Poison Pills (2)‏

**TMR16 – “Foreign National Access” says:**** **

**Foreign National Access to information systems is granted only after approval as required in DOE O 142.1 and/or DOE O 142.3. **
**Access to Sensitive Unclassified Information (SUI) systems by Foreign Nationals must include:**
**(1)General users must be screened to include a Human Resources Background Check and a National Agency Check.**
**(2)Privileged users must be screened to include a Human Resources Background Check and a National Agency Check with Inquiries.**
**(3)Administrators must be screened to include a Human Resources Background Check and a National Agency Check with Law and Credit.**

**“****Prohibit Foreign National privileged users and administrators from accessing any system from other than a Senior DOE Management operating unit facility. “**

**Huge administrative burden**
**FN Sysadmins in research out of their job**** **

## Others we watch out for

**P2P, especially skype and bittorrent etc endangered by “P2P” TMR18 **
**Portable devices “are presumed to contain“ personally identifyable Information, PII, apparently DOE's nightmare scenario after the VA stolen Laptop -- addressed in TMR22 “Sensitive Unclassified Information” **
**TMR19 “Remote Access”**** **

## Related Items

**We established  a desktop VTC working group**
**find a free, casual use, low-impact desktop videoconferencing solution that's below the “footprint” and threshold of VRVS and similar**
**have something ready should skype get outlawed at least for part of the house**
**report due in ~3 weeks**

## Sign Off

**Lots of progress, but also lots of things still to be improved**
**Have to counteract the recent trend to button down and close up the Labs**
**Fortunately, the Office of Science seems firmly on our side**
**Political high-level action might help generically – help reinforcing positive attitudes towards open science and the facilities as a resource open to all**
**I personally find that patient and polite explanations help, auditors are ready to sign off on pragmatic solutions that work**
**Need to watch out for subtle changes in policies or their interpretation – make a stand early because it's hard to undo later**
**At least at BNL, a very open dialog between UEC, ITD, and CSAC**