# Presentation: 013092

## MCS Systems Group Update & Computing Environment Directions for (the rest of) 1999

- systems@mcs.anl.gov
- http://www.mcs.anl.gov/systems/
- 630-252-6813

- Rémy Evard

## Topics

-   Systems Group Issues
-   Upcoming Significant Changes
-   Computing Environment Update
- Purpose:
- Communication
- Discussion and feedback

## The Realities of Systems Administration

- Our primary function is to enable science at MCS.
- A few activities are fundable in and of themselves, but most aren’t.
- The job market out there is really great.
- The pile of work is infinite.
- Because there are so many “immediate” problems, it is nearly impossible to make forward progress.

- The users hate us.

- Happily - not true at MCS.

## Our Opinion: MCS is Great

- Cool toys.
- Plenty of challenges and new problems.
- We’re fairly happy with how the infrastructure is developing.
- The Systems Group is a good bunch of folks.
- MCS people are supportive, friendly, and fun to work with.

## The Focus Teeter-totter

- Support mode

- Project mode

- Fixing problems
- Emergencies
- Questions
- Restores
- .....

- Hardware installs
- Software installs
- New abilities
- New tools
- Testing
- ...

- 2 TONS

- Interrupt-driven:

- Concentration:

## The Req Flow - a simplified version...

- A user sends
- a req to “systems”.

- The help desk human
- either fixes it or assigns it
- to someone.

- The req owner tries
- to solve that problem.

- The owner notifies the
- user that the req has been
- solved.

## The Req Flow - identifying problems

- A user sends
- a req to “systems”.

- The help desk human
- either fixes it or assigns it
- to someone.

- The req owner tries
- to solve them problem.

- The owner notifies the
- user that the req has been
- solved.

- No owner assigned.

- Mail not sent to systems.

- User doesn’t hear back.

- Slow resolution of problem.

## Improving the Req Flow

- The monthly req buzz.
- Make sure someone owns every req:
- The help desk human owns all incoming reqs.
- Emphasize the need to respond to the user.
- Coming soon:
- Automatic response to let you know your req #.
- Coming later:
- Better support for users to check on req status.

## Req Statistics

- The following charts are data from the last 3+ years of reqs.
- Caveats:
- 1999 data is, of course, not complete.
- Reqs are often resolved “late” - weeks after the real problem is solved.
- Reqs generally don’t include help desk requests, which have been a serious factor since 1998.
- These do not include SP-related requests.
- None of these numbers indicate actual satisfaction with the resolution.

## Reqs Per Year

## Reqs Per Month

## Req Resolution Speed

## Req Resolution by Category

## Observations

- The number of reqs just keeps on growing.
- Peaks around summer and SCXX.
- Each year, the steady state is the level of SCXX the year before.
- 50% of the reqs are resolved in < 1 day.
- More reqs means more difficult reqs.

## External Activities

- NCSA PACS
- Scheduling & Resource Management
- Storage
- Accounting
- SP XXL
- Cray/SGI Users Group
- Some sysadmin interaction with:
- UofC
- LANL
- ...

- Networking
- Cyberbelt
- ESNet
- ...
- Usenix LISA NT
- 1998 Program Chair
- 1998 and 1999 Paper Acceptance

## Upcoming Big Events

- Eliminating clear-text passwords.
- Email system upgrades.
- y2k.

## Clear Text Password Elimination

- The problem:
- Motivation to fix:
- Reduce serious security threat to MCS systems.
- Compliance with our collaborators - PACI, NASA, ...

- login:
- password:

- evard
- 2+2!=5

- The computer asks:

- I type:

- 2  +  2  !  =  5

- A cracker
- can see:

## Securing Passwords - Options

- 1. SSH - Secure Shell
- A secure alternative to telnet and rlogin.
- Widely available.
- 2. Kerberos, DCE, DFS, ...
- 3. SecureID cards
- 4. Firewalls

## Securing Passwords - The Fix

- 1. Use “ssh” instead of rlogin and telnet:
- 		ssh fire.mcs.anl.gov
- All MCS systems have ssh installed.  Remote Unix systems are likely to.
- PC client available for your PCs: SecureCRT.
- Worst case, dial in to MCS modems.
- 2. Use “scp” instead of rcp.
- 3. Tunnel ftp and POP through ssh:
- 		ssh -L 1234:fire.mcs.anl.gov:21 ssh-server

## Securing Passwords - The Schedule

- Now:
- ssh available on all systems.
- PC client available.
- FAQ sheet coming out soon.
- You should:
- Switch to using ssh or SecureCRT
- Tunnel POP through SecureCRT.
- Bring up sshd on any  red MCS systems you run.
- Install ssh on your home unix system.

- June 1st:
- We will disable rlogin, telnet, and ftp for all MCS systems.
- Selected hosts will be enabled.
- Tell us if you want a host on the enabled list.
- October 1st:
- All hosts will have rlogin, telnet, and ftp disabled.
- Except the anonymous ftp server.

## Mail System - The Problem

- Our existing mail architecture can not scale much farther.
- The total number of email messages is increasing.
- Currently around 9000 messages / day.
- The number of email users is increasing.
- Individual email messages are getting larger - attachments.
- Simply installing a bigger, faster server doesn’t help.
- NFS file locking problems result in lost messages.
- The current ‘mbox’ format stores mail in a single file for each user.
- This guarantees a bottlenecked server, simply from “new mail” checks.

## Mail System - The Solution

- Install a new server with substantially more disk space.
- Dedicate it to nothing but email.
- Switch to a different internal mail format.
- Only support mail access using POP and IMAP.
- Mail client support:
- MH, Pine, Eudora, Outlook, ...
- Not ‘mail’.  Not ‘elm’.
- Biff support is a problem.

## Mail System - User Issues

- Not-so-good things:
- If you use ‘mail’ or ‘elm’, you will need to switch mail readers.
- Try out ‘pine’, which is a lot better than either of those.
- We need to find UNIX ‘biff’ solutions, and those will be different.

- Not-so-bad things:
- No real change for MH, Eudora, & Outlook users.
- IMAP will be supported.
- Good for those of you who read email on more than one PC.
- Substantially better support for humongous email messages.
- (Rick.)
- Virtual mail hosting.
- Centralized spam and virus control.

## Mail System - The Schedule

- Now:
- New server is on order.
- Switch to ‘pine’ if you’re using mail or elm.
- FAQ sheet being built to help users transition.
- May:
- Early testing for thrill-seekers.
- June:
- Testing for other folks; gradual migration
- September:
- Switch the real system over from antares to the new system.
- January:
- The old antares gets really confused.

## y2k - Threat or Menace?

- MCS does not have any operational responsibility, such as power or paychecks.  Nor do we run any ancient, custom-built software.
- This reduces our y2k problem enormously.
- Thus we have three classes of y2k problems:
- Dependence on others who have y2k problems.
- Dependence on vendors of our systems.
- MCS-written code.

## y2k Problem - Summary

- Physical / External problems
- The electronic door locks break.  These are on schedule to be fixed.
- Argonne facilities & systems.  Unknown.
- Illinois facilities.  Unknown.
- Vendor Systems problems
- Older PCs and older Sun4s will break.
- These will all be decommissioned this fall, even if they didn’t have y2k problems.
- Possibly other systems.
- We’re tracking fixes as they get posted.
- MCS Code
- Systems utilities (req, for example...)
- Check your own projects.
- ECT has a y2k test facility available.

## MCS Computing Environment

- UNIX
- Workstations & Servers

- Supercomputers
- & Storage

- Visualization

- Windows

- Network

- Special Projects

- Home Systems

## MCS Computing Environment

**MCS Computing Environment**

- UNIX
- Workstations & Servers

- 200 hosts
**SunOS, Solaris, IRIX, **
**AIX, Linux**

- Supercomputers

- 128 cpu Reality M
- 80+ node IBM SP2
- 64 node IBM SP1

- Visualization

- CAVE, Idesks,
- Active Mural

- Special Projects

- Linux clusters,
- lab systems

- Windows

- 110 NT Desktops
- 60 Laptops

- Network

- ATM backbone
- 10/100 ethernet
- OC-3/12 external

**... ****approximately 600 machines.**

- Home Systems

## Networking Update

- Wiring:
- Completed downstairs wiring. 
- Continued upstairs wiring. 
- Services:
- Multicast works to all new wiring. 
- DHCP is now a production service. 
- Have begun to upgrade wireless.
- Architecture:
- Merged the main network into one large, switched net. 
- Substantially more reliable. 
- External Link:
- Up to OC-12 (655mbits).

## Networking Update - Stats

- Host connectivity:
- 26 hosts on dedicated100baseT ports
- ?? hosts on 43 shared 100baseT ports
- ?? hosts on 94 shared 10baseT ports, which includes old scary thicknet
- 3 OC-12 hosts
- 34 OC-3 hosts
- Thus, the majority of hosts are still on 10mb ethernet, but this is steadily shifting to 100mb.
- For maps and stats:
- 		http://www.mcs.anl.gov/net/equipment.html

## Networking Update - Anchor

- The Argonne Network Anchor - a collaboration between MCS and ECT to leverage research and production networking in order to increase our external networking capability and opportunity.

- ECT

- ANL

- MCS

- World

- Anchor

- ECT

- MCS

- World

- ANL

- Before

- Now

## Networking Goals - 1999

- Complete building wiring project.
- Shift off of old, scary thicknet.
- 100baseT port increase.
- Complete the wireless upgrade.
- Continue to expand external networking.
- Fully implement Anchor and start to take advantage of it.
- Improved monitoring.

## MCS High Performance Computing

-  128 node SGI O2K
-  200 node PC cluster*

-   ADSM and HPSS* HSMs
-   MPI-IO/ADIO
-   Oracle Database Server
-   ATM, HIPPI, FC     and Gb-Ethernet

-   1 CAVE
-   4 ImmersaDesks
-   1 Display Wall*
-   Visualization Servers
-   Desktop Media Servers

**MCS High Performance Computing**

## SP - Update

- Tested the HPGN and wrote a paper. 
- Shifted to using the Maui Scheduler.
- Generally quite stable.
- Missed goals:
- Double the memory in all nodes.
- Go to 140 nodes.
- Install HPSS, GPFS, latest PSSP, etc.
- Production mode around March.
- Users Group Meeting.

## SP Usage By Type - 1998

## SP Percent Utilization

## SP Goals - 1999

- It’s really unclear - wait and see.
- At the moment:
- No OS upgrade.
- No memory upgrade.
- Continuing to work on HPSS installation.
- Expect the old SP to go away:
- Old, unsupported technology.
- Floor space issues.

## SGI Origin 2000, err, Reality Monster

- Update:
- 	It’s here!  It’s pretty cool!
- Plans:
- 	Scheduler installed any day now.
- 	Nail down policy.  But, certainly:
- More users will be added at that point.
- Users will generally be more closely affiliated with MCS than, say, the current users of the SP.

## Chiba City

- “Chiba City” is the name for the Big Linux Cluster project.
- Plans are fairly fluid at this point, but it’s a major focus of activity.

## Visualization - Update

- Viz Systems:
- CAVE going to 4 walls 
- New Idesk for Numerical Group 
- New Portadesk 
- Active Mural
- Computing Systems:
- Yukon, Tundra, & Denali arrived.
- Organization:
- Viz support work is split between the FL and Systems.
- Established the “vizcom” group.
- Realized that to be involved in the viz work in MCS, one needs to be participating in the research activities.  Policy issues related to this are still open.

## Unix Servers - Update

- Servers:
- New Solaris servers:
- fire, ember, soot
- New MOO server.
- Generally more stable.
- Critical Tools:
- “cfg” system in regular use.
- Missed goals:
- Many other critical tools need serious development work:
- account system, /software tools, .software upgrade
- Servers re-architected for security, reliability.

## Unix Servers - Goals 1999

- Servers rearchitected:
- Order of magnitude user space storage increase.
- SunOS gone from servers.
- Separate and rebuild functionality of:
- mail, DNS, NIS, central NFS
- Move servers to “hidden” network for security.
- More compute servers - Solaris, Linux, IRIX.
- Complete some of the critical tools.

## Unix Environment - Update

- New Platforms:
- Replaced FreeBSD with Linux. 
- Solaris 2.7, IRIX 6.5, linux 2.2, all quite useable. 
- New/Upgraded Tools:
- latex, perl, ssh, ...
- More to go.
- Issues:
- Solaris software installation is messy.  Different systems will have to have different versions of vendor software, like compilers.

## Storage - Update

- ADSM backup problems.
- NFS access to HSM disabled.
- ADSM backups stabilized.
- We moved all backups to ADSM.
- The big RAID fiasco.

## Storage - Goals

- Solve home directory space problems.
- Install HPSS.
- Upgrade ADSM hardware to improve response time for backups and restores.

## Web Servers - Update

- New servers and web pages up. 
- Help people migrate over. 
- Decommission the old web server. 
- Virtual hosting. 
- Statistics gathering. 
- Search capability. 
- Intranet. 
- Integrate with Visual Source Safe.

## Web Server - 1999 Plans

- Server upgrades.
- SSL capability.
- Address “authenticated” intranet need.
- Secure code distribution support.
- Respond to issues from the MCS community.

## NT - Update

- Create an actual server environment. 
- Organize \\dfs . 
- Create standard NT build and methods. 
- Wrote a paper on it.
- Install software remotely. 
- Punted.  All NT desktops are now yellow.
- Missed goals:
- Sort out software licensing.
- Try our hardest to integrate with Unix
- Start a “collaboration tools” test project.

## NT - Goals

- NT 2000, perhaps.
- Unix password and account synchronization.
- File crossmounting.
- Better centralized management.

## Desktop Update

- The “PC as desktop” strategy is working well.
- We will continue this plan, improving the systems here and there as costs drop.
- SunOS is still out there, but may have y2k issues.
- All old, old, old suns and 486s gone after summer.

## Laptops - The Issue

- Laptops are a challenge, because:
- For the system to be useful the owner must be the primary administrator.
- Laptop configs change constantly, because they frequently get new software & hardware, so they have frequent problems.
- No laptop OS is particularly reliable.
- Fixing a single problem can take hours or days.

## Laptops - Policy Clarification

- Whatever you run on it is up to you.
- Backups and restores are up to you.
- ADSM does work.  Slowly.
- If you want us to help you with it, we can:
- Install win95 or win98, depending on what the vendor ships for that system.
- Get the network working if it’s win95 or win98.
- Hack on ThinkPad hardware to debug the hardware if it’s running win95 or win98.
- Expect that if we get stuck, we will have to do a complete rebuild.
- We will offer friendly advice for other OSs if we happen to have some knowledge about the issue, but we can not help you work on them.
- We’ll be putting together NT and Linux “advice” FAQs.

## Home Computing - Update

- Using ISPs to get into the lab is a real option.
- Cable modems
- *DSL (iffy results)
- ISDN
- Be aware of security issues.  Use ssh, at the least.
- Goals:
- Improve MCS modem situation.
- Debate: Should MCS play ISP or not?

## User Support

- Progress:
- Continued / improved help desk
- (What should we do about summer students on help desk?)
- Lots of info on the web.
- http://www.mcs.anl.gov/faq/
- win-users mailing list a success
- Goals:
- Better organized web info.
- More in-depth talks.

## Summary

- We’re busy, but generally keeping up.
- Development projects are the places where we are weakest.
- Lots of things are going on.
- Tell us what’s important to you so that we can adjust plans and prioritize.
- Most of the time, this is a lot of fun.

## MCS Systems Group Update & Computing Environment Directions for (the rest of) 1999

- systems@mcs.anl.gov
- http://www.mcs.anl.gov/systems/
- 630-252-6813

- Rémy Evard