# Presentation: 677919

## Central Web Services at Fermilab 

- Presented
- by
- Jim Fromm
- October 27,2006

## Presentation Overview

- Why we have a central web farm
- Configuration of farm
- Hardware
- Software
- Automated tools for administration
- Monitoring of web farm
- Log processing
- Futures

## Why a Central Webserver Farm?

- Eliminate experiments from worrying about configuration/maintaining their own web server
- Keep on top of security issues
- Maintain up-to-date versions of apache/perl/python etc
- Maintain valid ssl certs
- Leverage our expertise to provide basic consulting for cgi scripts, web page development etc..    The web managers are responsible ultimately, but if they ask nice...
- Manage file system space (AFS has a quota mechanism, we keep track of when an area is getting full).

## Overview of Central Webserver Farm

- 5 Computers – 1 Load balancing switch.
- 84 vhosts (and rising fast)
- 53 Additional web areas (conferences, projects, computing...)
- 1300 Web Content Managers
- Web hits/year: > 220 million
- Staff:
- Basically 0.5 of one person right now.

## Hardware Configuration	

- Alteon

- www01

- www02

- www03

- www04
- (cgi)

- www05
- (cgi)

- AFS

## Hardware Configuration - Details

- Alteon Load Balancing Switch
- Configuration allows for traffic to be directed depending on type.
- CGI scripts only executable on www04/05.
- Alteon AD3 switch, 8x 100Base-T ports with a single GigE uplink
- Web Servers
- Sun Netra X1
- AFS file system

## Alteon Configuration

-  51: 131.225.70.46, enabled, name www05-vhosts, weight 1, timeout 10 mins, maxcon 200000
-     backup none, inter 2, retry 4, restr 8
-     remote disabled, proxy enabled, submac disabled
-     handle URL cookie: disabled
-     exclusionary string matching: disabled
-     1: any
-     2: /cgi-bin
-     7: .php
-     real ports:
-       http: vport http, group 11
-         HTTP Application: urlslb
-           virtual server:  4, 131.225.70.20,   enabled
-       http: vport 8875, group 11
-           virtual server:  4, 131.225.70.20,   enabled
-       http: vport http, group 11
-         HTTP Application: urlslb
-           virtual server:  5, 131.225.70.52,   enabled
-       https: vport https, group 12
-           virtual server:  4, 131.225.70.20,   enabled
-       4443: vport https, group 99, pbind sslid

## Software	

- Apache v1_3
- Mid-range plans to upgrade to v2_0 as security requires.
- Perl
- Python
- PHP
- Wiki support on Plone server (on separate set of servers outside of the webfarm)

## Automated Tools

- No way we can keep on top of everything the old fashioned way.
- Tools (perl scripts mostly) written to automate routine tasks
- Create vhosts
- Symlink check
- Password check
- File perms check
- File space check

## Automated Tools(cont)

- Symlink check
- Run 1x week
- Check for symlinks to areas that are sharing data that should not be shared
- Check for symlinks pointing to non-existent data
- Check for circular links
- Sends email to web admin team

## Automated Tools (cont)

- Permission check
- 1x per week
- Scan for vulnerable cgi scripts, weak  permissions on files and directories
- This can be any variation that leave security holes or fit the profile of known exploits:
- wide read-access permissions to area where passwords are stored
- write-access to cgi area
- wide read-access permissions to top level directories

## Automated Tools(cont)

- Passwords (1x week)
- Password files, although encrypted, should not be shared
- algorithm is not particularly smart: looks for variations on the word "PASSWORD" in file name and reports these if file permissions or locations are problematic

## Web Server Monitoring	

- NGOP (FNAL developed monitoring system) remotely monitors and alerts on the following:
- Ping of Alteon switch
- Ping of each web host machine
- Ping of each web server IP address
- Fetch pages for each web server and virtual host
- Fetching pages for commonly used services (telephone,stock) and checking correct results
- Verifying that the httpd for each server is running on at least one webserver host

## Web Server Monitoring (cont)

- Verifying that "fs examine" succeeds on the main web volume for each server (/afs/fnal/files/expwww/*), and any specific other volumes associated with it
- Watching each webserver's error log on each web host, and reporting important error messages (i.e. "out of memory")
- Generally an error will cut a helpdesk ticket, and page the primary.

## Log Processing	with Urchin

- Urchin v5
- This product provides accurate web site analytics which supplies the executives, marketers, webmasters, and the web designers at your firm with the vital up-to-date information they need to make informed business decisions.  Blah blah blah.
- Recently taken over by google.
- Mixed opinions about the product...

## Urchin (cont)

- Urchin likes to just stop processing without notification.
- Lots of pretty pictures and gizmos.  Overkill for what we need, but if you are running an e-commerce business....
- We bought Urchin to remove dependence on a home grown log processing package that was very difficult to maintain.
- Urchin is easier to maintain when it works.

## Security Requirements

- Run Nessus Scans 2x year.
- Nessus comes with a library of known exploits scanning profiles.
- Scan various newsgroups looking for announcements.
- Rely on our security team to alert us.
- Follow defined baseline of apache webservers developed at Fermilab.

## Future Plans	

- We are looking to upgrade to using the Cisco load balancer switch
- SunFire V240 servers
- Considered moving to Linux, but...
- Not confident of support model.  Great for general purpose farms, not as confident for server level service.
- Lot’s of work to convert.  Software installs, possibly broken cgi scripts etc...
- Apache 2.0
- Conscious of user communities in content management systems.  CMS are hot items.

## Thanks for your attention!

- That’s all
- One more thing before I go....

## GO CARDS!!!!