# Presentation: 483948

## Site Visit Report

- Gemini ObservatoryHilo, HawaiiSept. 20, 2005

- Erik Gottschalk

- LHC@FNAL

## Outline:

- Introductory information
- Physical Layout
- Hardware and Software
- Networking and Security
- Cost to build and for administration
- Anecdotal information
- Comments

- LHC@FNAL

## Introductory Information

- Overview/purpose of project/control room:
- Remote operation of the Gemini Observatory and communication with the southern site in Chile.
- Date and time of visit:
- Sept. 20, 20058:30 am – 4:30 pm
- Names of visitors and contact information of host(s):
- Visitor(s): Erik GottschalkDr. James R. Kennedy, Associate Director of Operations670 N. A’ohoku PlaceHilo, Hawaii, 96720_[jkennedy@gemini.edu](mailto:jkennedy@gemini.edu)_     (808) 974-2515
- List of activities during the visit:
- Tour of the Hilo operations center, conference room, Dr. Kennedy’s office, view of Mauna Kea from the airplane

- LHC@FNAL

## Introductory Information (cont.)

- Two observatories with independent operation
- Gemini North – Mauna Kea (1st light in 2000)
- Gemini South – Chile (1st light in 2001)
- 8.1m primary mirror (F1.8)
- 380 tons
- Adaptive optics with laser “guidestar”
- Two people at summit when “running” but they are not expected to make critical decisions at any time due to “clouded thinking” at high altitude
- Gemini observatory operates as a national lab, with experimenters submitting proposals, web-based tool, simulation, approval process
- Gemini is transitioning from “classical observing” to “queue observing” (50/50 to 70/30, and then 100/0). Operations model is to alternate between the two modes, and between the two sites to optimize personnel.

- LHC@FNAL

## Physical Layout

- Sketch of the layout in the Control Room: see photo
- Describe the lighting: fluorescent, overhead
- Describe the flooring: carpeting
- What is done to control the Noise? not necessary
- Describe the console(s): modeled to look like NASA experiment control center
- Take photos of the operations center: see photos
- Describe the screens (size, position, number): see photos

- LHC@FNAL

## Sketch of control room

- LHC@FNAL

- LHC@FNAL

- LHC@FNAL

- LHC@FNAL

- LHC@FNAL

## Hardware and Software

- Describe the PC’s:
- Sun, Solaris, Xwindows based consoles
- Describe the monitors: CRT
- Describe the communications equipment
- PictureTel (ancient) to communicate with summit
- Polycom for 3-way communication with the summit at 14,000 feet, base camp at 9,000 feet, and Hilo control room (24/7 operation with daily meeting at 16:00)
- Polycom also used to communicate with teams of principal investigators who wish to be “intellectual participants” during the period in which their data is being collected
- “Remote viewing” is allowed. “Remote observing” is not allowed, since this implies **control** of the telescope which is never allowed.
- Note: very interesting discussion about risk analysis for Hubble operation involving Hubble/Goddard/STScI...
- No headsets (like at NASA)
- Describe the application software
- Display sharing using a homegrown Xwindows tool
- Permit access to through firewall by “opening” ports.

- LHC@FNAL

## Networking and Security

- What is the network-link speed?
- Currently 45 Mbps, will be upgraded to 100 Mbps ethernet in the next few weeks
- Is there a dedicated link?
- Yes, between base and summit, see photo
- How many users are connected?
- ~500 network connected devices (each, Hawaii & Chile)
- 85 staff at Hilo, 70 in Chile, + 10-15 visitors/postdocs
- What security is being used?
- Dedicated circuit between summit and base
- Cisco firewalls to outside world
- Hardware VPN links Gemini Hawaii and Gemini Chile LANs (sounds like cern.ch at FNAL)
- Software VPN to permit connections from home/travel
- The capability to give someone complete control of the telescope exists, but is RARELY used, and only for a limited (short!) period of time.

- LHC@FNAL

- LHC@FNAL

## Cost to build and administer

- What is the administrative load?
- 7 people for systems administration at Hilo
- 4 people in Chile
- Unix, Linux, Windows, Mac, network administration
- Most of the work is Unix administration, but not dedicated to the support of remote operations
- What is the cost?
- ~ $50K for console furniture, each room (see photo)
- 		Each of four locations (summit/base, Hilo & Chile)
- Have their been many upgrades?
- Progressive software improvements
- Future hardware upgrade to Linux is expected
- How long did it take to build?
- Concurrent with and paced by telescope over several months, but would be much faster today.

- LHC@FNAL

## Anecdotal Information

- What worked well?
- Technical implementation was well conceived, such as networking and software.
- Initially PictureTel video conferencing was used over an ATM circuit, but this has now been replaced with IP conferencing over Internet-2.
- What did not work?
- The vision was for more people to work in the Hilo operations center, but with the transition to queue observing (from classical observing) fewer people are needed in the Hilo operations center.
- What would you do differently?
- Having furniture fabricated to look like a NASA experiment control room was unnecessary.
- At a cost of ~ $50 per system, it is unnecessarily expensive.
- Clunky and hard to service equipment due to cabinetry.
- Would use flat-panel displays today, as is now done at the summit.
- Do you have plans to upgrade?
- Solaris systems will be replaced with Linux (upgrade takes time).
- Linux platforms are fast and cheap compared to Solaris;  give flexibility on hardware choices for the summit. Some hardware (CPUs, displays, disk drives) aren’t reliable at 14,000 feet.
- Mac OS X is popular with science users, but will not be used for the control system.

- LHC@FNAL

## Comments

- 8 hours of discussion was VERY helpful (and generous!!), but this is not likely to be the norm for other site visits.
- The visit to the Hubble control room should be very informative. Goddard runs the Hubble and accepts full responsibility when something goes wrong. Although the Space Telescope Science Institute (STScI), which accepts research proposals, has been working with Goddard for over 30 years, they DO NOT control the Hubble. This could serve as an operations model for us where Hubble/Goddard/STScI can be viewed as LHC/CCC/LHC@FNAL. Dr. Kennedy may be able to set up a meeting for us with someone at Goddard during the Hubble control room site visit.
- Remote operations is an excellent way of leveraging intellectual resources, ie. “intellectual participation.”
- E-mail is a dangerous form of communication to rely upon for operations, telephone is better, video conferencing is preferable (90% of information is visual). Dr. Kennedy is a firm believer in video conferencing as a tool for effective communication.
- For a staff of 85 people, they have ~10 Polycom Viewstations (+2 spares), and another ~10 Polycom ViaVideos.
- They have another 3 Viewstations at the summit, and 4 conference rooms in Hilo (including the operations center).

- LHC@FNAL

- LHC@FNAL

- LHC@FNAL

- LHC@FNAL

## More Comments

- Moving a video conferencing camera around remotely (for example, in Chile) provided a sense of presence. This feature CAN be disabled, if not desired.
- For Gemini they use this to see if someone is in a room to talk to, as opposed to shouting into the room and hoping that someone responds.
- Communication improves when you know the person you are talking to.
- To improve communication between Mauna Kea and the Chilean site, Gemini does periodic staff exchanges, from secretaries to scientists to foster relationships.  For example they exchange secretaries for 1 – 2 weeks every couple of years; it helps people at each site learn about the other site. “Oh, that is what you mean when you ...”
- Developing an operations model for LHC@FNAL is probably the best way for us to determine what role LHC@FNAL has for LHC and CMS. It can help us figure out how to run shifts for LHC@FNAL (for example), and will influence our requirements.

- LHC@FNAL

## Slide 21

- LHC@FNAL

- LHC@FNAL