**UVI data processing and data file maintenance (revised 05/15/07):**

**Every day:**

Check and delete mail on uvisun/campbrd and amos/campbrd. If Polar was
in science mode 1 during an nrt pass then the e-mail on uvisun will
contain the current instrument parameters. Note, as of March 6, 2007,
Polar should be in science mode 1 most if not all of the time. The
parameter listing includes the current values and the acceptable ranges.
If any of the values are out-of-range, contact Jim Spann or Glynn
Germany, see http://uvi.nsstc.nasa.gov/Contacts.htm.

Delete the uvinrt.log.mmddhhmm files in /data2/nrt. If the spacecraft is
in science mode 1, these files contain the times of the nrt images
created otherwise they just contain a header. The nrtmail.log file in
this directory contains instrument parameters at the beginning and end
of each nrt pass if Polar was in science mode 1. Periodically, this file
will need to be edited and the old information deleted.

Check to make sure that the daily images were created. The images are
created in /uvidata/yyyy/doy/gifs. They can be viewed from the UVI NSSTC
website http://science.nasa.gov/uvi/. Select \'Data Request\', then
\'Browse Images\', then yyyy, then doy.

**uvisun crontabs (as of 05/15/07):**

\#

\# Check every hour on the half hour to make sure uvinrt is running in
the

\# background.

30 \* \* \* \* /data2/nrt/checkuvinrt

\#

\# Copy UVI files from the staging server at GSFC

00 20 \* \* \* /export/home/campbrd/bin/ftp_uvi_data.pl -cron -year=2007
-limit=1

\# 10 00 \* \* \* /export/home/campbrd/bin/ftp_uvi_data.pl -cron
-year=2007 -limit=1

\#

#Copy nrt gifs to uvidata

0 17 \* \* \* /export/home/leebf/tcl/ftp_src/cp_nrt_gifs_to_uvidata.tcl
-cron

0 05 \* \* \* /export/home/leebf/tcl/ftp_src/cp_nrt_gifs_to_uvidata.tcl
-cron

\#

\# Process UVI PWG files every day

\# 0 01 \* \* \* /export/home/campbrd/bin/process_uvi

\# 0 02 \* \* \* /export/home/campbrd/bin/process_ptg

\# 0 03 \* \* \* /export/home/campbrd/bin/process_kp

\# 0 23 \* \* \* /export/home/campbrd/bin/process_nrt -cron

0 22 \* \* \* /export/home/campbrd/bin/process_uvi

0 23 \* \* \* /export/home/campbrd/bin/process_ptg

30 23 \* \* \* /export/home/campbrd/bin/process_kp

0 21 \* \* \* /export/home/campbrd/bin/process_nrt -cron

\#

\# Clean up the ftp/pub area every Sunday morning at 2 AM.

\# 0 02 \* \* 0 /export/home/campbrd/bin/clean_ftp

\#

\# backup mysql database

0 06 \* \* \* /export/home/campbrd/bin/csh_backup_UVI_mysql

**amos crontabs (as of 05/15/07):**

\# Crontab

\#

#Transfer any new uvi level-zero, scr level-zero, orbit, and attitude
files

#10 00 \* \* \* /export/home/campbrd/uvidb/ftp_uvi_data.pl -year=2007

20 00 \* \* \* /export/home/campbrd/uvidb/ftp_uvi_data_comp.pl
-uncompress=yes -year=2007

\#

#Update UVI monthly instrument parameter files

00 05 \* \* \* /export/home/campbrd/uvidb/csh_uvidb_txt

\#

\# Create UVI browse products every day at 04:00 AM

\# 00 04 \* \* \* /export/home/campbrd/bin/mkuvibp

00 01 \* \* \* /export/home/campbrd/bin/mkuvibp

\#

\# Create UVI kp mosaic plots everyday at 05:00 am

\# 00 05 \* \* \* /export/home/campbrd/kp_mosaic/process_kp.pl

\#

\# Get the platform slews list from the ISTP web site on Sundays at
10:00 PM

\# 00 22 \* \* 0 /export/home/campbrd/bin/get_platform_slews.pl

**At least once a week and as often as everyday:**

Create key parameter (kp) mosaic plots. A crontab (process_kp) on uvisun
should already have copied the new kp files to /kp/newkps. If not in
/export/home/campbrd, edit cp_kp_cdfs and copy any kp files that have
not been processed. If there are multiple versions of the files in
/kp/newkps, delete all but the most recent files. Then, on amos, in
/export/home/campbrd/kp_mosaic, create the key parameter (kp) mosaic
images and copy them to the web and the archive by running
process_kp.pl. If you have trouble running the script see obsolete below
for instructions on manually creating the files. The script processes
any kp files in /kp/newkps. Any files that did not process will be put
in newkps/bad and those that were processed will be put in
newkps/processed. Once the gifs are on the web and the archive, the gif
and log files in kp_mosaic/gifs and the kp files in newkps/processed can
be deleted. Note: I have tried to get process_kp.pl to run as a crontab
but I have had no luck. I'm not sure what the problem is. If the IDL
code doesn't run via process_kp.pl or manually (see below), then try
running an early version of IDL (see commented line in process_kp.pl).

**About once a week:**

Send the latest level-one files to gsfc. Before doing so, make sure that
only the most recent version is on the archive. Delete all the older
versions. The files are kept in the /uvidata/yyyy/doy/l1 directories.
There is a program in the bin subdirectory of the campbrd account on
uvisun called ftp_cdf_files.pl that automatically sends the files to the
NSSDC data ingest ftp site. There is a short shell script that runs the
program call run_cdf. The input to the program is the year of the files
to be sent, followed by the day-of-year range of the files to be sent.
Just edit run_cdf and change the doy range and the year if necessary and
run_cdf.

**About the 5th day of each month:**

NOTE: As of 05/08/07, Glynn Germany no longer needs these files. They
will continue to be created and stored on the archive but they do not
need to be sent to Glynn. \-\-- Send the instrument data file to Glynn
Germany and copy it to the archive using cp_instr_file in
/export/home/campbrd/uvidb on amos. The file is updated every day by a
crontab that runs on amos. The file is in /data6/UVI and is called
uvi_yymm.txt. gunzip it and then use the IDL procedure
read_uvi_instr_file.pro in /export/home/campbrd/uvidb to make sure that
every day of the month was processed. If all days are present, then ftp
it to a directory on the ftp.nsstc.org site. I use sloanma/outgoing/uvi.
gzip the file. Send Glynn Germany (germanyg@cspar.uah.edu) and Chakri
Deverapalli (deverapallic@cspar.uah.edu) a short email telling them
where it is and to use anonymous to login. This file is updated daily
via crontab jobs that run on amos.
/export/home/campbrd/uvidb/ftp_uvi_data.pl gets the most recent data and
/export/home/campbrd/uvidb/csh_uvidb_txt processes it. If a day is
missing, then it will need to be processed by hand. In
/export/home/campbrd/uvidb on the campbrd account on amos, run
./ftp_uvi_date.pl. It will prompt you for the date (yyyymmdd) of the
data that is missing. This program downloads the level-zero, spacecraft
housekeeping, orbit, and attitude files necessary to update the UVI
instrument file. Because of a network transmission problem, the data
transfer is very slow so it will take a long time for this program to
complete. If more than one day is missing, download all the days before
processing the data. The program puts the files in
/export/scratch/data/po. Once the download is complete then run
./csh_uvidb_txt to update the file. This program will delete the
downloaded files.

**At least once a month:**

Delete (rmdir) the empty directories in /data1/scratch.

Get rid of all the old versions of the files on /uvidata/yyyy.

Delete all the \*log files in /uvidata/yyyy/\* and in
/uvidata/yyyy/l1/\*.

Get rid of old files in /data1/ftp/pub/oa/pre.

Delete the nrt level-one files (po_nrt_uvi_yyyymmddhhmm.cdf) from the
/uvidata/yyyy/l1/.

Clean up nrtmail.log file in /data2/nrt on uvisun. This file contains a
housekeeping dump of the first and last record of each set of NRT data.
If the instrument is behaving correctly, you only need to keep about 10
days worth of information as a reference in case something does go
wrong. Delete the days at the top of the file.

**Once a month:**

Once a month when a new 70-day orbit/attitude and new orbit and attitude
daily files arrive (usually the morning after the third Wednesday of the
month):

UVI Team Telecon: Once a month, usually the first Monday of the month,
there is a UVI Team telecon. To make any changes to the schedule for the
first Monday of the month, call 1-877-232-6272. The confirmation number
is 4341820. There is a fine of some sort if a telecon is not held. Make
sure to get e-mail confirmation of any changes.

Change the passwords on the uvisun and amos campbrd accounts.

**Miscellaneous:**

**UVI Website:** The link for the NSSTC UVI home page is
http://uvi.nsstc.nasa.gov/. The home pages for the site are now on
xd12srv.nsstc.nasa.gov in the
/home/all-webusers/science.nasa.gov/htdocs/ssl/PAD/sppb/uvi directory.
The main page is default.htm. A lot of the data processing pages are on
uvisun in /web/UVI. If the navigation bar on the main pages has to be
modified, edit UVI-Template.html and the use updateuvi.pl to update all
the other web pages. If the mission is extended beyond 2007, the table
in <http://uvi.nsstc.nasa.gov/kpgs-Data.htm> will have to be edited to
add another year and monthly directories will have to be created in
/web/UVI/kp on uvisun.

**Redoing gifs:** If a new level-zero file comes done and the resulting
level-one file is a different size than the older versions, the gif
images will need to be recreated. First get into /uvidata/yyyy/doy and
delete the \*log files, then get into the gifs subdirectory and delete
all the gif images. Next, on the campbrd account on amos, cd to
/home/campbrd/bin. Edit giflist.txt and put in the yyyy/doy of the
images that need to be recreated. Then run ./run_gifs.

**Redoing orbit, attitude, l1 data:** All orbit, attitude, platform, key
parameter and level-zero files are available for downloading from the
pwgdata.gsfc.nasa.gov anonymous ftp site. The new files are in
pub/uncompressed, the old files are in pub/compressed. On uvisun,
download the files and put them in /archive/ftp/pub/ in the appropriate
subdirectories: level-zero in lz, key parameter cdfs in kp and orbit,
attitude, and platform files in oa/pre. To process the files (i.e. move
them to the archive), from /home/campbrd on uvisun run process_kp (key
parameter) and process_ptg (orbit, attitude, and platform files). To
create the level-one file, run process_uvi.

**nrtvu, l1vu - amos only (idl licenses):** There is only one IDL
license on uvisun and it is used continuously to monitor and create NRT
images, so no other graphics programs can be run from uvisun. The
graphics programs must be run from amos. nrtvu is a widget program used
to look at NRT data or level-zero data. l1vu reads and plots level-one
data. These programs will only be used is something seems to be going on
with the instrument. The nrt and the level-one files are in
/uvidata/yyyy/doy/l1/. The level-zero files will have to be downloaded
from the pwgdata.gsfc.nasa.gov. Put them in /export/scratch/sloan. To
use the programs, cd to the directory where the files are and simply
type l1vu or nrtvu.

**obsolete:**

Once a week:

Create key parameter (kp) mosaic plots. On uvisun, in
/export/home/campbrd, edit cp_kp_cdfs and copy the 8 most recent kp
files to /kp/newkps. This script is also available on amos in
/export/home/campbrd/kp_mosaic but it runs much faster from unvisun.
Move to /kp/newkps and delete all but the most recent versions of the kp
files. Once the files are on /kp/newkps, on amos, in
/export/home/campbrd/kp_mosaic, create the key parameter (kp) mosaic
images in IDL using \@kp_mosaic. It will process the kp files in
/kp/newkps. Any files that did not process will be put in newkps/bad and
those that were processed will be put in newkps/processed. Use cp_mosaic
in /export/home/campbrd/kp_mosaic to copy the gif images to the web site
and the archive. Edit cp_mosaic to set to correct directories on the web
and the archive. Watch out for month crossing. Once the gifs are on the
web and the archive, the gif files in kp_mosaic, the kp files in
newkps/processed, and the log files need to be deleted.

About the 5th day of each month**:** NOTE: As of 05/08/07, Glynn Germany
no longer needs these files. They will continue to be created and stored
on the archive but they do not need to be sent to Glynn. \-\-- Send the
instrument data file to Glynn Germany and copy it to the archive using
cp_instr_file in /export/home/campbrd/uvidb on amos. The file is updated
every day by a crontab that runs on amos. The file is in /data6/UVI and
is called uvi_yymm.txt. gunzip it and then use the IDL procedure
read_uvi_instr_file.pro in /export/home/campbrd/uvidb to make sure that
every day of the month was processed. If all days are present, then ftp
it to a directory on the ftp.nsstc.org site. I use sloanma/outgoing/uvi.
gzip the file. Send Glynn Germany (germanyg@cspar.uah.edu) and Chakri
Deverapalli (deverapallic@cspar.uah.edu) a short email telling them
where it is and to use anonymous to login. This file is updated daily
via crontab jobs that run on amos.
/export/home/campbrd/uvidb/ftp_uvi_data.pl gets the most recent data and
/export/home/campbrd/uvidb/csh_uvidb_txt processes it. If a day is
missing, then it will need to be processed by hand. In
/export/home/campbrd/uvidb on the campbrd account on amos, run
./ftp_uvi_date.pl. It will prompt you for the date (yyyymmdd) of the
data that is missing. This program downloads the level-zero, spacecraft
housekeeping, orbit, and attitude files necessary to update the UVI
instrument file. Because of a network transmission problem, the data
transfer is very slow so it will take a long time for this program to
complete. If more than one day is missing, download all the days before
processing the data. The program puts the files in
/export/scratch/data/po. Once the download is complete then run
./csh_uvidb_txt to update the file. This program will delete the
downloaded files.

Note: the nrt plots are no longer created but watch the directory just
incase generation starts up again. About once every 4-6 months have Bob
Dean delete all but about 20 days of data on /data1/ftp/pub/nrt. Data is
also on the archive (/uvidata).