PDF Analysis System

Stand-Alone Version

Table of Contents

[PDF Analysis System -- Stand Alone Installation
1](#pdf-analysis-system-stand-alone-installation)

[Introduction 1](#introduction)

[Quick Install 1](#quick-install)

[Audience 1](#audience)

[Scope 1](#scope)

[Acknowledgments 1](#acknowledgments)

[Authorship 1](#authorship)

[References 2](#references)

[PDF Analysis System Overview 3](#pdf-analysis-system-overview)

[Description 3](#description)

[Requirements 3](#requirements)

[Hardware 3](#hardware)

[Software 4](#software)

[PDF Source Code Organization 4](#pdf-source-code-organization)

[Input and Output Files 5](#input-and-output-files)

[Input 5](#input)

[Output 6](#output)

[System Configuration 7](#system-configuration)

[Data Directories 7](#data-directories)

[MiniSeed Data Directory and Files
7](#miniseed-data-directory-and-files)

[Response File Directory 8](#response-file-directory)

[Web Directory 8](#web-directory)

[Script Variables 9](#script-variables)

[Source Code Header File Modification
9](#source-code-header-file-modification)

[System Compilation and Installation
10](#system-compilation-and-installation)

[Compilation 10](#compilation)

[Compiler Options 10](#compiler-options)

[Installation 11](#installation)

[PDF System Execution 12](#pdf-system-execution)

[Channel-Specific Shell Script 12](#channel-specific-shell-script)

[Overall Execution 12](#overall-execution)

[Execution Features 12](#execution-features)

[Error Processing 13](#error-processing)

[Release Notes 14](#release-notes)

[Current Limitations 14](#current-limitations)

[Version Control 14](#version-control)

[Version 1.1 14](#version-1.1)

[Appdendix I -- File Formats 15](#appdendix-i-file-formats)

[Djjj.bin File 15](#djjj.bin-file)

[Hour.idx File 15](#hour.idx-file)

[Hjjj.bin File 15](#hjjj.bin-file)

[PDFAnalysis.bin File 15](#pdfanalysis.bin-file)

[PDFAnalysis.inf File 16](#pdfanalysis.inf-file)

[PDFanalysis.sts File 16](#pdfanalysis.sts-file)

[PDFAnalysisSR.bin File 17](#pdfanalysissr.bin-file)

[Appendix II -- Installation and Configuration Checklist
18](#appendix-ii-installation-and-configuration-checklist)

# PDF Analysis System -- Stand Alone Installation

## Introduction

Following from the successful implementation of the NEIC PDF Analysis
System, both at the IRIS DMC in Seattle, WA, and at the NEIC in Golden,
CO, it is desired to provide the system to the seismic community at
large in a stand-alone form. This, to allow other users of the seismic
community the opportunity to perform their own analyses against data
sets not being held by either of these two data centers.

## Quick Install

If the time required to read and understand this document in totality is
unavailable (but you'd like it installed and working as quickly as
possible), you must at least read the sections **System Configuration**,
**System Compilation and Installation**, and **PDF System Execution**
and follow the steps outlined there. Where problems occur, be certain to
consult the other various sections of this document that may provide
further information to solve your issue.

## Audience

This document is intended for all users of the siesmic community with an
interest in producing noise analyses of seismic data following the
algorithm laid out by Buland and McNamara of the USGS NEIC of Golden,
Co.

## Scope

This document is limited to the technical aspects of the **PDF Analysis
System**: installation, configuration and execution. As such, it does
not treat the functional aspects of the system: methodologies,
algorithm, philosophy, interpretation possibilities, usage of results,
etc. For a complete discussion of these and more, please consult the
various documents listed below.

## Acknowledgments

The following parties have significantly contributed to the development
of this system and are hereby acknowledged thus:

  ------------------ ----------------------------------------------------
  **Party**          **Contribution**

  Ray Buland and     Provided the original algorithm and proof-of-concept
  Daniel McNamara    implementation
  (of USGS NEIC,     
  Golden, CO)        

  USGS NEIC          Provided the funds for original development of
                     algorithm.

  NSF                Provided funds for original system development of
                     generic implementation at the IRIS Data Management
                     Center (Seattle, WA).

  IRIS               Sponsored the development of the generic
                     implementation at the DMC.

  Bruce Weertman (of Responsible for integration of the PDF system within
  IRIS)              the IRIS DMC's Quack framework.
  ------------------ ----------------------------------------------------

## Authorship

Both this document and the PDF analysis system itself was written by
Richard Boaz (of Boaz Consultancy:
[http://www.boazconsultancy.com](http://www.boazconsultancy.com/)). Any
and all comments and/or bug reports are welcome and are encouraged to be
forwarded to <riboaz@xs4all.nl>.

## References

The following table provides various references which may be of interest
to the reader:

  ----------------- -------------------- ------------------------------------------------------------------------------
  **Description**   **Name**             **Location**

  Original Abstract Ambient Noise Levels PDF Stand-Alone distribution docs directory
  (Adobe pdf)       in the Continental   
                    United States        

  Power Point       Noise Based          PDF Stand-Alone distribution docs directory
  Presentation      Detection Method for 
                    the ANSS             

  Discussion Paper  Determining True     PDF Stand-Alone distribution docs directory
  (Adobe pdf)       Global Ambient Noise 

  PDF Analysis      Ambient Noise        PDF Stand-Alone distribution docs directory
  Interpretation    Probability Density  
  (html document)   Functions            

  PDF Analyses at   USGS/ANSS Noise      <http://geohazards.cr.usgs.gov/staffweb/mcnamara/ANSSPDFweb/ANSSPDFweb.html>
  the USGS NEIC     Monitor              

  PDF Analyses at   DMC QUACK            <http://www.iris.washington.edu/servlet/quackquery/>
  the IRIS DMC      Information Query    

  PDF Analyses at   DMC QUACK            <http://www.iris.washington.edu/servlet/quackquery_us>
  the IRIS DMC (US  Information Query    
  Array)                                 
  ----------------- -------------------- ------------------------------------------------------------------------------

# PDF Analysis System Overview

A new system for analyzing data quality is now available to the
seismology community allowing users to evaluate the long-term seismic
noise levels for any broadband seismic data channel. The new noise
processing software uses a probability density function (PDF) to display
the distribution of seismic power spectral density (PSD) and has been
implemented against the entire continuous data-stream held by IRIS at
the DMC.

This noise processing system is unique in that there is no need to
screen the data for earthquakes, system glitches or general data
artifacts, as is commonly done in seismic noise analysis. Instead with
this new analysis, system transients map into a low-level background
probability while ambient noise conditions reveal themselves as high
probability occurrances. In fact, examination of artifacts related to
station operation and episodic cultural noise allows us to estimate both
the overall station quality and a baseline level of earth noise at each
site.

PDF noise plots are useful for characterizing the current and past
performance of existing broadband sensors, for detecting operational
problems within the recording system, and for evaluating the overall
quality of data for a particular station. The advantages of this new
approach include:

1.  provides an analytical view representing the true ambient noise
    levels rather than a simple absolute minimum;

2.  provides an assessment of the overall health of the
    instrument/station; and,

3.  provides an assessment of the health of recording and telemetry
    systems.

Please see the document **Ambient Noise Probability Density Functions**
for a more detailed discussion of the PDF plots themselves and their
interpretation possibilities.

## Description

Quite simply, the PDF Analysis system is comprised of three separate
processing components:

1.  An analysis program (written in C), responsible for producing
    analysis statistics for a single channel;

2.  A shell program (using GMT) to convert the analysis results produced
    in step 1 to a plot, in the form of a postscript format file;

3.  Image manipulation (using ImageMagick) commands to convert the
    postscript file created in step 2 to an actual graphical image (.png
    format).

Execution is provided in the form of a shell script per channel to
analyze, responsible for calling each of these components in turn (see
section **PDF System Execution** below for details).

## Requirements

### Hardware

No specific hardware requirements exist, per se. The program will
execute on any platform supporting a C compiler in addition to the other
software requirements listed below.

Depending on which compile time output option is chosen (please see
section **System Compilation and Installation: Compilation: Compile
Options** for a complete discussion of these options and their effects),
disk storage requirements are approximately the following (per channel
analyzed):

  -------------------------------------------- --------------------------
  **Output Option**                            **Maximum Disk Storage
                                               Requirement**

  No Daily or Hourly .bin output               5 Mb

  Only Daily .bin output                       15 Mb

  Both Daily and Hourly .bin output            50 Mb
  -------------------------------------------- --------------------------

### Software

The following table defines the software dependencies currently en
force:

  -------------- ------------- -------------------- --------------------------------
  **Software**   **Version**   **Description**      **Available At**

  C Compiler     User          Compiler (program    <http://gcc.gnu.org/>
                 preference    developed under gcc) 

  Scripting      Bash shell    Scripting tools      Local machine executing analysis

  GMT            Latest        Plotting tool        <http://gmt.soest.hawaii.edu/>
                 available                          

  ImageMagick    Latest        Image manipulation   <http://www.imagemagick.org/>
                 available     tool                 
  -------------- ------------- -------------------- --------------------------------

## PDF Source Code Organization

The following table defines the directories and files which make up the
source tree of the PDF Analysis System:

+---------------------+------------------------------------------------+
| **Directory/File**  | **Description**                                |
+---------------------+------------------------------------------------+
| PDF                 | Root system directory                          |
+---------------------+------------------------------------------------+
| PDF/PROD            | Production directory containing production     |
|                     | relevant files                                 |
+---------------------+------------------------------------------------+
| PDF/PROD/bin        | Directory containing shells and executables,   |
+---------------------+------------------------------------------------+
| PDF/PROD/script     | Directory containing executable scripts        |
+---------------------+------------------------------------------------+
| PDF/PROD/support    | Directory containing all necessary production  |
|                     | support files                                  |
+---------------------+------------------------------------------------+
| PDF/PROD/helper     | Directory containing helper scripts (system    |
|                     | mgmt, etc.)                                    |
+---------------------+------------------------------------------------+
| PDF/src             | Directory holding all source code:             |
|                     |                                                |
|                     | -   PDF analysis program                       |
|                     |                                                |
|                     | -   GMT plotting script                        |
|                     |                                                |
|                     | -   Execution scripts                          |
+---------------------+------------------------------------------------+
| P                   | Directory holding all source code for version  |
| DF/src/v*x*.*x*.*x* | *x*.*x*.*x* of the system                      |
+---------------------+------------------------------------------------+
| PDF/src/            | Source code directory of analysis program (C   |
| v*                  | code)                                          |
| x*.*x*.*x*/analysis |                                                |
+---------------------+------------------------------------------------+
| PDF/src/            | Miniseed data file reader source code - as     |
| v*x*.*x*            | library to main()                              |
| .*x*/analysis/mseed |                                                |
+---------------------+------------------------------------------------+
| PDF/src/            | Instrument response interpreter source code -  |
| v*x*.*x             | as library to main()                           |
| *.*x*/analysis/resp |                                                |
+---------------------+------------------------------------------------+
| PDF/src/            | Directory holding GMT source code - scripts    |
| v*x*.*x*.*x*/GMT    | and support files                              |
+---------------------+------------------------------------------------+
| PDF/src/            | Directory holding execution scripts            |
| v*x*.*x*.*x*/script |                                                |
+---------------------+------------------------------------------------+

# Input and Output Files

## Input

The following table defines the directories and files which are required
input, see sections **System Configuration** and **PDF System
Execution** for full description of specification and use:

+-----------------------+----------------------------------------------+
| **Directory/File**    | **Description**                              |
+-----------------------+----------------------------------------------+
| Data Directory        | Directory holding the miniseed data files to |
|                       | be analyzed.                                 |
|                       |                                              |
|                       | **N.B.** All data files requiring analysis   |
|                       | as part of a single execution must reside in |
|                       | this single directory.                       |
+-----------------------+----------------------------------------------+
| Miniseed Data Files   | Files to analyze, miniseed format only.      |
+-----------------------+----------------------------------------------+
| Analysis Directory    | Directory holding all output files.          |
|                       |                                              |
|                       | See section **Output** for a complete        |
|                       | description.                                 |
+-----------------------+----------------------------------------------+
| Response File         | Directory holding response files for         |
| Directory             | channels being analyzed.                     |
|                       |                                              |
|                       | See section **System Configuration** for a   |
|                       | complete discussion on set-up.               |
+-----------------------+----------------------------------------------+
| RESP.*N               | The file holding the response information    |
| TW*.*STN*.*LOC*.*CHN* | for the instrument and channel. This must be |
|                       | formatted as for input to the evresp()       |
|                       | function (format as produced by the rdseed   |
|                       | program).                                    |
|                       |                                              |
|                       | Where:                                       |
|                       |                                              |
|                       | *NTW* is the network name\                   |
|                       | *STN* is the station name\                   |
|                       | *LOC* is the location identifier\            |
|                       | *CHN* is the channel identifier              |
+-----------------------+----------------------------------------------+

##  

## Output

The following table defines the directories and files which are created
in the course of the PDF Analysis execution. All are located as
subdirectories to the analysis directory defined in the section
**Input** above and are automatically created in the course of
execution. Please consult **Appendix I -- File Formats** for a detailed
description of their contents.

+-----------------------+----------------------------------------------+
| **Directory/File**    | **Description**                              |
+-----------------------+----------------------------------------------+
| *NTW*.                | Graphical representation of analysis.        |
| *STN*.*LOC*.*CHN*.png |                                              |
+-----------------------+----------------------------------------------+
| Y*yyyy*               | Directory holding daily PSD .bin files, by   |
|                       | year                                         |
|                       |                                              |
|                       | Where:\                                      |
|                       | *yyyy* is the year                           |
+-----------------------+----------------------------------------------+
| Y*yyyy*/HOUR          | Directory holding the hourly PSD .bin files  |
+-----------------------+----------------------------------------------+
| LOG                   | Directory holding the various log files      |
|                       | created during the course of execution.      |
+-----------------------+----------------------------------------------+
| wrk                   | Directory holding various work files         |
+-----------------------+----------------------------------------------+
| Y*yyyy*/D*jjj*.bin    | Files holding individual day\'s PSD analysis |
|                       | results (currently unused, for future use).  |
|                       |                                              |
|                       | Where:\                                      |
|                       | *jjj* is the julian day                      |
+-----------------------+----------------------------------------------+
| Y*yyyy*/HOUR/hour.idx | Index file to H*jjj*.bin file.               |
+-----------------------+----------------------------------------------+
| Y*                    | Files holding individual hour\'s PSD         |
| yyyy*/HOUR/H*jjj*.bin | analysis results (currently unused, for      |
|                       | future use)                                  |
+-----------------------+----------------------------------------------+
| LOG/*NTW*.            | Log file of analysis program.                |
| *STN*.*LOC*.*CHN*.log |                                              |
+-----------------------+----------------------------------------------+
| LOG/plotGMT.log       | Log file of GMT plotting program.            |
+-----------------------+----------------------------------------------+
| LOG/convert.log       | Log file of ImageMagick convert program,     |
|                       | nothing output for normal execution.         |
+-----------------------+----------------------------------------------+
| LO                    | Analysis program error file, by year and     |
| G/*NTW*.*STN*.*LOC*.* | julian day                                   |
| CHN*.*yyyy*.*jjj*.err |                                              |
+-----------------------+----------------------------------------------+
| LOG/PDFanalysis.skp   | File listing those days when problems        |
|                       | occurred, information only                   |
+-----------------------+----------------------------------------------+
| wrk/PDFanalysis.bin   | Cumulative dB-based .bin file, results to    |
|                       | graph are contained here                     |
+-----------------------+----------------------------------------------+
| wrk/PDFanalysis.inf   | Information file holding various analysis    |
|                       | settings                                     |
+-----------------------+----------------------------------------------+
| wr                    | Cumulative period-based .bin file.           |
| k/PDFanalysis*SR*.bin |                                              |
|                       | Where:\                                      |
|                       | *SR* is the sample rate of the channel       |
+-----------------------+----------------------------------------------+
| wr                    | Information file as before, sample-rate      |
| k/PDFanalysis*SR*.inf | specific                                     |
+-----------------------+----------------------------------------------+
| wrk/PDFanalysis.sts   | File holding various statistics for analysis |
|                       | results, input to GMT                        |
+-----------------------+----------------------------------------------+
| wrk/PDFanalysis.ps    | GMT postscript file output, deleted upon     |
|                       | conversion to .png file.                     |
+-----------------------+----------------------------------------------+
| wrk/pdf.grd           | GMT temp file, deleted upon completion of    |
|                       | GMT step.                                    |
+-----------------------+----------------------------------------------+

# System Configuration

Configuration of the PDF system amounts to the setup of various
directories and script variables. This section lays out these
requirements for the PDF system setup to result in a successful
installation and subsequent execution. Failure to define these precisely
as described herein will result in a non-functioning system.

**Appendix II** provides a checklist for each parameter and variable
which must be defined as part of system setup. Please print, define the
values accordingly and supply them in their proper location.

## Data Directories

### MiniSeed Data Directory and Files

A directory is required to contain the miniseed data files to be
analyzed. This directory and the miniseed data files themselves must
adhere to the following:

1.  All miniseed data files representing a single channel's worth of
    waveform data, for all time, must be contained within the same
    directory.

2.  All miniseed data files must contain exactly (or as close to) one
    day's worth of data, from 00:00 hours to 24:00 hours.

3.  The directory structure must conform to the following conventions:

> *DATAROOT*/*NTW*/*STN*
>
> Where
>
> *DATAROOT* is the root directory of the miniseed data files (script
> variable of such specified below)
>
> *NTW* is the name of the network
>
> *STN* is the name of the station

4.  The name of the miniseed file itself must conform to the following
    naming convention:

> *STN*. *NTW*.*LOC*.*CHN*.*yyyy*.*jjj*
>
> Where
>
> *STN* is the name of the station
>
> *NTW* is the name of the network
>
> *LOC* is the location identifier
>
> *CHN* is the channel identifier
>
> *yyyy* is the year of the data file
>
> *jjj* is the julian day of the data file

(N.B. Where no location identifier exists, field should be null. This
would render, for example, a filename for station ATKA and network AK
as: ATKA.AK..BHE.2004.261)

Assuming your directory structure and miniseed data files do not
naturally conform to these requirements, this directory structure and
filenaming convention can be easily accommodated for through the
following:

-   Create the fixed directory structure itself: *DATAROOT*/*NTW*/*STN*

-   For each miniseed data file, create softlinks (ln --s) to the
    actual/real miniseed data files such that the naming convention
    above is adhered to.

Further, this can be automated via a script rendering this requirement
as trivial as possible. Please see the script linkMseed.US (located in
pdf/PROD/helper/linkmseed) for an example and modify as necessary.

N.B. Because the filenaming convention uniquely identifies the channel
of data, this directory may contain all miniseed data file for all
channels of a station, i.e., it is not necessary to create separate
directories for each channel, rather, a separate directory only for the
station itself and containing the miniseed data files for all channels.

### Response File Directory

A directory must exist containing all response files used to deconvolve
the signal back to absolute ground motion in the course of analysis.
This directory and the files themselves must adhere to the following:

1.  The directory itself may be specified as per user
    desires/requirements, no specific directory structure requirement
    exists. The directory holding these response files must be specified
    in two places: in the script variable RESPDIR (see section **Script
    Vaiables** below), and within the PDFuser.h file of the PDF analysis
    program source code (see section **Source Code Header File
    Modification** below).

2.  The filename of the response file itself must conform to the
    following naming convention:

> RESP.*NTW*.*STN*.*LOC*.*CHN*
>
> Where
>
> RESP is exactly as specified: RESP
>
> *NTW* is the name of the network
>
> *STN* is the name of the station
>
> *LOC* is the location identifier

*CHN* is the channel identifier

> (N.B. This naming convention follows from the response file output
> generated by the rdseed program. And as before, where no location
> identifier exists, field should be null.)

3.  The internal format of the response file information must conform to
    the format produced by the rdseed program, subsequently readable by
    evresp().

### Web Directory

A directory must be created used to collect all .png files created
during the course of execution. The .png files themselves are contained
in the analysis directory for the channel being analyzed, making
collective viewing annoying since all are held within disparate
directories. This annoyance is alleviated through the existence of this
directory.

Create a directory for these to be contained in and define this location
in the **.vars-user** file. With this directory, the last action in the
course of analysis is for a softlink to be created in this directory
pointing to the .png file found in the analysis directory.

Additionally, if it is desired to publish the results, it is this
directory that can be made available to the web in whatever manner/means
appropriate.

## Script Variables 

The following script variables are installation-specific and must be
pre-defined by the user and provided in the shell script file
**.vars-user** (located in directory PDF) before the system can be
installed. Failure to do so will result in a non-functioning system.

  ---------------- ------------------------------------------------------
  **Script         **Description**
  Variable Name**  

  PDFROOT          Root directory of PDF analysis system (directory
                   holding the .vars-user file)

  WEBDIR           Directory of collected .png files

  RESPDIR          Directory holding response files

  DATAROOT         Root directory of miniseed data files (parent to
                   *NTW*/*STN*)

  STATSROOT        Root directory of PDF analysis results/statistics

  GMTROOT          GMT installation root directory

  IMROOT           ImageMagick installation root directory
  ---------------- ------------------------------------------------------

## Source Code Header File Modification

The sole configuration requirement within the source code of the PDF
analysis program is the following #define parameter to be specified:
(N.B. needless to say, this must be the same as defined as part of the
Script Variables above.)

  ---------------- ----------------------- -----------------------------------------
  **#define        **Description**         **Location**
  parameter**                              

  #define RESPDIR  Directory holding       PDF/src/v*x*.*x*.*x*/analysis/PDFuser.h
                   response files, inside  
                   quotes "".              
  ---------------- ----------------------- -----------------------------------------

# System Compilation and Installation

## Compilation

Compilation of the PDF Analysis program employs straightforward C/Unix
standards, i.e, a **C compiler** and **make**. In addition to the
analysis portion of the program, there are two subdirectories of
libraries requiring compilation as well. As such, a script is provided
that will traverse each of these subdirectories, making the dependent
libraries in turn. This script is located in the PDF Analysis source
directory and conforms to the following invocation specifications:

**makesh \[ *clean \|* *all* \]**

where

***clean*** will execute **make clean**, removing all dependant
libraries and object files.

***all*** will execute **make all** in each directory of the PDF
analysis program, creating all dependant libraries and object files
necessary to ultimately link the PDF analysis executable.

Alternatively, the program may be built when installing the system as a
whole, alleviating the need to compile and link by hand. Please see the
section **Installation** below for details.

### Compiler Options

Two compile-time options exist for the PDF Analysis program. Namely,
defining whether or not daily and/or hourly PSD information is output.
(Please see section **Requirements: Hardware** for detailed overall disk
storage requirements.)

With daily PSD information output, cumulative .bin files are generated
for each day analysed (amounting to \~30Kb/day/channel analysed).

With hourly PSD information output, .bin files are generated for each
hour analysed (amounting to \~100Kb/day/channel analysed).

The system is delivered, by default, to output both daily and hourly
.bin files. Output of this data is anticipated to be used in future
versions of the software, such that existence of these files will allow
PDFs to be produced for specific user-defined time periods. For example,
a PDF graph representing only the months of January thru March; or a PDF
graph representing all months but only between the hours of 6AM and 6PM.

If it is anticipated that these more specific sorts of analyses will be
of interest, no action is required, both daily and hourly .bin files
will be generated.

If this is not desired, or disk space is an issue, both daily and hourly
.bin file generation may be suppressed. The following table defines
these compile time options:

  ---------------------------- ------------------------------------------
  **Compiler Option**          **Effect**

  -DNO_DAILY_PSD               No daily PSD .bin files output

  -DNO_HOURLY_PSD              No hourly PSD .bin files output
  ---------------------------- ------------------------------------------

Either or both (they are mutually independent) of these options can be
specified in the **CFLAGS** section of the **Makefile** for the main PDF
Analysis program. The resulting executable will subsequently **NOT**
output incremental PSD information.

N.B. Since these are compiler options, these settings have a system-wide
influence, i.e., these options **cannot** be implemented on a per
channel basis. (One way around this, however, would be to install more
than a single system.)

## Installation

Installation is provided via the shell script **installPDF** located in
the directory PDF and provides for the following functionality:

1.  optionally compiles the PDF Analysis program, and

2.  copies all relevant files to the PROD directory.

And conforms to the following usages:

**command**: *installPDF --h*

**output**: Usage: install \[-h\] \[make\] version#

**description:** prints the usage for the command.

**command**: *installPDF v1.1*

**output**: Copying v1.1 executables and support files to PROD dir...

**description:** installs all relevant executables, scripts and support
files for *version#* to the PROD directory structure. (N.B. command line
argument *version#* must be as specified in the PDF source directory
structure.)

**command**: *installPDF make v1.1*

**output**: Compiling v1.1 PDF Analysis program...

Copying v1.1 executables and support files to PROD dir...

**description:** as for command *installPDF v1.1*, however, cleans and
compiles the PDF analysis program before copying and installing to the
PROD directory structure (recommended for first install since no object
files exist as part of delivery).

# PDF System Execution

System execution comes in the form of two scripts, **executePDF** and
**PDFscript** (both located in PDF/PROD/bin).

**PDFscript** is a shell script template used to create the individual
channel-specific execution script, this script ultimately responsible
for the PDF analysis execution of a specific channel.

The **executePDF** script executes all channel-specific scripts located
in PDF/PROD/script in turn.

## Channel-Specific Shell Script

Execution of the PDF Analysis System is provided in the form of a shell
executable script (**PDFscript**) responsible for carrying out the three
steps of execution described in section **PDF Analysis System Overview:
Description**. This execution script is created by replacing various
strings in the generic file PDFscript with execution-specific values,
thus creating a unique script for each channel to be analyzed.

This channel-specific executable shell can be easily created using the
following shell script command:

**makePDFscript** *NTW STN LOC CHN*

where

> *NTW* is the network name\
> *STN* is the station name\
> *LOC* is the location identifier (use \-- for no location)\
> *CHN* is the channel identifier

The will create the channel-specific script to be executed, named
*NTW*.*STN*.*LOC*.*CHN*.sh and saved to the directory PDF/PROD/script.
In addition, the analysis directory will also be created if it does not
exist (assuming the STATSROOT directory exists).

Once this script has been created for a specific channel, it can be
simply repeatedly executed (daily, weekly, monthly, as desired) to
update the analysis results.

## Overall Execution

Executing the script **executePDF** will result in all scripts located
in PDF/PROD/script to be executed in turn. Specifically, it will execute
all files having ".sh" as their filename suffix. Thus, individual
analyses can be turned on and off simply by renaming the suffix of the
executable script in question.

A simple logfile of **executePDF**, detailing the channels analyzed, is
generated and written to the file PDF/PROD/LOG/PDF.log.

Further, this process may be automated using the UNIX **crontab**
command. At specified times, merely execute the **executePDF** script
and all analyses will be performed and updated.

## Execution Features

Please note the following features of the PDF analysis system:

1.  Analysis is performed against all data files found in the specified
    data directory only up to and until two days prior to analysis
    execution date. This allows for systems where data availability can
    suffer up to two days of latency.

2.  The program does cross the day boundary to analyze the last hour of
    data held within the miniseed files. As such, the last file
    identified for analysis is not fully analyzed; it is treated in
    subsequent executions.

3.  When crossing the day boundary, program assumes data is coincident
    if ending time and starting time of respective miniseed files differ
    by less than one sample (1) of time. Where data is identified not to
    be coincident, an error message is output to the channel-specific
    log file.

## Error Processing

Error handling is very much dependent on the type of error encountered.
The following table lists the major errors that may be encountered, how
each is handled, and suggested follow-up action.

  ------------- ----------- ----------------------- -------------------------
  **Error**     **Type**    **How Handled**         **User Follow-up**

  Response File Fatal       Analysis execution      Provide proper response
  not found                 suspended; error        file, verify naming
                            message written to Log  convention is adhered to.
                            file                    

  Response      Fatal       Analysis execution      Provide file containing
  Information               suspended; error        response information for
  not found                 message written to Log  appropriate date/time
                            file                    range, verify file format
                                                    is adhered to.

  Error reading Non-fatal   Analysis execution      Determine if miniseed
  miniseed data             skips this day of data; data file can be
  file                      error message written   repaired.
                            to day-specific error   
                            file located in LOG     
                            directory of analysis   
                            output.                 

  Internal      Fatal       Analysis execution      Contact
  Processing                suspended               <riboaz@xs4all.nl> with
  Error                                             all relevant information

  No files ever Non-fatal   Program successfully    Verify miniseed directory
  found to                  executes but nothing    structure and filenaming
  analyze                   analyzed                conventions are adhered
                                                    to.

  Unable to     Fatal       Analysis execution      Confirm existence and
  create or                 suspended               access permissions of
  access                                            directory in question.
  directories                                       
  ------------- ----------- ----------------------- -------------------------

# Release Notes

## Current Limitations

The limitations are currently defined to be:

1.  unable to analyze long period channels, limited to 20 samples/sec or
    more;

2.  unable to provide analysis for user-defined time periods, only
    analysis of all data available is currently provided for;

3.  output provided to end-user currently limited to .png graphical file
    format;

4.  unable to analyze any data prior to the last day analyzed. This
    means that the program cannot fill in gaps nor replace the results
    of previously analyzed data. This will be treated in a future
    release.

## Version Control

This section defines the current and historical releases of the PDF
Analysis System.

### Version 1.1

Release Date: 29-October-2004

**Modifications**

1.  Stand-alone version created and made available for general release.

# Appdendix I -- File Formats

This appendix defines the formats of the various files produced by the
PDF analysis system.

## Djjj.bin File

Definition: Cumulative db-based .bin file holding **daily** PSD analyses
for julian day *jjj*.

Directory: Y*yyyy*

Internal Format:

-   Data Format: ASCII

-   Individual lines each defining:

*FREQ POWER #HITS*

Where:

*FREQ* is the frequency (in Hz)

*POWER* is the power bin (in dB)

*#HITS* is the number of times

## Hour.idx File

Definition: Index file to H*jjj*.bin file, index defined by julian day
and *HH:MM* start time of PSD.

Directory: Y*yyyy/*HOUR

Internal Format:

-   Data Format: ASCII

-   Individual lines each containing:

*JDAY HH:MM REF*

Where:

*JDAY* is the julian day\
*HH:MM* is the hour and minute start time of the PSD\
*REF* is the reference identifier, for accessing/extracting from the
H*jjj*.bin file

## Hjjj.bin File

Definition: Cumulative db-based .bin file holding **hourly** PSD
analyses for julian day *jjj*.

Directory: wrk

Internal Format:

-   Data Format: ASCII

-   Individual lines each containing:

*REF FREQ POWER*

Where:

*REF* is the reference identifier from the hour.idx file\
*FREQ* is the frequency (in Hz)\
*POWER* is the power bin (in dB)

## PDFAnalysis.bin File

Definition: Cumulative db-based .bin file holding **overall** PSD
probabilities for julian day *jjj*.

Directory: wrk

Internal Format:

-   Data Format: ASCII

-   Individual lines each defining:

*FREQ POWER PROB*

Where:

*FREQ* is the frequency (in seconds)

*POWER* is the power bin (in dB)

*PROB* is the normalized (to probability) number of hits

## PDFAnalysis.inf File

Definition: Information file containing various settings/values
pertaining to analysis.

Directory: wrk

Internal Format:

-   Data Format: ASCII

-   Individual lines each defining:

*VALUE* :SETTING

-   With the following SETTINGs currently being provide for, appearing
    in the following order:

  ------------------------ ----------------------------------------------
  **SETTING**              **Definition**

  Analysis Start Date      Start day of the analysis (format: *YYYY:JJJ*)

  Analysis Stop Date       Stop day of the analysis (format: *YYYY:JJJ*)

  Total Number of Days     Total number of days analyzed

  Total Number of PSD\'s   Total number of PSD's making up the analysis

  Total Number of Problem  Total number of days encountering a problem
  Days                     (information only)

  SAMPLE RATE              Sample rate of the channel

  NETWORK                  Network name

  STATION                  Station name

  LOCATION                 Location identifier

  CHANNEL                  Channel identifier

  NYQUIST                  Nyquist value for this analysis
  ------------------------ ----------------------------------------------

## PDFanalysis.sts File

Definition: File holding various statistics for this analysis.

Directory: wrk

Internal Format:

-   Data Format: ASCII

-   Individual lines each defining:

*FREQ MIN AVE 50% 90% MAX MODE*

Where:

*FREQ* is the frequency in question (in Hz)

*MIN* is the minimum PSD value

*AVE* is the average PSD value

*50%* is the 50^th^ percentile PSD value

*90%* is the 90^th^ percentile PSD value

*MAX* is the maximum PSD value

*MODE* is the mode PSD value (most common)

## PDFAnalysisSR.bin File

Definition: Cumulative db-based .bin file holding **overall** PSD for
julian day *jjj*.

Directory: wrk

Internal Format:

-   Data Format: ASCII

-   Individual lines each defining:

*FREQ POWER PROB*

Where:

*FREQ* is the frequency (in Hz)

*POWER* is the power bin (in dB)

*PROB* is the number of hits for this bin

# Appendix II -- Installation and Configuration Checklist

This appendix contains a table listing all of the required configuration
parameters. Please print this page, providing the appropriate values to
be used. Confirm that all values have been properly defined in the
specified file(s).

  ------------- ------------------ ----------------------------------------- --------------------------
  **Config      **Description**    **Where Specified**                       **Value**
  Parameter**                                                                

  PDFROOT       Root directory of  PDF/.vars-user file                       
                PDF analysis                                                 
                system                                                       

  WEBDIR        Directory of .png  PDF/.vars-user file                       
                files accessible                                             
                from www                                                     

  RESPDIR       Directory          PDF/.vars-user file                       
                containing                                                   
                response files                                               

  DATAROOT      Root directory of  PDF/.vars-user file                       
                miniseed data                                                
                files                                                        

  STATSROOT     Root directory of  PDF/.vars-user file                       
                PDF analysis                                                 
                results                                                      

  GMTROOT       GMT installation   PDF/.vars-user file                       
                root directory                                               

  IMROOT        ImageMagick        PDF/.vars-user file                       
                installation root                                            
                directory                                                    

  RESPDIR       Directory          PDF/src/v*x*.*x*.*x*/analysis/PDFuser.h   
                containing                                                   
                response files                                               
  ------------- ------------------ ----------------------------------------- --------------------------