# Presentation: 672512

## Update on a New EPICS Archiver

**Kay Kasemir and Leo R. Dalesio**
**09/27/99**

## Outline

- Performance numbers for the data storage and retrieval.
- Disk Utilization Numbers
- Current Archiver Use
- User Interface for operation and data viewing
- Code Structure
- Data Managemente
- Conclusions

## Performance Numbers - Archiving

- These tests were performed on a 450 MHz Pentium.
- 500 channels changing at 10 Hz and written every 15 seconds uses 20% of the CPU time during collection and 100% of the CPU time for 5 seconds while writing to disk.
- 1,000 channels changing at 10 Hz and written every 15 seconds uses 100% of the CPU continually, but allows the status viewer to run, but more slowly.

## Performance - Retrieval

- The retrieval tests were done in two stages.
- 1) The data file was from the previous 12 months
-      The data was in over 1,000 files (early files were made every 4 hours).
-      These files contained over 3 Gbytes
- 2) The data was compressed and started from June 1st
-      There were only 3 files - 1 per month.
-      These files were 1.3 Gbytes
- The retrieval tests were performed on a Sun Ultra 1 running Solaris.

## Performance Numbers - Retrieval from over 1,000 files

- # RFQHR_KDvrInRfl_1_sig:
- 09/01/99        2 (0.07)        09/01/99 00:00:30.000
- 08/01/99        11 (0.19)       08/02/99 12:43:00.000
- 07/01/99        6 (0.19)        07/01/99 06:35:00.000
- 06/01/99        9 (0.27)        06/01/99 12:40:30.000
- 05/01/99        18 (0.25)       05/03/99 17:28:00.000
- 04/01/99        15 (0.36)       04/05/99 08:10:30.000
- 03/01/99        85 (1.12)       03/02/99 09:59:30.000
- 02/01/99        98 (1.02)       02/01/99 11:46:00.000
- 01/01/99        48 (1.1)        01/04/99 09:25:30.000
- 12/01/98        63 (1.16)       12/10/98 09:31:00.000
- 11/01/98        58 (1.13)       11/02/98 09:40:30.000
- 10/01/98        1 (0.04)        10/21/98 11:04:24.844180846
- # RFQHR_KCirRfl_3_sig:
- 09/01/99        1 (0.12)        09/01/99 00:00:00.666656968
- 08/01/99        5 (0.12)        08/02/99 12:42:30.000
- 07/01/99        47 (0.29)       07/01/99 06:34:30.000
- 06/01/99        45 (0.49)       06/01/99 12:37:30.000
- 05/01/99        21 (0.45)       05/03/99 17:27:30.000
- 04/01/99        28 (0.67)       04/05/99 08:09:00.000
- 03/01/99        41 (0.87)       03/02/99 10:00:00.000
- 02/01/99        42 (0.76)       02/01/99 11:44:30.000
- 01/01/99        43 (0.82)       01/04/99 09:25:30.000
- 12/01/98        56 (1.07)       12/01/98 15:08:30.000
- 11/01/98        48 (1.03)       11/02/98 09:40:30.000
- 10/01/98        1 (0.06)        10/21/98 11:04:14.427514796

- # RFQLR_HPM_K3_Lvl06: 03/15/99 12:05:59.287883068 - 09/09/99 02:57:30.916664800
- 09/01/99        2 (0.12)        09/01/99 00:00:00.616657362
- 08/01/99        6 (0.13)        08/02/99 15:19:30.000
- 07/01/99        3 (0.1)          07/01/99 12:44:30.000
- 06/01/99        1 (0.07)        06/11/99 15:04:30.000
- 05/01/99        0 (0.07)        05/03/99 17:27:30.000
- 04/01/99        3 (0.14)        04/06/99 14:09:30.000
- 03/01/99        0 (0.05)        03/15/99 12:05:59.287883068
- # rp_bp_diag_07_ilock.A: 03/16/99 09:39:39.183333246 - 09/09/99 09:22:00.000
- 09/01/99        0 (0.05)        09/02/99 15:29:00.000
- 08/01/99        0 (0.07)        08/05/99 07:02:00.000
- 07/01/99        1 (0.08)        07/03/99 20:33:30.000
- 06/01/99        1 (0.09)        06/11/99 15:03:00.000
- 05/01/99        0 (0.08)        05/01/99 14:31:00.000
- 04/01/99        1 (0.07)        04/01/99 09:58:30.000
- 03/01/99        0 (0.05)        03/16/99 09:39:39.183333246
- # RFQHR_KFilRmp_1_sig: 10/21/98 11:04:24.844180846 - 09/09/99 09:22:00.000
- 09/01/99        1 (0.07)        09/01/99 07:59:30.000
- 08/01/99        3 (0.14)        08/02/99 12:43:00.000
- 07/01/99        2 (0.09)        07/01/99 06:35:00.000
- 06/01/99        4 (0.15)        06/01/99 12:40:30.000
- 05/01/99        1 (0.07)        05/03/99 17:28:00.000
- 04/01/99        4 (0.2)          04/05/99 08:10:30.000
- 03/01/99        3 (0.14)        03/02/99 09:59:30.000
- 02/01/99        1 (0.14)        02/01/99 11:46:00.000
- 01/01/99        2 (0.12)        01/04/99 09:25:30.000
- 12/01/98        2 (0.15)        12/10/98 09:31:00.000
- 11/01/98        0 (0.11)        11/02/98 09:40:30.000
- 10/01/98        0 (0.06)        10/21/98 11:04:24.844180846

## Performance Numbers - Retrieval from 3 files

- # RFQLR_HPM_D14A_FldCvt.VALA:
- 09/01/99        0 (0.18)        09/01/99 00:00:26.666656320
- 08/01/99        0 (0.05)        08/02/99 15:19:30.000
- 07/01/99        0 (0.06)        07/01/99 12:44:30.000
- 06/01/99        1 (0.06)        06/12/99 15:31:30.534141192
- # RFQHR_KFilRmp_1_sig:
- 09/01/99        0 (0.05)        09/01/99 07:59:30.000
- 08/01/99        0 (0.05)        08/02/99 12:43:00.000
- 07/01/99        0 (0.06)        07/01/99 06:35:00.000
- 06/01/99        0 (0.05)        06/01/99 12:40:30.000
- # RFQWTR_A23H_slope_S:
- 09/01/99        0 (0.07)        09/01/99 07:58:00.000
- 08/01/99        0 (0.06)        08/02/99 13:03:00.000
- 07/01/99        0 (0.05)        07/01/99 07:07:00.000
- 06/01/99        0 (0.05)        06/02/99 10:36:00.000
- # RFQWTR_WVF_B_RPM:
- 09/01/99        0 (0.28)        09/01/99 00:00:00.616657716
- 08/01/99        0 (0.13)        08/02/99 13:03:00.000
- 07/01/99        0 (0.13)        07/01/99 07:07:00.000
- 06/01/99        0 (0.05)        06/02/99 10:36:00.000
- # rp_bp_rf_08_ilock.A:
- 09/01/99        0 (0.05)        09/02/99 15:29:00.000
- 08/01/99        1 (0.06)        08/05/99 07:02:00.000
- 07/01/99        0 (0.05)        07/03/99 20:33:30.000
- 06/01/99        0 (0.04)        06/11/99 15:03:00.000

- # RFQHR_KDvrInRfl_1_sig:
- 08/01/99        0 (0.13)        08/02/99 12:43:00.000
- 07/01/99        1 (0.25)        07/01/99 06:35:00.000
- 06/01/99        0 (0.04)        06/01/99 12:40:30.000
- # RFQHR_KCirRfl_3_sig:
- 09/01/99        0 (0.14)        09/01/99 00:00:00.666656968
- 08/01/99        0 (0.07)        08/02/99 12:42:30.000
- 07/01/99        0 (0.16)        07/01/99 06:34:30.000
- 06/01/99        1 (0.04)        06/01/99 12:37:30.000
- # RFQLR_HPM_K3_Lvl06:
- 09/01/99        0 (0.13)        09/01/99 00:00:00.616657362
- 08/01/99        0 (0.08)        08/02/99 15:19:30.000
- 07/01/99        0 (0.06)        07/01/99 12:44:30.000
- 06/01/99        0 (0.05)        06/11/99 15:04:30.000
- # rp_bp_diag_07_ilock.A:
- 09/01/99        0 (0.06)        09/02/99 15:29:00.000
- 08/01/99        0 (0.05)        08/05/99 07:02:00.000
- 07/01/99        0 (0.05)        07/03/99 20:33:30.000
- 06/01/99        0 (0.04)        06/11/99 15:03:00.000
- # RFQHR_KCthV_2_sig:
- 09/01/99        0 (0.18)        09/01/99 00:00:00.299990172
- 08/01/99        1 (0.11)        08/02/99 12:42:00.000
- 07/01/99        0 (0.11)        07/01/99 06:33:30.000
- 06/01/99        0 (0.06)        06/01/99 12:38:00.000

## Performance - Retrieval

- Retrieval on the second set of data was always within one second and therefore not an issue.
- Retrieval from the first set was always reasonable for the first month - under 2 seconds of elapsed time.
- Retrieval over longer periods became unacceptable for some portion of the test and then improved as more tests were done.
- Perhaps we received more dedicated resources - perhaps the file structure had some problems - we do not understand the results yet.
- As shown in previous tests - it is better to have fewer files.

## Disk Utilization

- Files are created every user configured period specified in hours. Perhaps this should have been expressed in days.
- Within a file, the first buffer allocated will hold 64 samples, if this fills the second holds 256 samples and all subsequent buffers will hold 1,024 samples.
- The compression utility moves the data into one month files (currently hard-coded) and allocates buffers that contain up to 4,000 samples. The data buffers are sized to fit the data exactly.
- Every sample contains the value, time stamp, status and severity - nothing is done to save disk space.

## User Interfaces to the Archiver

- User  interfaces are required for control and status from the Archiver as well as archive data viewing.
- The first pass of the Archiver had an X windows user interface that was managed in the main line of the code. The limitations of this approach are that  the Archiver cannot be started on a machine until the window system is available and this structure does not support management from multiple workstations.
- The first archive data viewer (XARR from Jlab) is also X based and calls the retrieval library directly. The utility is UNIX based and requires that the disks containing the archive data be mounted.
- Approaches were considered to provide network support for archive engine management and data viewing.

## Alternatives Studied

## Data Management Screens

## Data Management Screens

## Data Viewing Screens

## Code Structure

- Rewritten in C++
- Data and Channel Iterator Classes allow storage and retrieval code to share the same base class.
- The viewing and extraction tools shown here use the iterator class and XARR is expected to convert to use it soon.
- New data stores, like a relational database, could be used by replacing the Data and Channel Iterators. The data taking engine and retrieval tools would all be reusable for this alternative data storage.

## Code Structure - Use the Iterator Class

## Data Management

- Currently there is a utility that allows a user to split an archive file.
- The split utility compresses data by reducing the files to one every 3 months (could easily be configurable) and it compresses multiple data buffers for (64, 256, 1024) into buffers of 4,000 samples.
- True data management is required to all selective data winnowing and algorithmic compression. This work is expected to be completed in the next 12 months.
- Supporting statistical compression would require significant changes all tools as each sample currently has just one value - need to allow for mean, median, standard deviation etc....