# Presentation: 028565

## burtSave

- Albert Kagarmanov
- Matthias Clausen

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Outline

- burSave and caPut loggingMonitoring any change using ca
- caPut logging and iocPutLogServer
- burtSave and warm rebootMonitoring changes of a give record list
- ioc reboot options (conditional)
- burtwb
- dbloadDatabase

- EPICS 2004 @ JAERI/ KEK

- December 2004

## burtSave

- BurtSave is clone of standard APS BackUp and Restore Tool. Output of burtSave is compatible with BURT so we can use it as input for burtwb.
- BurtSave works in WS-level (Sun-Solaris). This program write data to disk periodically (program option, default is 20 sec.) or by request (writing a value to special binary-soft-channel, typically by medm).

- EPICS 2004 @ JAERI/ KEK

- December 2004

## burtSave

_**BurtSave has 2 main regimes: **_
- 1) Full regime and 2) List regime
- _1) Full regime_ – all “external” changing of all PVs will be monitored and written (based on CAS-hook)
- At DESY for a each IOC we have special  RSL-record. This is string array-record.(Resident String Logger )
- Last CAS request is stored here. Typical RSL-value is
- 01-DEC-04 11:13:12 epicsg mkk_user fred_ao old=0.12 new=0.13 (min=0.05 max=0.34)

- EPICS 2004 @ JAERI/ KEK

- December 2004

## burtSave (monitoring any caPut)

- burtSave is simply monitoring this record on WS-side.
- It’s possible to exclude storing some record subset by using special option:
- burtSave –exclude “*soft*ao  CGG_*   HNSI61GG_ai “

- EPICS 2004 @ JAERI/ KEK

- December 2004

## burtSave (monitoring any caPut)

- Advantages of this method:
- Capture all changes written to IOC
- Keep overview of all fields ever changed in a single file
- Disadvanatages of this method:
- The file might not be useful for writeback(the order things have been changed is not necessarily the right order to write them back during reboot)
- Internal changes (record<->record/ SNL-> records) are not captured

- EPICS 2004 @ JAERI/ KEK

- December 2004

##  burtSave and ioc(Put)LogServer

*IOC*

- CAS

**caPut**

- rslRecord

- Disk

- Epics Monitor

**CA**

- WS
- Caput-call

- WS
- BurtSave(1)

- RngLog
- Task

- MQ

- EPICS 2004 @ JAERI/ KEK

- December 2004

## burtSave(monitoring list of records – 2nd regime)

- 2) List regimen– a list of PVs is monitored
- Monitor any desired field of any record( Capture also IOC-internal changes)
- Setup the list in a way that records and fields are written in the ‘right’ order

- EPICS 2004 @ JAERI/ KEK

- December 2004

##  burtSave and ioc(Put)LogServer

*IOC*

- CAS

**caPut**

- rslRecord

- Disk

- Epics Monitor

**CA**

- WS
- Caput-call

- WS
- BurtSave(1)

- RngLog
- Task

- MQ

- WS
- BurtSave (2)

- EPICS 2004 @ JAERI/ KEK

- December 2004

## burtSave(and the problem during IOC reboot)

- If IOC is disconnected or switch off:
- We also add special saveEnable binary record. If this PV is unavailable or it’s value is equal zero burtSave does nothing. In IOC initialization script this value is set to 1. This allows us to not store the first value which is coming from static db-file during initialization, and also stop all store activity during shutdown or power off.

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Bumpless Reboot:Special output for burtSave

- Standart burtwb does not allow really smooth warm reboot. Why?  Because for any PVs we set value twice  in dbLoad (before ioc_init())   and in burtwb after ioc_init() .Actually we can have some *gap * in data (see next picture)

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Bump’full’ reboot using burtwb

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Bumpless Reboot using db files

- A real bumpless rebot is possible if the desired data are either written to the loaded database, or of the databases is ‘overloaded’ by the desired settings.
- burSave can also create db files
- Typical db-output created by burtSave:
- #1102000618  - unix-time this is commentary line
- record(diglog,"HZF:V17_dlog") {
-  field(CMD,"0")
- }    .....
- record(diglog,"HZF:V21_dlog") {
-  field(CMD,“3")
- }

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Bumpless Reboot loading db files(conditional loading)

- Then we can overload db files by using the dbLoadRecord(filename) – call.
-  There might be cases where stored data are too old for a smooth reboot.
- If the conditions changed
- Loading db files with most recent changes only if the data are not too old.

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Bumpless Reboot(loading db file on time condition)

- To allow (time) conditional loading we use a small C-program which checks the difference between the current ioc time (T1=now)  and the file time T2 which comes from comment line of db-file (written by burtSave).
- For example   If |T1 – T2| < 20 min. dbLoadRecord(burtSaveDB-file)
- Else do nothing, default values from the ioc database files will not be overwritten.

- EPICS 2004 @ JAERI/ KEK

- December 2004

## caPut logging and IocLogMsgServer

- Communication between IOC part and WS level is simple TCP/IP-socket based client –server mechanism.
- WS caPutLogger and IOCLogMsgServer is one executable Unix server task with different ports:
- Export EPICS_IOC_CA_PUT_LOG_PORT=7010
- Export  EPICS_IOC_LOG_MSG_PORT=7004

- EPICS 2004 @ JAERI/ KEK

- December 2004

## caPut logging and IocLogMsgServer

- Use CAS implementation as a hook for CA-put loging
- Info has format
- 28-Sep-04 14:39:12 kryksunh kagarman HZF:CV 4_pid.KP new=0.2 old=0.3
- This info coming from IOC message queue and tcp/ip socket task to WS-level.
- The same procedure for logMsg-IOC-call. Typical format
- 2004-11-09 09:37:38 Illegal  Request V: HZF:T16P_ai.TIME error detected in routine: db_name_to_addr error
- 2004-08-09 16:35:01 0x7728390 (Seq_Alarms): 33 of 8 0 assigned channels have connected

- EPICS 2004 @ JAERI/ KEK

- December 2004

##  burtSave and ioc(Put)LogServer

*IOC*

- CAS

**caPut**

- rslRecord

- Put Logging
- Client

- Disk

- Epics Monitor

- TCP/IP socket Tasks

**CA**

- WS
- Caput-call

- WS
- BurtSave(1)

- WS
- caPutLogServer

- RngLog
- Task

- MQ

- WS
- BurtSave (2)

- EPICS 2004 @ JAERI/ KEK

- December 2004

##  burtSave and ioc(Put)LogServer

*IOC*

- CAS

**caPut**

- task

- rslRecord

- Put Logging
- Client

- Disk

- Epics Monitor

- TCP/IP socket Tasks

**CA**

- WS
- Caput-call

- WS
- BurtSave(1)

- WS
- caPutLogServer

- WS
- LogMsqServer

- LogMsg
- Client

- RngLog
- Task

- MQ

- logMsg

- MQ

- WS
- BurtSave (2)

- EPICS 2004 @ JAERI/ KEK

- December 2004

## caPut logging and IocLogMsgServer

- Output for ioclogServer is splitted by “logical IOC host”
- Physical2logical table as static file:
- epicsGPFC01 mkk10KVA1
- epicsGPFC06 mkk10KV2B
- ...
- epicsPC11 datLog11
- epicsPC21 mthKryoStand
- epicsPC22 heraKryoZeus

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Conclusion - burtSave

- The latest burtSave version provides real warm reboot features using:
- List of records.fields to be monitored
- db files
- time-conditional loading
- IocName:burtEnable_bi (save files only if this binary record is valid AND not zero)

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Conclusion - iocPutLogServer

- One image for iocLogMessages and iocPutLogServer
- One logfile for each IOC
- Mapping IP-Names to logical IOC names
- Execute command (send mail) on logfile size
- iocPutLogServer keeps track of any external (caPut) change to the IOC

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Appendix:

- Option for burtSave:
- burtSave -c confFile -l lockFile -d saveDir
- [-i interval]
- [-e=epicsDbFileOutputFacility]
- [-b=directMonitorFacility]
- [-x ''formatList'' forSubSetOfRecordExcluding]

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Appendix:

- Config file for burtSave
- ###########################
- 1.4                            # burtSave version
- # first 4 uncomment line is special PVs suffixes, then IOC names
- ###########################
- :caPutMsgLog_rsl     # rsl-record suffix full name is  epicsPC04:caPutMsgLog_rsl
- :setRebootSav_bi     # immediately create snap-file
- :last_burt_str             # time for previous
- :burtEnable_bi           # Enable/Disable snap-file writing
- # applications IOC host alias name here:
- epicsPC04
- heraKryoZeus

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Appendix:

- burtSave Snap-file (as in APS burt)
- --- Start BURT header
- Time: Fri Sep 3 11:00:00 2004
- Login ID: archiver (Archiver User)
- Eff UID: 901 Group ID: 100
- Keywords: Comments: Type: Absolute
- Directory /import/epicsf2/applic/burtSave/bin
- Req File: /applic/burtSave/conf/burtSave.conf
- --- End BURT header
- HZF:P1_ai.HIGH 1 1.23457
- HZF:P1_ai.VAL 1 2.34568
- HZF:V22_dlog.CMD 1 0
- HZF:P10_ai.HIGH 1 0
- HZF:CV8_ao.VAL 1 0
- HZF:V2_dlog.CMD 1 0
- HZF:V3_dlog.CMD 1 0
- HZF:V17_dlog.CMD 1 0

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Appendix:

- burtSave DB-file
- #1102086487 comment line Unix-time for file creation
- record(diglog,"HZF:V17_dlog") {
-             field(CMD,"0")
- }
- record(diglog,"HZF:V18_dlog") {
-              field(CMD,"0")
- }
- record(pid,"HZF:CV28_pid") {
-              field(SOUT,"2.700000e+01"
- }

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Appendix:

- Phys2logical.tbl
- epicsGPFC01 mkk10KVA1
- epicsGPFC02 mkk10KVB1
- epicsPC11 datLog11
- epicsPC12 datLog12
- epicsPC13 datLog13
- epicsPC14 datLog14
- epicsPC15 datLog15
- epicsPC21 mthKryoStand
- epicsPC22 heraKryoZeus
- epicsPPC01 radmes
- epicsPPC02 mkkPPC02
- epicsPPC03 vacuumCompress2
- epicsPPC04 VC2Test
- epicsPPC05 ttfRadMes
- epicsPPC11 mkkSender
- epicsVME07 krykWetter

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Appendix:

- Environment variable for iocLogServer
- 	EPICS_IOC_LOG_PORT=7004 EPICS_IOC_LOG_FILE_NAME=$TOP/log/iocLogServer.log EPICS_IOC_LOG_FILE_LIMIT=1000000 EPICS_IOC_LOG_CONFIG_FILE_NAME=$TOP/iocLogServer.conf EPICS_IOC_LOG_PHYS2LOG_TBL_NAME=$TOP/phys2log.tbl EPICS_IOC_LOG_MAIL_SUBJECT='iocLogServer output file is too big!' EPICS_IOC_LOG_COMMAND='sh $BIN/smsToLogbook.sh' EPICS_IOC_LOG_COMMAND_HOWTO=23 EPICS_IOC_LOG_COMMAND_SUBJECT='iocLogServer output file is too big'

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Appendix:

- Config file for iocLogServer
- epicsVME70.desy.de {
- 	monthly {
- 	logfile=/applic/iocLogServer/mth/mthKryoLT.log; limit=1000000; mail=michael.schwarzer@desy.de
- 	}
- }
- epicsVME35.desy.de {
- 	monthly {
- 	logfile=/applic/iocLogServer/hera/heraKryoFel.log; limit=1000000; mail=bernd.schoeneburg@desy.de
- 	}
- }
- epicsVME40.desy.de {
- 	monthly {
- 	logfile=/applic/iocLogServer/ttf/ttfKryoFV.log; limit=1000000; mail=bernd.schoeneburg@desy.de
- 	}
- }

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Appendix:

- outPut file for iocCaPutLogServer:
- [epicsPC22 <-> heraKryoZeus] 2004-08-09 16:33:52 09-Aug-04 16:38:54 kryksung claus heraKryoZeus:setRebootSav_bi.VAL new=0 old=0
- [epicsPC22 <-> heraKryoZeus] 2004-08-09 16:33:57 09-Aug-04 16:38:54 kryksung claus heraKryoZeus:last_burt_str.VAL new=2004-08-09:16.33 old=2004-08-09:16.36
- [epicsPC22 <-> heraKryoZeus] 2004-08-10 08:59:59 10-Aug-04 09:05:05 kryksunh schoeneb HZF:CV4_pid.AM new=M old=A
- [epicsPC22 <-> heraKryoZeus] 2004-08-10 09:00:06 10-Aug-04 09:05:08 kryksunh schoeneb HZF:CV4_pid.SOUT new=100 old=100 min=29.8701 max=100
- [epicsPC22 <-> heraKryoZeus] 2004-08-10 09:01:00 10-Aug-04 09:06:06 kryksunh schoeneb HZF:CV7_ao.OMSL new=supervisory old=closed_loop
- [epicsPC22 <-> heraKryoZeus] 2004-08-10 09:01:04 10-Aug-04 09:06:09 kryksunh schoeneb HZF:CV7_ao.VAL new=100 old=100 min=46.7533 max=100
- [epicsPC22 <-> heraKryoZeus] 2004-08-10 09:01:06 10-Aug-04 09:06:11 kryksunh schoeneb HZF:CV7_ao.OMSL new=closed_loop old=supervisory
- [epicsPC22 <-> heraKryoZeus] 2004-08-10 09:01:07 10-Aug-04 09:06:12 kryksunh schoeneb HZF:CV16_ao.OMSL new=supervisory old=closed_loop

- EPICS 2004 @ JAERI/ KEK

- December 2004

## Appendix:

- outPut file for iocLogMsgServer:
- [epicsPC22 <-> heraKryoZeus] 2004-11-08 13:05:14 0x74e7e94 (CA_event): rsrv: socket shutdown error was S_errno_EPIPE
- [epicsPC22 <-> heraKryoZeus] 2004-11-09 09:37:38 filename="../recGbl.c" line number=62
- [epicsPC22 <-> heraKryoZeus] 2004-11-09 09:37:38 Illegal Database Request Type P V: HZF:T16P_ai.TIME error detected in routine: db_name_to_addr error
- [epicsPC22 <-> heraKryoZeus] 2004-11-09 09:37:38 filename="../recGbl.c" line number=62
- [epicsPC22 <-> heraKryoZeus] 2004-11-09 09:37:38 Illegal Database Request Type P V: HZF:T16P_ai.TIME error detected in routine: db_name_to_addr error
- [epicsPC22 <-> heraKryoZeus] 2004-11-09 09:37:38 filename="../recGbl.c" line
- number 12

- EPICS 2004 @ JAERI/ KEK

- December 2004

##  burtSave and ioc(Put)LogServer

*IOC*

- CAS

**caPut**

- task

- rslRecord

- Put Logging
- Client

- Disk

- Epics Monitor

- TCP/IP socket Task

**CA**

- WS
- Caput-call

- WS
- BurtSave(1)

- WS
- caPutLogServer

- WS
- LogMsqServer

- LogMsg
- Client

- RngLog
- Task

- MQ

- logMsg

- MQ

- WS
- BurtSave (2)

- EPICS 2004 @ JAERI/ KEK

- December 2004