Evidian

SafeKit User's GuideHigh Availability Software for
Critical Applications

Overview
Subject

This document covers all the phases of the SafeKit implementation:architecture, installation, tests, administration & troubleshooting,support, and command line interface.

Intended
Readers

Architectures“High availability architectures” page 15“SafeKit cluster in the cloud” page 293

Installation“Installation” page 25

Console“The SafeKit web console” page 37“Securing the SafeKit web service” page 175

Advancedconfiguration“Cluster.xml for the SafeKit cluster configuration”
page 203“Userconfig.xml for a module configuration” page
209
“Scripts for a module configuration” page 267“Examples of userconfig.xml and module scripts”
page 273

Administration“Mirror module administration” page 95“Farm module administration” page 107“Command line interface” page 141“Advanced administration” page 155

Support“Tests” page 69“Troubleshooting” page 111“Access to Evidian support” page 133“Log Messages Index” page 309

Other“Table of Contents” page 5“Third-Party Software” page 305

Release

SafeKit 8.2

Supported
OS

Windows and Linux; for a detailed list of supported OS, see here

Web Site

Evidian marketing site:

http://www.evidian.com/safekit

Evidian support site:

https://support.evidian.com/safekit

39 A2 38MC 03

3

SafeKit User's Guide

Ref

39 A2 38MC 03
If you have any comments or questions related to this documentation, please mail us at
institute@evidian.com

Copyright © Evidian, 2024
The trademarks mentioned in this document are the propriety of their respective owners. The terms Evidian,AccessMaster, SafeKit, OpenMaster, SSOWatch, WiseGuard, Enatel and CertiPass are trademarks registered by
Evidian.

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by anymeans, electronic, mechanical or otherwise without the prior written permission of the publisher.

Evidian disclaims the implied warranties of merchantability and fitness for a particular purpose and makes noexpress warranties except as may be stated in its written agreement with and for its customer. In no event is
Evidian liable to anyone for any indirect, special, or consequential damages.

The information and specifications in this document are subject to change without notice. Consult your EvidianMarketing Representative for product or service availability.

4

39 A2 38MC 03

Table of Contents
SafeKit User's Guide High Availability Software for Critical Applications ........... 1
Overview ............................................................................................................ 3
Table of Contents ............................................................................................... 5
1.

High availability architectures .................................................................... 15

1.1

SafeKit cluster definition ............................................................................ 15

1.2

SafeKit module definition - application integration ........................................ 15

1.3

Mirror module: synchronous real time file replication and failover ................... 16

1.3.1

File replication and failover ............................................................................................. 16

1.3.2

Step 1. Normal operation ............................................................................................... 16

1.3.3

Step 2. Failover ............................................................................................................. 16

1.3.4

Step 3. Failback and reintegration ................................................................................... 17

1.3.5

Step 4. Return to normal operation .................................................................................. 17

1.3.6

Synchronous, fault-tolerant replication that loses no data when a server fails ....................... 18

1.4
1.4.1

Network load balancing and failover ................................................................................. 18

1.4.2

Principle of a virtual IP address with network load balancing ............................................... 19

1.4.3

Load balancing for stateful or stateless web services .......................................................... 19

1.5

Combining mirror and farm modules ........................................................... 20

1.5.1

Active/Active: 2 mirror modules backuping each other ....................................................... 20

1.5.2

N-to-1: N mirror modules with a single backup ................................................................. 20

1.6

2.

Farm module: network load balancing and failover ........................................ 18

The simplest high availability cluster in the cloud .......................................... 22

1.6.1

Mirror cluster in Microsoft Azure, Amazon AWS and Google GCP .......................................... 22

1.6.2

Farm cluster in Microsoft Azure, Amazon AWS and Google GCP ........................................... 23

Installation ................................................................................................. 25

2.1

SafeKit install ........................................................................................... 25

2.1.1

Download the package ................................................................................................... 25

2.1.2

Installation directories and disk space provisioning ............................................................ 25

2.1.3

Install procedure ........................................................................................................... 26

2.1.4

Use the SafeKit console or command line interface ............................................................ 28

2.1.5

SafeKit license keys ....................................................................................................... 29

2.1.6

System specific procedures and characteristics ................................................................. 29

2.2

Mirror installation recommendation ............................................................. 30

2.2.1

Hardware prerequisites .................................................................................................. 30

2.2.2

Network prerequisites .................................................................................................... 30

2.2.3

Application prerequisites ................................................................................................ 30

39 A2 38MC 03

5

SafeKit User's Guide

2.2.4

2.3

Hardware prerequisites .................................................................................................. 31

2.3.2

Network prerequisites .................................................................................................... 31

2.3.3

Application prerequisites ................................................................................................ 31

SafeKit upgrade ........................................................................................ 31

2.4.1

When proceed to an upgrade? ......................................................................................... 31

2.4.2

Prepare the upgrade ...................................................................................................... 31

2.4.3

Uninstall procedure ........................................................................................................ 32

2.4.4

Reinstall and postinstall procedure ................................................................................... 32

2.5

SafeKit full uninstall .................................................................................. 34

2.5.1

On Windows as administrator .......................................................................................... 34

2.5.2

On Linux as root ............................................................................................................ 34

2.6

SafeKit documentation .............................................................................. 35

The SafeKit web console ............................................................................. 37

3.1

Start the web console ................................................................................ 37

3.1.1

Start a web browser ...................................................................................................... 37

3.1.2

Connect to a SafeKit server ............................................................................................ 38

3.2

Configure the Cluster ................................................................................ 39

3.2.1

Cluster configuration wizard ............................................................................................ 39

3.2.2

Cluster configuration home page ..................................................................................... 42

3.3

Configure a module ................................................................................... 44

3.3.1

Select the new module to configure ................................................................................. 44

3.3.2

Module configuration wizard............................................................................................ 45

3.3.3

Modules configuration home page .................................................................................... 51

3.3.4

Add module scripts ........................................................................................................ 52

3.4

Monitor a module ...................................................................................... 53

3.4.1

Module state and status ................................................................................................. 55

3.4.2

Module control menus .................................................................................................... 56

3.4.3

Module details ............................................................................................................... 58

3.4.4

Module states timeline ................................................................................................... 64

3.5

Snapshots of module for support ................................................................ 65

3.6

Secure access to the web console ............................................................... 66

4.

Tests ........................................................................................................... 69

4.1

Installation and tests after boot .................................................................. 69

4.1.1

Test package installation ................................................................................................ 69

4.1.2

Test license and version ................................................................................................. 70

4.1.3

Test SafeKit services and processes running after boot ...................................................... 71

4.1.4

Test start of SafeKit web console ..................................................................................... 72

4.2

6

Farm installation recommendation .............................................................. 31

2.3.1

2.4

3.

File replication prerequisites ........................................................................................... 30

Tests of a mirror module ........................................................................... 72

39 A2 38MC 03

High availability architectures

4.2.1

Test start of a mirror module on 2 servers

STOP (NotReady) .......................................... 72

4.2.2

Test stop of a mirror module on the server

PRIM (Ready) ............................................. 72

4.2.3

Test start of a mirror module on the server

STOP (NotReady) ........................................ 73

4.2.4

Test restart of a mirror module on the server

PRIM (Ready) .......................................... 73

4.2.5

Test swap of a mirror module from one server to the other ................................................. 73

4.2.6

Test virtual IP address of a mirror module ........................................................................ 74

4.2.7

Test file replication of a mirror module ............................................................................. 75

4.2.8

Test mirror module shutdown on the server

PRIM (Ready) ............................................ 76

4.2.9

Test mirror module power-off on the server

PRIM (Ready) ............................................ 77

4.2.10

Test split brain with a mirror module................................................................................ 78

4.2.11

Continue your mirror module tests with checkers .............................................................. 78

4.3
4.3.1

Test start of a farm module on all servers

STOP (NotReady) .......................................... 79

4.3.2

Test stop of a farm module on one server

UP (Ready) .................................................. 79

4.3.3

Test restart of a farm module on one server

UP(Ready)................................................. 79

4.3.4

Test virtual IP address of a farm module .......................................................................... 80

4.3.5

Test TCP load balancing on a virtual IP address ................................................................. 82

4.3.6

Test split brain with a farm module .................................................................................. 83

4.3.7

Test compatibility of the network with invisible MAC address (vmac_invisible)....................... 84

4.3.8

Test farm module shutdown of a server

UP (Ready) ..................................................... 85

4.3.9

Test farm module power-off of a server

UP (Ready) ..................................................... 85

4.3.10Continue your farm module tests with checkers................................................................. 85

4.4

5.

Tests of a farm module .............................................................................. 79

Tests of checkers common to mirror and farm .............................................. 86

4.4.1

Test <errd>: checker of process with action restart or stopstart ......................................... 86

4.4.2

Test <tcp> checker of the local application with action restart or stopstart ........................... 87

4.4.3

Test <tcp> checker of an external service with action wait ................................................. 88

4.4.4

Test <interface check="on"> on a local network interface and with action wait ..................... 89

4.4.5

Test <ping> checker with action wait............................................................................... 90

4.4.6

Test <module> checker with action wait .......................................................................... 91

4.4.7

Test <custom> checker with action wait .......................................................................... 92

4.4.8

Test <custom> checker with action restart or stopstart ...................................................... 93

Mirror module administration ..................................................................... 95

5.1

Operating mode of a mirror module ............................................................ 96

5.2

State automaton of a mirror module (STOP, WAIT, ALONE, PRIM, SECOND NotReady, Transient, Ready) ................................................................. 97

5.3

First start-up of a mirror module (safekit prim command) .......................... 98

5.4

Different reintegration cases (use of bitmaps) .............................................. 99

5.5

Start-up of a mirror module with the up-to-date data
STOP (NotReady) WAIT (NotReady) ................................................................................... 100

39 A2 38MC 03

7

SafeKit User's Guide

5.6

Degraded replication mode (5.7

Automatic or manual failover ................................................................... 103

5.8

Default primary server (automatic swap after reintegration) ........................ 105

5.9

Prim command fails: why? (safekit primforce command)......................... 106

6.

Farm module administration ..................................................................... 107

6.1

Operating mode of a farm module ............................................................ 107

6.2

State automaton of a farm module (STOP, WAIT, UP - NotReady, Transient,
Ready) .................................................................................................. 108

6.3

Start-up of a farm module ....................................................................... 109

7.

Troubleshooting ....................................................................................... 111

7.1

Connection issues with the web console ..................................................... 111

7.1.1

Browser check............................................................................................................. 112

7.1.2

Browser state clear ...................................................................................................... 112

7.1.3

Server check ............................................................................................................... 112

7.2

Connection issues with the HTTPS web console ........................................... 113

7.2.1

Check server certificates .............................................................................................. 113

7.2.2

Check certificates installed in SafeKit ............................................................................. 115

7.2.3

Revert to HTTP configuration ........................................................................................ 115

7.3

How to read logs and resources of the module? .......................................... 116

7.4

How to read the commands log of the server? ............................................ 116

7.5

Stable module

7.6

Degraded module

7.7

Out of service module

7.8

ModuleSTOP (NotReady): restart the module ........................................ 118

7.9

ModuleWAIT (NotReady): repair the resource="down" ........................... 119

7.10

Module oscillating from

7.11

Message on stop after maxloop ................................................................ 121

7.12

Module7.13

Mirror module

ALONE (Ready) -7.14

Farm module

UP(Ready)but problem of load balancing in a farm............... 124(Ready) and(Ready)and/(Ready) ................................................ 117/(NotReady) ................................... 117(NotReady) and(Ready) to/(NotReady) .................. 117(Transient) ............................... 120(Ready) but non-operational application .................................... 122
WAIT/

STOP (NotReady) .................... 123

7.14.1

Reported network load share are not coherent ................................................................ 124

7.14.2

virtual IP address does not respond properly .................................................................. 124

7.15

Problem after Boot .................................................................................. 124

7.16

Analysis from snapshots of the module ...................................................... 125

7.16.1Module configuration files ............................................................................................. 125

7.16.2Module dump files ....................................................................................................... 126

7.17
8

ALONE (Ready) degraded) ........................... 101

Problem with the size of SafeKit databases ................................................ 128
39 A2 38MC 03

High availability architectures

7.18

Problem for retrieving the certification authority certificate from an external
PKI ....................................................................................................... 129

7.18.1

7.19
8.

Export CA certificate(s) from public certificates ............................................................... 130

Still in Trouble ........................................................................................ 132

Access to Evidian support ......................................................................... 133

8.1

Home page of support site ....................................................................... 133

8.2

Permanent license keys ........................................................................... 134

8.3

Create an account ................................................................................... 135

8.4

Access to your account ............................................................................ 135

8.5

Call desk to open a trouble ticket .............................................................. 136

8.5.1

Call desk operations..................................................................................................... 136

8.5.2

Create a call ............................................................................................................... 136

8.5.3

Attach the snapshots ................................................................................................... 137

8.5.4

Answers to a call and exchange with support .................................................................. 138

8.6
8.6.1

Two areas of download and upload ................................................................................ 139

8.6.2

Product download area ................................................................................................. 139

8.6.3

Private upload area...................................................................................................... 140

8.7
9.

Download and upload area ....................................................................... 139

Knowledge base ..................................................................................... 140

Command line interface ............................................................................ 141

9.1

Distributed commands ............................................................................. 141

9.2

Command lines for boot and for shutdown ................................................. 143

9.3

Command lines to configure and monitor safekit cluster .............................. 144

9.4

Command lines to control modules ........................................................... 146

9.5

Command lines to monitor modules .......................................................... 148

9.6

Command lines to configure modules ........................................................ 149

9.7

Command lines for support ...................................................................... 151

9.8

Examples ............................................................................................... 152

9.8.1

Cluster configuration with command line ........................................................................ 152

9.8.2

New module configuration with command line ................................................................. 152

9.8.3

Module snapshot with command line .............................................................................. 153

10. Advanced administration .......................................................................... 155
10.1

SafeKit environment variables and directories ............................................ 155

10.1.1

Global ........................................................................................................................ 155

10.1.2Module ....................................................................................................................... 156

10.2

SafeKit processes and services ................................................................. 157

10.3

Firewall settings ..................................................................................... 158

10.3.1

Firewall settings in Linux .............................................................................................. 158

10.3.2Firewall settings in Windows ......................................................................................... 159

39 A2 38MC 03

9

SafeKit User's Guide

10.3.3

10.4

Other firewalls ............................................................................................................ 159

Boot and shutdown setup in Windows........................................................ 163

10.4.1

Automatic procedure .................................................................................................... 163

10.4.2

Manual procedure ........................................................................................................ 163

10.5

Securing module internal communications ................................................. 164

10.5.1

Configuration with the SafeKit Web console .................................................................... 164

10.5.2

Configuration with the Command Line Interface .............................................................. 164

10.5.3

Advanced configuration ................................................................................................ 165

10.6

SafeKit web service configuration ............................................................. 166

10.6.1

Configuration files ....................................................................................................... 167

10.6.2Connection ports configuration ...................................................................................... 168

10.6.3

HTTP/HTTPS and user authentication configuration .......................................................... 169

10.6.4

SafeKit API ................................................................................................................. 169

10.7

Mail notification ...................................................................................... 169

10.8

SNMP monitoring .................................................................................... 170

10.8.1

SNMP monitoring in Windows ........................................................................................ 170

10.8.2

SNMP monitoring in Linux ............................................................................................. 170

10.8.3The SafeKit MIB .......................................................................................................... 171

10.9

Commands log of the SafeKit server ......................................................... 172

10.10 SafeKit log messages in system journal ..................................................... 172
11. Securing the SafeKit web service.............................................................. 175
11.1

Overview ............................................................................................... 175

11.1.1

Default setup .............................................................................................................. 176

11.1.2Predefined setups ........................................................................................................ 176

11.2

HTTP setup ............................................................................................ 177

11.2.1

Default setup .............................................................................................................. 177

11.2.2

Unsecure setup based on identical role for all .................................................................. 179

11.3

HTTPS setup .......................................................................................... 180

11.3.1HTTPS setup using the SafeKit PKI ................................................................................ 181

11.3.2HTTPS setup using an external PKI ................................................................................ 189

11.4

User authentication setup ........................................................................ 193

11.4.1File-based authentication setup ..................................................................................... 194

11.4.2

LDAP/AD authentication setup....................................................................................... 196

11.4.3

OpenID authentication setup......................................................................................... 199

12. Cluster.xml for the SafeKit cluster configuration ...................................... 203
12.1

12.1.1Cluster.xml example .................................................................................................... 203

12.1.2Cluster.xml syntax....................................................................................................... 204

12.1.3<lans>, <lan>, <node> attributes ................................................................................ 204

12.2

10

Cluster.xml file ....................................................................................... 203

SafeKit cluster Configuration .................................................................... 205

39 A2 38MC 03

High availability architectures

12.2.1

Configuration with the SafeKit web console ..................................................................... 205

12.2.2

Configuration with command line ................................................................................... 206

12.2.3

Configuration changes.................................................................................................. 207

13. Userconfig.xml for a module configuration ............................................... 209
13.1

Macro definition (<macro> tag) ................................................................ 210

13.1.1<macro> example ....................................................................................................... 210

13.1.2<macro> syntax ......................................................................................................... 210

13.1.3<macro> attributes ..................................................................................................... 210

13.2

Farm or mirror module (<service> tag) ..................................................... 210

13.2.1<service> example ...................................................................................................... 210

13.2.2<service> syntax ........................................................................................................ 211

13.2.3<service> attributes .................................................................................................... 211

13.3

Heartbeats (<heart>, <heartbeat > tags) ................................................. 213

13.3.1<heart> example ........................................................................................................ 213

13.3.2<heart> syntax ........................................................................................................... 214

13.3.3<heart>, <heartbeat > attributes ................................................................................. 214

13.4

Farm topology (<farm>, <lan> tags) ........................................................ 215

13.4.1<farm> example ......................................................................................................... 215

13.4.2<farm> syntax ........................................................................................................... 216

13.4.3<farm>, <lan> attributes............................................................................................. 216

13.5

Virtual IP address (<vip> tag) .................................................................. 217

13.5.1<vip> example in farm architecture ............................................................................... 217

13.5.2<vip> example in mirror architecture ............................................................................ 217

13.5.3

Alternative to <vip> for servers in different networks ...................................................... 217

13.5.4<vip> syntax .............................................................................................................. 218

13.5.5<vip><interface_list>, <interface>, <virtual_interface>, <real_interface>, <virtual_addr>
attributes ................................................................................................................... 219

13.5.6<loadbalancing_list>, <group>, <cluster>, <host> attributes .......................................... 222

13.5.7<vip> Load balancing description .................................................................................. 224

13.6

File replication (<rfs>, <replicated> tags) ................................................. 225

13.6.1<rfs> example ............................................................................................................ 225

13.6.2<rfs> syntax .............................................................................................................. 225

13.6.3<rfs>, <replicated> attributes ...................................................................................... 226

13.6.4<rfs> description ........................................................................................................ 234

13.7

Enable module scripts (<user>, <var> tags) ............................................. 243

13.7.1<user> example ......................................................................................................... 243

13.7.2<user> syntax ............................................................................................................ 243

13.7.3<user>, <var> attributes ............................................................................................. 243

13.8

Virtual hostname (<vhost>, <virtualhostname> tags) ................................ 244

13.8.1<vhost> example ........................................................................................................ 244

13.8.2<vhost> syntax .......................................................................................................... 244

13.8.3<vhost>, <virtualhostname> attributes ......................................................................... 244

39 A2 38MC 03

11

SafeKit User's Guide

13.8.413.9<vhost> description .................................................................................................... 245

Process or service death detection (<errd>, <proc> tags) ........................... 245

13.9.1<errd> example .......................................................................................................... 245

13.9.2<errd> syntax ............................................................................................................ 246

13.9.3<errd>, <proc> attributes ........................................................................................... 246

13.9.4<errd> commands ...................................................................................................... 250

13.10 Checkers (<check> tag) .......................................................................... 251
13.10.1<check> example ....................................................................................................... 252

13.10.2<check> syntax .......................................................................................................... 252

13.11 TCP checker (<tcp> tags) ........................................................................ 253
13.11.1<tcp> example ........................................................................................................... 253

13.11.2<tcp> syntax .............................................................................................................. 253

13.11.3<tcp> attributes ......................................................................................................... 253

13.12 Ping checker (<ping> tags) ..................................................................... 254
13.12.1<ping> example ......................................................................................................... 254

13.12.2<ping> syntax ............................................................................................................ 255

13.12.3<ping> attributes ........................................................................................................ 255

13.13 Interface checker (<intf> tags) ................................................................ 256
13.13.1<intf> example ........................................................................................................... 256

13.13.2<intf> syntax ............................................................................................................. 256

13.13.3<intf> attributes ......................................................................................................... 256

13.14 IP checker (<ip> tags) ............................................................................ 257
13.14.1<ip> example ............................................................................................................. 257

13.14.2<ip> syntax ............................................................................................................... 257

13.14.3<ip> attributes ........................................................................................................... 257

13.15 Custom checker (<custom> tags) ............................................................. 258
13.15.1<custom> example ..................................................................................................... 258

13.15.2<custom> syntax ........................................................................................................ 258

13.15.3<custom> attributes.................................................................................................... 258

13.16 Module checker (<module> tags) ............................................................. 260
13.16.1<module> example ..................................................................................................... 260

13.16.2<module> syntax ........................................................................................................ 261

13.16.3<module> attributes ................................................................................................... 261

13.17 Splitbrain checker (<splitbrain> tag)......................................................... 262
13.17.1<splitbrain> example................................................................................................... 262

13.17.2<splitbrain> syntax ..................................................................................................... 263

13.17.3<splitbrain> attributes ................................................................................................. 263

13.18 Failover machine (<failover> tag) ............................................................ 263

12

13.18.1<failover> example ..................................................................................................... 264

13.18.2<failover> syntax ........................................................................................................ 264

13.18.3<failover> attributes ................................................................................................... 264

39 A2 38MC 03

High availability architectures

13.18.4<failover> commands .................................................................................................. 264

13.18.5

Failover rules .............................................................................................................. 265

14. Scripts for a module configuration ............................................................ 267
14.1

List of scripts ......................................................................................... 267

14.1.1

Start/stop scripts......................................................................................................... 267

14.1.2Other scripts ............................................................................................................... 268

14.2

Script execution automaton ..................................................................... 269

14.3

Variables and arguments passed to scripts ................................................. 270

14.4

SafeKit special commands for scripts ......................................................... 270

14.4.1

Commands for Windows ............................................................................................... 270

14.4.2

Commands for Linux .................................................................................................... 271

15. Examples of userconfig.xml and module scripts ....................................... 273
15.1

Generic mirror module example with mirror.safe...................................... 274

15.2

Generic farm module example with farm.safe ........................................... 275

15.3

A Farm module depending on a mirror module example .............................. 277

15.4

Dedicated replication network example ...................................................... 278

15.5

Network load balancing examples in a farm module .................................... 278

15.5.1

TCP load balancing example.......................................................................................... 278

15.5.2

UDP load balancing example ......................................................................................... 279

15.5.3

Multi-group load balancing example ............................................................................... 279

15.6

Virtual hostname example with vhost.safe ............................................... 280

15.7

Software error detection example with softerrd.safe ................................ 282

15.8

TCP checker example .............................................................................. 284

15.9

Ping checker example .............................................................................. 284

15.10 Interface checker example ....................................................................... 285
15.11 IP checker example ................................................................................. 286
15.12 Custom checker example with customchecker.safe ................................... 286
15.13 Module checker example with leader.safe and follower.safe ................... 288
15.14 Mail notification example with notification.safe ..................................... 289
15.14.1

Notification on the start and the stop of the module ......................................................... 289

15.14.2

Notification on module state changes ............................................................................. 290

16. SafeKit cluster in the cloud ....................................................................... 293
16.1

SafeKit cluster in Amazon AWS ................................................................. 293

16.1.1

Mirror cluster in AWS ................................................................................................... 294

16.1.2

Farm cluster in AWS .................................................................................................... 295

16.2

SafeKit cluster in Microsoft Azure .............................................................. 297

16.2.1

Mirror cluster in Azure .................................................................................................. 297

16.2.2

Farm cluster in Azure ................................................................................................... 299

16.3

SafeKit cluster in Google GCP ................................................................... 300

39 A2 38MC 03

13

SafeKit User's Guide

16.3.1

Mirror cluster in GCP .................................................................................................... 301

16.3.2

Farm cluster in GCP ..................................................................................................... 303

17. Third-Party Software ................................................................................ 305
Log Messages Index ....................................................................................... 309
Index .............................................................................................................. 313

14

39 A2 38MC 03

1. High availability architectures
1.1 “SafeKit cluster definition” page 15
1.2 “SafeKit module definition - application integration” page 15
1.3 “Mirror module: synchronous real time file replication and failover” page 16
1.4 “Farm module: network load balancing and failover” page 18
1.5 “Combining mirror and farm modules” page 20
1.6 “The simplest high availability cluster in the cloud” page 22

1.1

SafeKit cluster definition

A SafeKit cluster is a set of servers where SafeKit is installed and running.
All servers belonging to a given SafeKit cluster share the same cluster configuration (listof servers and networks used) and communicate with each other’s to have a global viewof SafeKit modules configurations. The same server can not belong to many SafeKit
clusters.
Setting the cluster configuration is a prerequisite to SafeKit modules installation andconfiguration since the 7.2 release of SafeKit and of the web console. The cluster
configuration is set through the web console as described in section 3.2 page 39. The
web console provides the ability to administer one or more SafeKit clusters.

1.2

SafeKit module definition - application integration

A SafeKit module is associated with an application. A module is customizable by the user,
and it defines the behavior of the high availability solution for the application. Different
modules can be defined for different applications.
In practice, an application module is an easy-to-setup file that contains:a main configuration file userconfig.xml, which lists networks used forcommunication between servers, files to replicate in real time (for a mirror module),
virtual IP configuration, network load balancing criteria (for a farm module) andmore...
application stop and start scripts
SafeKit offers two types of modules detailed in this chapter:the mirror module
the farm module
Combining multiple application modules allows the implementation of advanced
architectures:
active/active: 2 mirror modules backuping each otherN-1: N mirror modules with a single backup
mixed farm and mirror: mixing network load balancing, file replication and failover

39 A2 38MC 03

15

SafeKit User's Guide

1.3
Mirror module: synchronous real time file replication andfailover1.3.1

File replication and failover

The mirror architecture is a primary-backup high-availability solution that is suitable for
all applications. The application runs on a primary server and is restarted automaticallyon a secondary server if the primary server fails.
The mirror architecture can be configured with or without file replication. With its filereplication function, this architecture is particularly suitable for providing high availability
for back-end applications with critical data to protect against failure. Indeed, thesecondary server data are highly synchronized with the primary server and the failover isdone on the secondary server from the most up-to-date data. If the application
availability is more critical than the application data synchronization, the default policy
can be relaxed by allowing a failover on the secondary server when the time elapsed
since the last synchronization is below a configurable delay.
Microsoft SQL Server.Safe, MySQL.Safe, and Oracle.Safe are examples of "mirror" typeapplication modules. You can write your own mirror module for your application, basedon the generic module Mirror.Safe.
The failover mechanism works as follows.

1.3.2

Step 1. Normal operation

For replication, only the names of file directories are configured in SafeKit. There are no
pre-requisites on the disk organization for the two servers. Directories to replicate can belocated in the system disk.
Server 1 (PRIM) runs the application.
SafeKit replicates files opened by the application. Only the changes made by theapplication in the files are replicated in real time across the network, thus limiting traffic.
Thanks to the synchronous replication of file write operations on the disks of bothservers, no data is lost in case of failure.

1.3.3

Step 2. Failover

When Server 1 fails, Server 2 takes over. SafeKit switches the cluster’s virtual IP address
and restarts the application automatically on Server 2. The application finds the files
replicated by SafeKit in the identical state they were when Server 1 failed, thanks to the16

39 A2 38MC 03

High availability architecturessynchronous replication. The application continues to run on Server 2, locally modifying
its files, which are no longer replicated to Server 1.
The switch-over time is equal to the fault-detection time (set to 30 seconds by default)
plus the application start-up time. Unlike disk replication solutions, there is no delay forremounting file systems and running recovery procedures.

1.3.4

Step 3. Failback and reintegration

Failback involves restarting Server 1 after fixing the problem that caused it to fail.
SafeKit automatically resynchronizes the files, updating only the files that were modified
on Server 2 while Server 1 was stopped.
This reintegration takes place without disturbing the applications, which can continue torun on Server 2. This is a major feature that differentiates SafeKit from other solutions,which require you to stop the applications on Server 2 to resynchronize Server 1.
To optimize file reintegration, different cases are considered:
1. The module must have completed the reintegration (on the first start of the module, itruns a full reintegration) before enabling the tracking of modification into bitmaps
2. If the module was cleanly stopped on the server, then at restart of the secondary, only
the modified zones of modified files are reintegrated, according to a set of modification
tracking bitmaps.
3. If the secondary crashed (power off) or was incorrectly stopped (exception in nfsbox
replication process), the modification bitmaps are not reliable, and are therefore
discarded. All the files bearing a modification timestamp more recent than the last known
synchronization point minus a graceful delay (typically one hour) are reintegrated.
4. A call to the special command second fullsync triggers a full reintegration of all
replicated directories on the secondary when it is restarted.
5. If files have been modified on the primary or secondary server while SafeKit was
stopped, the replicated directories are fully reintegrated on the secondary

1.3.5

Step 4. Return to normal operation

After reintegration, the files are once again in mirror mode, as in step 1. The system isback in high-availability mode, with the application running on Server 2 and SafeKit
replicating file updates to the backup Server 1.
If the administrator wants to run the application on Server 1, he/she can execute a swap
command either manually at an appropriate time, or automatically through configuration.
39 A2 38MC 03

17

SafeKit User's Guide

1.3.6
Synchronous, fault-tolerant replication that loses no data whena server fails
There is a significant difference between synchronous replication, as offered by the
SafeKit mirror solution, and asynchronous replication traditionally offered by other file
replication solutions.
With synchronous replication, when a disk IO is performed by the application or by thefile cache system on the primary server onto a replicated file, SafeKit waits for the IO
acknowledgement from the local disk and from the secondary server, before sending the
IO acknowledgement to the application or to the file system cache.
The synchronous, in real time, replication of files updated by an application eliminates
the loss of data in case of server failure. Synchronous replication ensures that any data
committed on a disk by a transactional application is also present on the secondary
server.
The bandwidth required to implement synchronous data replication is in the order ofmagnitude of a typical modern LAN, or extended LAN between two computer rooms
located a few kilometers apart.
With asynchronous replication implemented by other solutions, the IOs are placed in aqueue on the primary server but the primary server does not wait for the IO
acknowledgments of the secondary server. So, the data that did not have time to becopied across the network on the second server is lost if the first server fails. In
particular, a transactional application loses committed data in case of failure.
Asynchronous replication can be used for data replication through a low-speed WAN, toback up data remotely over more than 100 kilometers.
SafeKit provides an asynchronous solution with no data loss, ensuring the asynchrony
not on the primary machine but on the secondary one. In this solution, SafeKit always
waits for the acknowledgement of the two machines before sending theacknowledgement to the application or the system cache. But on the secondary, there
are 2 options asynchronous or synchronous. In the asynchronous case (option <rfsasync="second">), the secondary sends the acknowledgement to the primary uponreceipt of the IO and writes to disk after. In the synchronous case (<rfs async="none">),the secondary writes the IO to disk and then sends the acknowledgement to the primary.
The async="none" mode is required if we consider a simultaneous double power outageof two servers, with inability to restart the former primary server and requirement to restart on the secondary.

1.4

Farm module: network load balancing and failover

1.4.1

Network load balancing and failover

The farm architecture provides both network load balancing, through transparent
distribution of network traffic, and software and hardware failover. This architecture
provides a simple solution for increasing system load. The same application runs on eachserver, and the load is balanced by the distribution of network activity between the
different servers of the farm.
18

39 A2 38MC 03

High availability architectures

Farm architecture accommodates/implements well with front-end applications like webservices. Apache_farm.Safe and Microsoft IIS_farm.safe are examples of farm application
modules. You can make your own farm module, modified to suit your application, fromthe generic module Farm.safe.

1.4.2

Principle of a virtual IP address with network load balancing

The virtual IP address is configured locally on each server of the farm. The input traffic
for this address is split among them at low level by a filter inside each server's kernel.
The load balancing algorithm inside the filter is based on the identity of the client packets(client IP address, client TCP port). Depending on the identity of the client packet input, asingle filter instance in a server farm transmits the packet to the upper network layers;
the other filter instances in other servers drop it. Once a packet is accepted by the filter
on a server, only the CPU and memory of this server are used by the application that
responds to the request of the client. The output messages are sent directly from theapplication server to the client.
If a server fails, the SafeKit membership protocol reconfigures the filters in the farm to
re-balance the traffic on the remaining available servers.

1.4.3

Load balancing for stateful or stateless web services

With a stateful server, there is session affinity. The same client must be connected to thesame server on multiple HTTP/TCP sessions to retrieve its context from the server. In
this case, the SafeKit load balancing rule is configured on the client IP address. Thus, thesame client is always connected to the same server on multiple TCP sessions. And
different clients are distributed across different servers in the farm. This configuration isused when there is a need for session affinity.
With a stateless server, there is no session affinity. The same client can be connected todifferent servers in the farm on multiple HTTP/TCP sessions; because there is no context
stored locally on a server from one session to another. In this case, the SafeKit loadbalancing rule criteria is the TCP client session identity. This configuration is the bestsolution to distribute sessions between servers, but it can only load balance a TCP service
without session affinity.
Other load balancing algorithms are available for UDP services.

39 A2 38MC 03

19

SafeKit User's Guide

1.5

Combining mirror and farm modules

1.5.1

Active/Active: 2 mirror modules backuping each other

Two active servers mirroring each otherIn an active / active architecture, there are two servers and two mirror application
modules in mutual takeover (Appli1.Safe and Appli2.Safe). Each application server is a
backup of the other server.

If one application server fails, both applications will be active on the same physicalserver. After restart of the failed server, its application will run again on its default
primary server.
A mutual takeover cluster is a more economical solution than two separate mirror
clusters, because there is no need to invest in back-up servers that will spend most of
their time sitting idle waiting for the primary server to fail. Note that during a failure, theremaining server must be able to handle the combined workload of both applications.

1.5.2

N-to-1: N mirror modules with a single backup

Shared backup for multiple active servers
In N-to-1 architecture, there are N mirror application modules installed on N primaryservers and one backup server.

If one of the N active servers fails, the single backup server restarts the module of thefailed server. Once the problem is fixed and the failed server is restarted, the application
switches back to its original server.
In case of failure, unlike the active/active architecture, the backup server doesn't have tohandle a double workload when a primary server fails. Assuming that there is only one
failure at a time - the solution can support multiple primary server failures at the sametime, but in this case the single back-up server will have to handle the combined
workload of all the failed servers.

20

39 A2 38MC 03

High availability architectures

Mixed farm/mirror: network load balancing, file replication, failover
Network load balancing, file replication and failover
You can mix farm and mirror application modules on the same cluster of servers.
This option allows you to implement a multi-tier application architecture, such as
Apache_farm.Safe (farm architecture with load balancing and failover) and MySQL.safe(mirror architecture with file replication and failover) on the same application servers.

As a result, load balancing, file replication and failover are managed coherently on thesame servers. Specific to SafeKit, this mixed architecture is unique on the market!

39 A2 38MC 03

21

SafeKit User's Guide

1.6

The simplest high availability cluster in the cloud

SafeKit brings in the Microsoft Azure, Amazon AWS and Google clouds the simplest
solution for a high availability cluster. It can be implemented on existing virtual machines
or on a new virtual infrastructure, that you create by simply clicking on a button that
deploys and configures everything for you in Azure or AWS clouds.
For a full description, see section 16 page 293.

1.6.1

Mirror cluster in Microsoft Azure, Amazon AWS and Google GCP

SafeKit brings in the Azure, Aws and GCP clouds the simplest solution for a highavailability cluster with real-time replication and failover (mirror module).
For a quick start, refer to mirror cluster in Azure, mirror cluster in AWS or mirror cluster
in GCP.

the critical application is running on the PRIM server
users are connected to a primary/secondary virtual IP address which is configured in
the cloud load balancer
SafeKit brings a generic checker for the load balancer. On the PRIM server, thechecker returns OK to the load balancer and NOK on the SECOND serverin each server, SafeKit monitors the critical application with process checkers andcustom checkers
SafeKit automatically restarts the critical application when there is a software failure
or a hardware failure thanks to restart scripts
SafeKit makes synchronous real-time replication of files containing critical data

22

39 A2 38MC 03

High availability architectures

a connector for the SafeKit web console is installed in each server. Thus, the highavailability cluster can be managed in a quite effortless way to avoid human errors

1.6.2

Farm cluster in Microsoft Azure, Amazon AWS and Google GCP

SafeKit brings in the Azure, AWS and Google clouds the simplest solution for a highavailability cluster with load balancing and failover (farm module).
For a quick start, refer to farm cluster in Azure, farm cluster in AWS or farm cluster in
GCP.

the critical application is running in all servers of the farmusers are connected to a virtual IP address which is configured in the cloud loadbalancer
SafeKit brings a generic checker for the load balancer. When the farm module isstopped in a server, the checker returns NOK to the load balancer which stops theload balancing of requests to the server. The same behavior happens when there is a
hardware failurein each server, SafeKit monitors the critical application with process checkers andcustom checkers
SafeKit automatically restarts the critical application in a server when there is a
software failure thanks to restart scripts
a connector for the SafeKit web console is installed in each server. Thus, the loadbalancing cluster can be managed in a quite effortless way to avoid human errors.

39 A2 38MC 03

23

SafeKit User's Guide

24

39 A2 38MC 03

2. Installation
2.1 “SafeKit install” page 25
2.2 “Mirror installation recommendation” page 30
2.3 “Farm installation recommendation” page 31
2.4 “SafeKit upgrade” page 31
2.5 “SafeKit full uninstall” page 34
2.6 “SafeKit documentation” page 35

2.1

SafeKit install

2.1.1

Download the package

1.

Connect to https://support.evidian.com/safekit

2.

Go to <Version 8.2>/Platforms/<Your platform>/Current versions

3.

Download the package
In Windows, two packages are available:✓

A Windows Installer package (safekit_windows_x86_64_8_2_x_y.msi). It
depends on the VS2022 C runtime which must be previously installed✓

A standalone executable bundle (safekit_windows_x86_64_8_2_x_y.exe), which
includes the SafeKit installation and the VS2022 C runtime

Choose one or the other package depending on whether the VS2022 C runtime is
installed or not.

2.1.2

Installation directories and disk space provisioning

SafeKit is installed in:in Windows
SAFE

SAFE=C:\safekit
if %SYSTEMDRIVE%=C:

Minimum free disk space: 97MB

in Linux
SAFE=/opt/safekit
in Windows
SAFEVAR= C:\safekit\var
SAFEVAR

Minimum free disk space: 20MB + at
least 20MB (up to 3 GB) per module
for dumps

if %SYSTEMDRIVE%=C:in Linux
SAFEVAR=/var/safekit

39 A2 38MC 03

25

SafeKit User's Guide

2.1.3

Install procedure

2.1.3.1

On Windows as administrator

2.1.3.1.1

SafeKit package install

1. Log-in as administrator
2. Locate the downloaded file safekit_windows_x86_64_8_2_x_y.msi (orsafekit_windows_x86_64_8_2_x_y.exe)
3. Install in interactive mode by double-clicking it and go through the installer wizard
It is also possible to install the .msi in non-interactive mode by running in a PowerShellterminal: msiexec /qn /i safekitwindows_8_2_x_y.msi

2.1.3.1.2

Firewall setup

This step is mandatory to enable communications between SafeKit cluster nodes and withthe web console.
1. Open a PowerShell console as administrator
2. Go to the root of the SafeKit installation directory SAFE (by default
SAFE=C:\safekit if %SYSTEMDRIVE%=C:)
cd c:\safekit
3. Run .\private\bin\firewallcfg.cmd add
This configures the Microsoft firewall for SafeKit. For details or other firewalls, see section
10.3 page 158
2.1.3.1.3

Web service initialization

This step is mandatory to initialize the default configuration of the web service, which isaccessed by the web console and the global safekit command. By default, authenticationis required to access the service. The following script makes it easy to implement byinitializing it with the admin user and the given password pwd, for example.
1. Open a PowerShell console as administrator
2. Go to the root of the SafeKit installation directory SAFE (by default
SAFE=C:\safekit if %SYSTEMDRIVE%=C:)
cd c:\safekit
3. Run.\private\bin\webservercfg -passwd pwd
This then allows to access to all the web console's features, by logging in withadmin/pwd, and to run distributed commands. For details, see 11.2.1 page 177.

The password must be identical on all nodes that belong to the same
SafeKit cluster. Otherwise, web console and distributed commands will failwith authentication errors.

26

39 A2 38MC 03

Installation

On upgrade, this step can be skipped if it has already been done during theprevious install of SafeKit 8.2. If it is reapplied, it will reset the password with
the new value.

2.1.3.2

On Linux as root

2.1.3.2.1

SafeKit package install

1. Open a Shell console as root
1. Go to the directory that contains the downloaded filesafekitlinux_x86_64_8_2_x_y.bin
auto extractible zip file
3. Run chmod +x safekitlinux_x86_64_8_2_x_y.bin
4. Run./safekitlinux_8_2_x86_64_x_y.bin
it extracts the package and the safekitinstall script
5. Install in interactive mode by executing ./safekitinstall
During the installation:✓

reply to “Do you accept that SafeKit automatically configure thelocal firewall to open these ports (yes|no)?”If you answer yes, it configures firewalld or iptable Linux firewall for
SafeKit. For details or other firewalls, see section 10.3 page 158.

✓

reply to “Please enter a password or "no" if you want to set it
later”
This step is mandatory to initialize the default configuration of the webservice. The web service requires authentication to access the service.
It initializes it with the admin user and the given password pwd, forinstance. It then allows to access to all the web console's features, bylogging in with admin/pwd, and run distributed commands. For details, see
11.2.1 page 177.
The password must be identical on all nodes that belong to thesame SafeKit cluster. Otherwise, web console and distributedcommands will fail with authentication errors.

or
5. Install in non-interactive mode, by executing:./safekitinstall -q
Use the option -nofirewall for disabling the firewall automatic setup
Use the option -passwd pwd for initializing the web service authentication (where
pwd is the password set for the admin user)

2.1.3.2.2

Firewall setup

No action required when firewall automatic configuration has been performed during
install. Otherwise see section 10.3 page 158.

39 A2 38MC 03

27

SafeKit User's Guide

2.1.3.2.3

Web service initialization

This step is mandatory to initialize the default configuration of the web service, which isaccessed by the web console and the global safekit command. The web service requires
authentication to access the service. No action required when the web service
initialization has been performed during install. Otherwise, see section 11.2.1 page 177.

2.1.4

Use the SafeKit console or command line interface

Once installed, the SafeKit cluster must be defined. Then modules can be installed,
configured, and administered. All these actions can be done with the SafeKit console or
the command line interface.
2.1.4.1

The SafeKit console

1.

Start a web browser (Microsoft Edge, Firefox, or Chrome)2.

Connect it to the URL http://host:9010 (where host is the name or IP addressof one of the SafeKit nodes)

3.

In the login page, enter admin as user’s name and the password you gave on
initialization (e.g., pwd)

4.

Once the console is loaded, the admin user can access to

Monitoring andConfiguration in the navigation sidebar, as he has the default Admin role
For details see section 3 page 37.

2.1.4.2

The SafeKit command line interface

It is based on the single safekit command located at the root of the SafeKit installation
directory. Almost all safekit commands can be applied locally or on a list of nodes in the
SafeKit cluster. This is called global or distributed command.
To use the safekit command:
1. Open a PowerShell console as administrator

In Windows

2. Go to the root of the SafeKit installation directory SAFE (bydefault SAFE=C:\safekit if %SYSTEMDRIVE%=C:)
cd c:\safekit
3. Run .\safekit.exe <arguments>
1. Open a Shell console as root

In Linux

2. Go to the root of the SafeKit installation directory SAFE (bydefault SAFE=/opt/safekit)
cd /opt/safekit
3. Run ./safekit <arguments>

For details, see section 9 page 141.

28

39 A2 38MC 03

Installation

2.1.5

SafeKit license keys

If you do not install any license keys, the product will stop every 3 days
You can download a one-month trial key (which is accepted on any hostname/anyOS) from the following address: http://www.evidian.com/safekit/requestevalkey.php
To obtain permanent keys see section 8.2 page 134
Save the key into the SAFE/conf/license.txt file (or any other file in SAFE/conf) on
each server
If files in SAFE/conf contain more than one license keys the most favorable key will
be chosen
Check the key conformance with the command safekit level

2.1.6

System specific procedures and characteristics

2.1.6.1

Windows

Apply a special procedure to properly stop SafeKit modules at machine shutdown andto start safeadmin service at boot: see section 10.4 page 163.
For network interfaces with teaming and with SafeKit load balancing, it is necessary
to uncheck "Vip" on physical network interfaces of teaming and keep it checked only
on teaming virtual interface.
2.1.6.2

Linux

In Linux, the SafeKit package depends on other system packages. Most of them areinstalled automatically, except those specific to the implementation of load-balancing
in a farm and file replication in a mirror.
For an updated list of required packages, see the SafeKit Release Notes.
The user safekit and a group safekit are created: all users belonging to thesafekit group, and the user root can execute SafeKit commands
In a farm module with load balancing on a virtual IP address, the vip kernel module iscompiled when the module is configured. To compile successfully, Linux packages
must be installed, as well as the devel package corresponding to the kernel version
installed (kernel-devel).
For a farm with SafeKit load balancing on a bonding interface, no ARP should be set in
the bonding configuration. Otherwise the association <virtual IP address, invisible
virtual MAC address> is broken in client ARP caches with physical MAC address of thebonding interface: see section 4.3.4 page 80
For a mirror, if using file replication, install nfs-util package and remove thelogwatch package (rpm -e logwatch); otherwise NFS service and SafeKit are
stopped every night

39 A2 38MC 03

29

SafeKit User's Guide

2.2

Mirror installation recommendation

ip 1.1

ip 1.2

virtual ip =ip 1.10

mirror(app1)=

app1
dir1

2.2.1

dir1

Hardware prerequisites

2 servers with the same Operating System
Supported OS: https://support.evidian.com/supported_versions/#safekit
Disk drive with write-back cache recommended for the performance of the IOs

2.2.2

Network prerequisites

1 physical IP address per server (ip 1.1 and ip 1.2)
If you need to set a virtual IP address (ip 1.10), both servers must be in the same IP
network with the standard SafeKit configuration (LAN or extended LAN between two
remote computer rooms). For setting a virtual IP address with servers in different IP
networks, see section 13.5.3 page 217.

2.2.3

Application prerequisites

The application is installed and starts on both servers
Application can be started and stopped using command lines
On Linux, command lines like service "service" start|stop or su –user "applicmd"
On Windows, command lines like net start|stop "service"
If necessary, application with a procedure to recover after crash
Remove automatic application start at boot and configure the boot start of themodule instead

2.2.4

File replication prerequisites

File directories that will be replicated are created on both servers
They are located at the same place on both servers in the file tree
It is better to synchronize clocks of both server for file replication (NTP protocol)
On Linux, align uids/gids on both servers for owners of replicated directories/files
See also system specific procedures and characteristics in section 2.1.6 page 2930

39 A2 38MC 03

Installation

2.3

ip 1.1

2.3.1

Farm installation recommendation

ip 1.2

ip 1.3virtual IP =ip 1.20

ip 1.20

ip 1.20

farm (app2) =

app2

app2

app2

Hardware prerequisites

At least 2 servers with the same Operating System
Supported OS: https://support.evidian.com/supported_versions/#safekit
Linux: kernel compilation tools installed for vip kernel module

2.3.2

Network prerequisites

1 physical IP address per server (ip 1.1, ip 1.2, ip 1.3)
If you need to set a virtual IP address (ip 1.20), servers must be in the same IP
network with the standard SafeKit configuration (same LAN or extended LAN between
remote computer rooms). For setting a virtual IP address with servers in different IP
networks, see section 13.5.3 page 217.
See also system specific procedures and characteristics in section 2.1.6 page 29

2.3.3

Application prerequisites

The same prerequisites as for a mirror module described in section 2.2.3 page 30

2.4

SafeKit upgrade

2.4.1

When proceed to an upgrade?

If you encounter a problem with SafeKit, see the Software Release Bulletin containing thelist of fixes on the product.
If you want to take advantage of some new features, see the SafeKit Release Notes. This
document also tells you if you are in the case of a major upgrade (ex. 7.5 to 8.2) which
requires a different procedure from the one presented here.
The upgrade procedure consists in uninstalling the old package and then installing thenew package. All nodes in the same cluster must be upgraded.

2.4.2

Prepare the upgrade

1. Note the state "on" or "off" of SafeKit services and modules started automatically atboot safekit boot webstatus; safekit boot status -m AM (where AM is thename of the module) and in Windows: safekit boot snmpstatus;
The start at boot of the module can be defined in its configuration file. If
so, the use of the safekit boot command becomes unnecessary.

39 A2 38MC 03

31

SafeKit User's Guide

2. for a mirror module
note the server in the ALONE or PRIM status to know which server holds the up-todate replicated files
3. optionally, take snapshots of modules
Uninstalling/reinstalling will reset logs and dumps of each module. If you want tokeep this information (logs and last 3 dumps and configurations), run the commandsafekit snapshot –m AM /path/snapshot_xx.zip (replace AM by the module
name)

2.4.3

Uninstall procedure

On Windows as administrator and on Linux as root:
1. stop all modules using the command safekit shutdown
For a mirror in the PRIM-SECOND status, stop first the SECOND server to avoid an
unnecessary failover
2. close all editors, file explorers, shells, or terminal under SAFE and SAFEVAR (to avoid
package uninstallation error)
3. uninstall SafeKit package
In Windows

Use the Control Panel-Add/Remove Programs applet

In Linux

Use the command safekit uninstall

4. undo all configurations that you have done manually for the firewall setup (seesection 10.3 page 158)
Uninstalling SafeKit includes creating a backup of the installed modules in
SAFE/Application_Modules/backup, then unconfiguring them.

2.4.4

Reinstall and postinstall procedure

1. Install the new package as described in section 2.1 page 25
2. Check with the command safekit level the installed SafeKit version and the validityof the license (which has not been uninstalled)
If you have a problem with the new package and the old key, take a temporary
license: see section 2.1.5 page 29
3. If you use the web console, clear the browser cache and refresh pages in the webbrowser
4. Since SafeKit 8.2.1, previously configured modules are automatically reconfigured on
upgrade.
However, you may still need to reconfigure module to apply any configuration
changes coming with the new version (see the SafeKit Release Notes). Reconfigure
the module either with:32✓

the web console by navigating to
Configure the module/

Configuration/Modules configuration/✓

the web console by directly entering the URL
http://host:9010/console/en/configuration/modules/AM/config/
39 A2 38MC 03

Installation✓

the command safekit config –m AM

where AM is the module name
5. If necessary, reconfigure the automatic start of modules at bootThe start at boot of the module can be defined in its configuration file. If so, skip thisstep. Otherwise, run the command safekit boot –m AM on (replace AM by themodule name)
6. Restart the modules
The module must be started as primary on the node with theupdated replicated files (former PRIM or ALONE) either with:✓the web console by navigating tonode/Force start/As primary

Monitoring/of the✓the command safekit prim –m AM (replace AM by themodule name)

Check that the application is working properly once the module isin ALONE state, before starting the other node.
MirrormoduleOn the other node (former SECOND), the module must be started
in secondary mode either with:✓the web console by navigating tonode/Force start/As secondary

Monitoring/of the✓the command safekit second –m AM (replace AM by themodule name)Once this initial start has been performed by selecting the primaryand secondary nodes, subsequent starts can be performed with:✓the web console by navigating to
node/

✓

Monitoring/of theStart/

the command safekit start –m AM (replace AM by themodule name)Start the module either with:✓the web console by navigating tomodule/Farm module✓

Monitoring/of theStart/

the command safekit start –m AM (replace AM by themodule name)

Furthermore, in exceptional cases where you have modified the default setup of the
SafeKit web service or SNMP monitoring :the SafeKit web service safewebserver✓

If its automatic start at boot had been disabled, disable it again with thecommand safekit boot weboff✓

If you had modified configuration files and these have evolved in the new version,
your modifications are saved into SAFE/web/conf before being overwritten by the

39 A2 38MC 03

33

SafeKit User's Guide

new version. Carrying over your old configuration to the new version may require
some adaptations. For details on the default setup and all predefined setups, seesection 11 page 175.
For HTTPS and login/password configurations, certificates, and user.conf /group.conf generated for the previous release should be compatible.

The SafeKit SNMP monitoring✓

In Windows, if its automatic start at boot had been enabled, enable it again withthe command safekit boot snmpon✓

If you had modified configuration files and these have evolved in the new version,
your modifications are saved into SAFE/snmp/conf before being overwritten by
the new version. Carrying over your old configuration to the new version may
require some adaptations. For details, see section 10.8 page 170.

2.5

SafeKit full uninstall

For completely removing the SafeKit package, follow the procedure described below.

2.5.1

On Windows as administrator

1. stop all modules using the command safekit shutdown
2. close all editors, file explorers, shells, or cmd under SAFE and SAFEVAR (to avoid
package uninstallation error)(SAFE=C:\safekit if %SYSTEMDRIVE%=C: ; SAFEVAR=C:\safekit\var if%SYSTEMDRIVE%=C:)
3. uninstall SafeKit using the Control Panel-Add/Remove Programs applet
4. reboot the server
5. delete the folder SAFE that is the installation directory of the previous install of
SafeKit
6. undo all configurations that you have done for SafeKit boot/shutdown (see section
10.4 page 163)
7. undo all configurations that you have done for firewalls rules setting (see section 10.3
page 158)

2.5.2

On Linux as root

1. stop all modules using the command safekit shutdown
2. close all editors, file explorers, shells, or terminal under SAFE and SAFEVAR(SAFE=/opt/safekit ; SAFEVAR=/var/safekit)
3. uninstall SafeKit using the safekit uninstall –all command and answer yes when
prompted to delete all SafeKit folders
4. reboot the server
5. undo all configurations that you have done for firewalls rules setting
See section 10.3 page 158
6. delete the user/group created by the previous install (default is safekit/safekit)
with the commands:
34

39 A2 38MC 03

Installation

userdel safekit
groupdel safekit

2.6

SafeKit documentation

SafeKit Solution

The SafeKit solution is fully described.

SafeKit Training

Refer to this online training for a quick start in using
SafeKit.
It presents:

SafeKit Release Notes

Software Release Bulletin✓latest install instructions✓

major changes✓

restrictions and known problems✓

migration instructions

Bulletin listing SafeKit 8.2 packages, with descriptions ofchanges and fixed issues.
List of known SafeKit issues and restrictions.

SafeKit Knowledge Base

SafeKit user's guide

39 A2 38MC 03

Other KBs are available on the Evidian support site, but
are only accessible to registered users. For more details
on the support site, see section 8 page 133.
This is the guide. Please refer to the guide corresponding
to your SafeKit version number. It is delivered with the
SafeKit package and can be accessed via the web
console under / User’s guide.
The link opposite takes you to the latest version of thisguide.

35

SafeKit User's Guide

36

39 A2 38MC 03

3. The SafeKit web console
3.1 “Start the web console” page 37
3.2 “Configure the Cluster” page 39
3.3 “Configure a module” page 44
3.4 “Monitor a module” page 53
3.5 “Snapshots of module for support” page 65
3.6 “Secure access to the web console” page 66
The SafeKit 8 web console and API have evolved from previous versions. As a result, theconsole delivered with SafeKit 8 can only administer SafeKit 8 servers, which cannot be
administered with an older console.
See the Release Notes, at https://support.evidian.com/safekit, for
restrictions and known problems with the SafeKit web console.

3.1

Start the web console

The web console permits to administer one SafeKit cluster. A SafeKit cluster is a set ofservers where SafeKit is installed and running. All servers belonging to a given SafeKit
cluster share the same cluster configuration (list of servers and networks used) andcommunicate with each other’s to have a global view of SafeKit modules configurations.
The same server can not belong to many SafeKit clusters.

3.1.1

Start a web browser

The web browser runs on any allowed SafeKit nodes or workstation that can reach the
SafeKit servers over the network.
Network, firewall and proxy configuration must allow access to all the servers that areadministered with the web console
JavaScript must be available and enabled in the web browser
Tested browsers are Microsoft Edge, Firefox, and Google Chrome
To avoid security popups in Microsoft Edge, you may add the SafeKit servers
addresses into the Intranet or Trusted zone
The messages in the web console are displayed in French or English languages,
according to the selected language into the console
After SafeKit upgrade, you must clear the browser’s cache to get the new web
console pages. A quick way to do this is a keyboard shortcut:
1. Open the browser to any web page and hold CTRL and SHIFT while tapping the
DELETE key
2. A dialog box will open to clear the browser. Set it to clear everything and click
Clear Now or Delete at the bottom

39 A2 38MC 03

37

SafeKit User's Guide

3. Close the browser, stop all background processes that may be still running andre-open it fresh to reload the web console

3.1.2

Connect to a SafeKit server

By default, access to the web console requires the user to authenticate himself with aname and password. On SafeKit install, you had to initialize it with the user admin andassign a password. This admin name and password are sufficient to access all the
console's features. For more details on this configuration, see 11.2.1 page 177.
1.

Start a web browser (Microsoft Edge, Firefox, or Chrome)2.

Connect it to the URL http://host:9010 (where host is the name or IP address of oneof the SafeKit servers). If HTTPS is configured, there is an automatic redirection tohttps://host:9453.
The SafeKit server to which the console is connected (host in the URL) is called theconnection node. This node acts as a proxy to communicate on behalf of theconsole with all other SafeKit servers.
You can connect to any node of the cluster since the console offer global
view and actions. On connection error with one node, connect to another
node.

3.

In the login page, enter admin as user’s name and the password you gave on
initialization (e.g., pwd).

4.

The SafeKit web console is loaded•

When the console is connected to a SafeKit server on which the cluster isconfigured, the name of the node corresponding to the server (as defined in thecluster configuration) is displayed in the header. This is the connection node
(node1 in the example).
If the cluster is not yet configured, no name is displayed.

•

38(1) Click on
to open the menu to read the SafeKit User’s Guide, select thelanguage, enable/disable the dark mode and logout.

39 A2 38MC 03

The SafeKit web console•(2) Click on

to collapse or expand the navigation sidebar.

•(3) Click on
Configuration to configure the cluster and the modules.
Configuration is only authorized to users that have Admin role. By default, theadmin user has the Admin role.

•(4) Click on
Monitoring to monitor and control the configured modules.
Monitoring is authorized to users that have Admin, Control and Monitor roles.
With Monitor role, actions on modules (start, stop…) are prohibited.
The web console offers contextual help by clicking on the

3.2

icon.

Configure the Cluster

The SafeKit cluster must be defined before installing, configuring, or starting a SafeKit
module. A Safekit cluster is defined by a set of networks and the addresses, on these
networks, of a group of SafeKit servers, named nodes. These nodes implement one ormore modules. Each server is not necessarily connected to all the networks, but at leastone.
The cluster configuration is saved on the servers’ side into the cluster.xml file (seesection 12 page 203). For a correct behavior, it is required to apply the same cluster
configuration on all the nodes.
You must fully define the cluster configuration before installing and configuring
modules since the modification of the cluster can affect the configuration or the
execution of installed modules.
The cluster configuration home page is available :✓Directly via the URL http://host:9010/console/en/configuration/clusterOr✓

By navigating the console via

Configuration/Cluster configuration

If the cluster is not yet configured, the cluster configuration wizard is automatically
opened.

3.2.1

Cluster configuration wizard

Open the configuration wizard:✓

Directly via the URL http://host:9010/console/en/configuration/cluster/config

Or✓

Navigate in the console via
Configure the cluster/

39 A2 38MC 03

Configuration/Cluster configuration/

39

SafeKit User's Guide

The cluster configuration wizard is a step-by-step form:

1. Edit cluster configuration described page 40
2. Check result described page 42
3.

to Exit cluster configuration wizard

3.2.1.1•

40

Edit cluster configuration(1) Fill in the form to first assign a user-friendly name for the lan. This name isused for configuring heartbeat networks used by a module.

39 A2 38MC 03

The SafeKit web console

Click oncluster.

to add another node/lan or on

to remove the node/lan from theWhen a node/lan is removed from the cluster, all modules using it
in its configuration may become unusable.

•(2) Fill in the IP address of the node and then press the Tab key to check theserver connectivity and automatically insert the server hostname
The icon next to the address reflects the reachability of the node.

means that the SafeKit server is available. The tooltip gives
information on the server.

means that there was no reply from the server within the timeout
delay. Fix the problem to be able to administer this node. It may be a
bad address, a network or host failure, a bad configuration of the webbrowser or the firewall, the stop of the SafeKit web service on the node.
For solving the problem, refer to the section 7.1 page 111.
•

Change the node name if necessary. This name is the one that will be used by the
SafeKit administration service for uniquely identifying a SafeKit node. It is also theone displayed into the SafeKit web console.

•(3) If you prefer, click on Advanced configuration to switch to XML cluster editing.
Click onto open the SafeKit User’s Guide on the configuration description in thecluster.xml file.

•

Click on Reload to discard your current modifications and reload the original
configuration.

•(4) Once the edition is completed, click on Save and Apply to save and apply theedited configuration to all nodes in the cluster.

39 A2 38MC 03

41

SafeKit User's Guide

If required, you can reapply the configuration to all nodes without
modifying it.

3.2.1.2••

Check result(1) Read the result of the operation on each node:✓

Successmeans the configuration was successful.

✓

Failuremeans the configuration has failed. Click to read the output ofcommands executed on the node and search for the error. You may need tomodify the parameters entered or connect to the node to correct the problem.
Once the error has been corrected, Save and apply again.

(2) Click on Configure modules to exit the cluster configuration wizard and navigate
to modules configuration.

Or•(3) Click on
to exit the cluster configuration wizard and navigate to the cluster
configuration home page.

3.2.2

Cluster configuration home page

When the cluster is configured, the cluster configuration home page is available.
Open it:✓Directly via the URL http://host:9010/console/en/configuration/cluster

Or
✓

42

By navigating the console via

Configuration/Cluster configuration

39 A2 38MC 03

The SafeKit web console

In this example, the console is loaded from 10.0.0.107, which corresponds to node1 inthe existing cluster. This is the connection node.

•(1) Click on

Configuration in the navigation sidebar•(2) Click on Cluster configuration tab•

Nodes configured in the cluster are listed with their configuration date.

•(3) Click on
to display details about the node (names of lans and addresses
defined in the cluster configuration…).

•(4) Click on one of the buttons:✓

to modify the cluster configuration and/or re-apply it. This opens thecluster configuration wizard and loads the cluster configuration from theconnection node.

✓

to download the cluster configuration in XML format from the connection
node.

✓

to unconfigure the cluster on one or more nodes.

39 A2 38MC 03

43

SafeKit User's Guide

3.3

Configure a module

Once the cluster has been set up, you can configure a new module on the cluster. The
module configuration home page is accessible :✓Directly via the URL http://host:9010/console/en/configuration/modules

Or✓

By navigating the console via

Configuration/Modules configuration

If no module has been configured, the console automatically presents the page for
configuring a New module.

3.3.1

Select the new module to configure

In this example, the console is loaded from 10.0.0.107, which corresponds to node1 inthe existing cluster. This is the connection node.

•(1) Click on•(2) Click on Modules configuration tab•(3) Click on New Module

44

Configuration in the navigation sidebar

39 A2 38MC 03

The SafeKit web console•

The page proposes to select a new module among several proposals visible byclicking on :✓The Main modules, including the generic mirror.safe and farm.safe
modules for integrating a new application into a mirror or farm architecture.
Here are the modules stored on the connection node, node1, under
SAFE/Application_Modules/generic, SAFE/Application_Modules/demo
and SAFE/Application_Modules/published.

✓

Backup modules archived on the connection node, which are saved when amodule is uninstalled on this node.
They are loaded from node1 under SAFE/Application_Modules/backup.

✓

Other modules which are examples of SafeKit features used in modules
supplied for testing purposes only.
They are loaded from node1 under SAFE/Application_Modules/other.

✓

A locally stored module accessible from Upload module.

•(4) Select a module to configure from those listed above. In the example,
mirror.safe.

•(5) Click on the button•

A dialog opens to give the new module name•(6) Enter the name of the new module.

•(7) Click on Confirm•

The module configuration wizard is opened. This is described below.

3.3.2

Configure the new module.

Module configuration wizard

The module configuration wizard is a step-by-step form:

1.

Edit module configuration described page 46

2.

Edit module scripts (Optional) described page 47

3.

Enable communication encryption (Optional) described page 48

39 A2 38MC 03

45

SafeKit User's Guide

4.

Save and apply described page 48

5.

Check result described page 50

6.

to Exit module configuration wizard

Note that module reconfiguration can only be applied to nodes on which the module inquestion is not started. Therefore, stop the module before starting the configuration
wizard.

3.3.2.1

Edit module configuration

Below is an example of editing the mirror.safe module configuration.

•(1) Fill in the form to assign values to the various components, add or remove
them. Click on
to open the detailed panel for each component.
This form is used to enter only the main module configuration parameters.

46

39 A2 38MC 03

The SafeKit web console

The names of the Heartbeat networks proposed are the namesof the lans entered during cluster configuration.

•(2) For advanced module configuration, exhaustive compared to the form, click on
Advanced configuration. This switches to editing the module configuration file in
XML format, userconfig.xml.
Click onto open the SafeKit User’s Guide describing the configuration of thevarious components in the userconfig.xml file.

•If necessary, click on Reload to discard your modifications and reload the complete
original configuration (including scripts if these were modified in the next step).

•(3) Once you have finished editing the module configuration, click on Next step.

3.3.2.2

Edit module scripts

Below is an example of editing the mirror.safe module scripts.

•(1) Click on start_prim or stop_prim to edit it and insert your application
start/stop.
Click on to copy the content and edit it with your favorite syntax editor. Once
done, paste the modified content into the input field with .

•(2) If necessary, click on Advanced configuration to list the other module’s scripts
and edit them (prestart, poststop, scripts for checkers…).

39 A2 38MC 03

47

SafeKit User's Guide•

Click on•If necessary, click on Reload to discard your modifications and reload the complete
original configuration (including the module configuration if it was modified in theprevious step).

•(3) Once you have finished editing the module scripts, click on Next step.

3.3.2.3

to open the SafeKit User’s Guide describing the module scripts.

Enable communication encryption

Encryption of internal module communications between cluster nodes is enabled bydefault. For details, see section 10.5 page 164.

•

(1) Click Enable to enable or disable encryption of module communications.
When the module's encryption key is not identical on all nodes,
internal communication is impossible. The configuration must
be reapplied to all nodes to propagate the same key.
To generate new encryption keys, you need to:
1. disable encryption, then Save and apply configuration to all nodes
2. enable encryption, then Save and apply configuration to all nodes•

If necessary, click on Reload to discard your modifications and reload the complete
original configuration (including the module configuration and scripts if these weremodified in the previous steps).

•(2) Once this step is complete, click on Next step.

3.3.2.4

Save and apply

Step to select the nodes affected by the configuration.

48

39 A2 38MC 03

The SafeKit web console•(1) Check/uncheck to select/unselect nodes. Please note that the connection node
(node1 in the example) is mandatory.
There are 2 cases where Save and Apply is disabled:

The module on the selected node is started and, in a state, other than
STOP (NotReady).

There was no reply from the node within the timeout delay. It may be a
bad address, a network or host failure, a bad configuration of the webbrowser or the firewall, the stop of the SafeKit web service on the node.
For solving the problem, refer to the section 7.1 page 111..
In both cases, uncheck the node or click on Save and check to apply it later, after
stopping the module or solving the communication problem.
•(2) Click on Save and check to save the edited configuration on the connection
node and check its consistency. It then proceeds to the next step to display the
result of this operation.
Once this operation has been completed, any changes are saved on the connection
node. The configuration wizard can be exited and relaunched later to apply the
saved configuration. Until the saved configuration is applied, the last applied
configuration of the module remains active.

39 A2 38MC 03

49

SafeKit User's Guide•(3) Click on Save and apply to save and apply the edited configuration on selected
nodes. It then proceeds to the next step to display the result of this operation.
If this operation is successful, the applied configuration becomes the active one forthe module.

On the server side, the module configuration is saved under
SAFE/modules/AM(where AM is the module name). When reconfiguring a
module, this directory is deleted and overwritten with the changes madein the console. Thus, on the servers’ side, you must close all editors, file
explorers, shells or cmd under SAFE/modules/AM before applying theconfiguration (otherwise there is a risk that the apply fails).
3.3.2.5

Check result

The example below shows the result of the Save and Apply operation. The layout for
Save and Verify is similar••(1) Read the result of the operation on each node:✓

Successmeans the operation was successful.

✓

Failuremeans the operation has failed. Click to read the output of commands
executed on the node and search for the error. You may need to modify theparameters entered or connect to the node to correct the problem. Once theerror has been corrected, repeat the operation from the previous step.

(2) Click on Monitor modules to exit the module configuration wizard and navigate
to modules monitoring.

Or
•

50

(3) Click on
to exit the module configuration wizard and navigate to the modules
configuration home page.

39 A2 38MC 03

The SafeKit web console

3.3.3

Modules configuration home page

Once the first module has been configured, the module configuration home page isavailable. It allows you to view the modules installed on the cluster and to access theconfiguration of a new module.
Open it:✓Directly via the URL http://host:9010/console/en/configuration/modules

Or✓

By navigating the console via

Configuration/Modules configuration

In this example, the console is loaded from 10.0.0.107, which corresponds to node1 inthe existing cluster. This is the connection node.

•(1) Click on•(2) Click on Modules configuration tab.

•Modules installed on the cluster are listed with the date the configuration wasapplied and, if applicable, the date the configuration was saved but not yet applied.

•(3) Click on one of the buttons associated with the module:

39 A2 38MC 03

Configuration in the navigation sidebar.

51

SafeKit User's Guide✓

to modify its configuration or reapply its current configuration. This opens
the module configuration wizard and loads its current configuration from theconnection node.

✓

to download the .safe, consisting of all module files (userconfig.xml,
scripts) from the connection node.

✓

to reconfigure the module from the contents of a locally stored .safe.

✓

to restore a previous module configuration.
SafeKit keeps a copy of the last three successful configurations (stored
under SAFE/modules/lastconfig on the server side). All module
configuration files are packaged in a .safe file, whose name is of the type of
AM_<date>_<time> (where AM is the module name).

✓✓

to remove internal files for the module on one or more nodes, without
uninstalling it. The user configuration files are kept for later re-application.
to completely uninstall the module on one or more nodes.
All module configuration files are packaged in a .safe file, which is archived
on the server side under SAFE/Application_Modules/backup.
Before each reconfiguration, deconfiguration and uninstallation, on eachnode, close all editors, file explorer, shells or cmd under SAFE/modules/AM(or risk the operation failing).

•

To configure a new module, click on New module.

3.3.4

Add module scripts

You may need to add module scripts, such as custom checkers, to your current
configuration of the module.
In this example, a script is added to mirror module.

52

39 A2 38MC 03

The SafeKit web console•(1) Click on•(2) Click on Modules configuration tab.

•(3) Click on•(4) Edit the mirror.safe that is a zip file to add your module script files into bin
directory (checker.ps1 in the example).

•(5) Upload the modified mirror.safe (.zip extension is also accepted).

•(6) Click on•

The module configuration wizard is launched with the contents of this file. The new
scripts are visible with the Advanced configuration in step 2. Got to step 4 to Saveand apply this new configuration.

3.4

Configuration in the navigation sidebar.
to download the mirror.safe on your workstation.

to select the file to be uploaded then Confirm.

Monitor a module

Once a module is configured, you can monitor its state and run actions on it (start,
stop…).
The modules monitoring home page is accessible :✓Directly via http://host:9010/console/en/monitoringOr✓

By navigating the console via

Monitoring

In this example, the console is loaded from 10.0.0.107, which corresponds to node1 inthe existing cluster. This is the connection node. Two modules are configured: farmand mirror.

39 A2 38MC 03

53

SafeKit User's Guide•(1) Click on

Monitoring in the navigation sidebar

For each installed module, it displays:✓

the module name and nodes name on which it is installed✓

the module state and status on the node✓a notification on state change if the user has allowed them, and the URL is httpsor http://localhost

For a description, see 3.4.1 page 55.
•(2) Click on
to open the menu of global actions (start, stop…) on the module that
apply on all nodes (node1, node2 in the example).

•(3) Click on
to open menu of actions (start, stop…) on the module that applies
only to the node (node1 in the example).
For a description, see 3.4.2 page 56.

•(4) Click on the node panel (mirror>node1 in the example) to open details for themodule on this node (logs, resources…). Since SafeKit 8.2.2, Click instead on
toopen/close the details.
For a description, see 3.4.3 page 58.

•(5) Click on
to open/close the module states timeline on all nodes where it isinstalled. Available since SafeKit 8.2.2.
For a description, see 3.4.4 page 64.

54

39 A2 38MC 03

The SafeKit web console

3.4.1

Module state and status

The module state on one node is one of the following.
The module is installed but not configured:

The node is not responding:Fix the problem to be able to administer the module on this node. It
may be a bad address, a network or host failure, a bad configuration of
the web browser or the firewall, the stop of the SafeKit web service on
the node. For solving the problem, refer to the section 7.1 page 111.
This may also be due to the temporary unavailability of the connection
node. In this case, reload the console from another SafeKit node.
The module is configured, and the node is responding:
STOP

stopped (ready for starting)WAIT

waiting for one resource

ALONE

primary without secondary (mirror module)

PRIM

primary with secondary (mirror module)

SECONDsecondary with primary (mirror module)UP

active (farm module)

With the associated icon/color that means:or

NotReady

blocked stateTransienttransiting state

Readystable state

For details on state changes of a mirror module, see section 5.2 page97.
For details on state changes of a farm module, see section 6.2 page108.

39 A2 38MC 03

55

SafeKit User's Guide

The module status is one of the following.
For a mirror module, it displays the status of replicated directories:
uptodate or not uptodate.
In the special degraded case (see 7.6 page 117), it displays:

For a farm module, it displays the current network load share on thevirtual IP: 0%, 50% or 100% (for 2 nodes).
When the module (farm or mirror) is in state
WAIT (NotReady), thereason is displayed, usually the name of the failover rule that blocks themodule until the associated resource comes back from down to up. For
details, see 7.9 page 119.

In the example above, the module is blocked by the failover rule namedc_checkfile. To analyze the problem, read the logs and resources
states as described later.
When the node is not responding, the status is connection error.

3.4.2

Module control menus

Control a mirror module
In the example, the module mirror is configured on node1 and node2.

56

39 A2 38MC 03

The SafeKit web console•

Click onto open the menu of actions on node1.

•

Use Force start when you need to decide which node should start
primary or secondary.

•For instance, on the 1st start of a mirror module, you must Forcestart/As primary the node which has the up-to-date replicatedfolders.

•

For subsequent starts, click onmost up-to-date node.

•

Click on Debug to download module logs or snapshots from asingle node, or from all nodes.

Start, as SafeKit retains theRefer to sections listed below:✓ For the first start-up of a mirror module, see section 5.3 page 98✓

For the start-up of a mirror module with the up-to-date data, seesection 5.5 page 100✓

To continue the tests, see 4 Tests page 69✓

To understand and check the correct behavior of a mirror module,see section 5 page 95

39 A2 38MC 03

57

SafeKit User's Guide

Control a farm module
In the example, the module farm is configured on node1 and node2.

•(1) Click on

to open the global menu of actions, applied .

•(2) Click on

Start to start the module node1 and node2.

•(3) Click on

to open the menu and run actions only on node2.

•

Click on Debug to download module logs or snapshots from asingle node, or from all nodes.

Refer to sections listed below:✓ To continue the tests, see 4 Tests page 69
✓

3.4.3

To understand and check the correct behavior of a farm module, seesection 6 page 107

Module details

You can display details for a module on one node:✓

Directly via the URL http://host:9010/console/en/monitoring/modules/AM/nodes/node (replace node by the node name and AM by the module
name)

Or
✓

By navigating the console via

Monitoring/Click on

for the module>node

The selected module>node is highlighted with a blue color.

58

39 A2 38MC 03

The SafeKit web console

In the example, the detail for the module mirror on node1 is displayed.

•(1) Click on(mirror>node1 in the example) to open/close details for themodule on this node (logs, resources…).

•(2) Click on Logs tab to visualize the module logs.

•(3) Click on Resources tab to visualize the module resources.

•(4) Click on Information tab to visualize information on the node (SafeKit
version and license…).

3.4.3.1

Module log and scripts log

You can display logs of a module on one node:✓

Directly via the URL http://host:9010/console/en/monitoring/modules/AM/nodes/node/logs (replace node by the node name and AM by the module
name)

Or
✓

By navigating the console via

39 A2 38MC 03

Monitoring/Click on the module>node/Logs tab

59

SafeKit User's Guide

The left panel displays the non verbose module log for the selected module>node.

•(1) Click on

to resume/suspend the view in real time of the module log.

•

(2) Click on

to download the module log (verbose or not verbose).

•(3) Select the message type to view:

C(ritical) messages such as error detection
E(vent) messages such as local and remote states
U(ser) messages when the user run action on themoduleS(cript) messages when module scripts are executed•

60

Click on a message to display the verbose module log or the script log (output
of scripts) into the log detail into the right panel.

39 A2 38MC 03

The SafeKit web console

To display the script log, click on the
view.

•

•

(1) Click theS(cript) message whose output you want toS(cript) message consisting of:✓the date and time of the execution of the script✓

the name of the script executed✓

the name of the name of the corresponding userlog file

The userlog file content is displayed into the right panel. In the example, it is
the content of the file SAFEVAR/modules/AM/userlog_2024-0212T091410_start_prim.ulog (where AM is the module name)

39 A2 38MC 03

61

SafeKit User's Guide

To display the verbose module log, click on a message other than••S(cript).

(1) Click the message consisting of:✓the date and time of the event✓the module message

All verbose messages between the selected message and the previous one in
the table are displayed in the right-hand panel.

Refer to section 7 page 111 for messages examples.

3.4.3.2

Module resources

You can display resources of a module on one node:✓

Directly via the URL http://host:9010/console/en/monitoring/modules/AM/nodes/node/resources (replace node by the node name and AM by themodule name)

Or
✓

62

By navigating the console via

Monitoring/Click on the module>node/Resources tab

39 A2 38MC 03

The SafeKit web console

The left panel displays the current state of the resources for the selected
module>node.

•(1) Select the group of resources to view:
Module status
Main resources, especially the ones of files
replication for a mirror module
Checkers
Ressources set by checkers
File replication
File replication-specific resources that
demonstrate synchronization progress
All resources•

Click on a resource to display its value over time in the right panel. This history
may be empty for some resources (unassigned or cleaned).

Resource’s state is controlled by the failover machine to trigger a failover on failures(see section 13.18 page 263).

39 A2 38MC 03

63

SafeKit User's Guide

To display a resource's value history, click on the resource you're interested in.

•

•

3.4.4

(1) Click on the line consisting of:✓

the last date the resource was assigned✓

the name and category of the resource. The full resource name is like<category>.<name> (custom.checkfile in the example).

The history of resource values is displayed in the right panel. In the example,
this is the custom.checkfile resource corresponding to a resource assigned bya custom checker.

Module states timeline

Since SafeKit 8.2.2, you can display the module states timeline:✓

By navigating the console via

Monitoring/Click on

for the moduleThis provides a global view of the module's state on the cluster. Be aware that the clocksof the two nodes must be synchronized for the mapping of state changes to bemeaningful.
It opens a panel that displays a reverse timeline: the module states on all nodes overtime, by starting by the newest date.

64

39 A2 38MC 03

The SafeKit web console•(1) Click on
to open/close the timeline. The timeline displayed is the one
available at the time of loading.

•(2) Click on•(3) Click on a state change event to display the module log for the node
starting at this date

3.5

to refresh the timeline with the latest state changes.

Snapshots of module for support

When the problem is not easily identifiable, it is recommended to take a snapshot of themodule on all nodes as described below. Snapshots allows an offline and in-depth
analysis of the module and node status as described in section 7.16 page 125. If thisanalysis fails, send snapshots to support as described in section 8 page 133.
In the following example, the module AM is configured on node1 and node2. Note thata snapshot can be downloaded in any state of the module.

39 A2 38MC 03

65

SafeKit User's Guide•(1) Click on

to open the menu of global actions.

•(2) In case of file replication issues, it may be necessary to Generate the dump
files at the time the problem occurs.
The dump contains the module logs and information on the system and SafeKit
state at the time of the dump. It is generated on the server side into
SAFEVAR/snapshot/modules/AM/dump_AAAA_MM_DD_hh_mm_ss.

•(3) Click on Download the snapshots to create and download the snapshot of
the module for each node.
The web console relies on the web browser's download settings to save thesnapshot on the workstation. Some browsers may ask confirmation to
download many files and zip files.
The snapshot generation command generates a new dump and creates a .zip
file containing the last 3 dumps and the last 3 module configurations.

In this example, it downloads 2 snapshots : snapshot_node1_AM.zip andsnapshot_node2_AM.zip.

3.6

Secure access to the web console

SafeKit offers different security policies for the web console that are implemented bymodifying the SafeKit web service configuration. These configurations also offer role
management:
Admin role

This role grants all administrative rights by allowing access to
Configuration andMonitoring in the navigation sidebar

Control role

This role grants monitoring and control rights by allowing access only to
Monitoring in the navigation sidebar

Monitor role

This role grants only monitoring rights, prohibiting actions on modules(start, stop…) in
Monitoring in the navigation sidebar.

66

39 A2 38MC 03

The SafeKit web console

SafeKit provides different setups for the web service to enhance the security of the
SafeKit web console. The predefined setups are listed below from least secure to mostsecure:HTTP. Same role for all users without authentication
This solution can only be implemented only in HTTP and is not compatible with user
authentication methods. It is intended to be used for troubleshooting only.
HTTP/HTTPS with user authentication based on Apache files and optional role
management
It relies on Apache files to store username/password for authenticating users and,optionally, to store the associated role for restricting their access. To connect to theconsole, the user must enter the username and password as configured with the
Apache mechanisms.
This is the default active configuration, applied for HTTP and initialized with a single
admin user with the Admin role. The default setup can be extended to add users or toswitch to HTTPS.
HTTP/HTTPS with user authentication based on LDAP/AD authentication. Optional role
management
It relies on LDAP/AD authentication server to authenticate users and, optionally,
restricts their access based on roles. To connect to the console, the user must enter
the username and password as configured into the LDAP/AD server. It supports HTTP
or HTTPS.
HTTP/HTTPS with user authentication based on OpenId Connect authentication.
Optional role management
It relies on OpenID Identity Provider server to authenticate users and, optionally,
restricts their access based on roles. To connect to the console, the user must enter
the username and password as configured into the Identity Provider server. It
supports HTTP or HTTPS.
To implement them, refer to the section 11 page 175.

39 A2 38MC 03

67

SafeKit User's Guide

68

39 A2 38MC 03

4. Tests
4.1 “Installation and tests after boot” page 69
4.2 “Tests of a mirror module” page 72
4.3 “Tests of a farm module” page 79
4.4 “Tests of checkers common to mirror and farm” page 86
Subsequently, analysis of test results may require consulting the module log, the scriptslog (which contains the output of module scripts) and the state of module resources. To
read these logs and resources, see section 7.3 page 116.

4.1

Installation and tests after boot

4.1.1

Test package installation

Package installation:
Replace below node1 by the node name and AM by the module name.
safekit –p executed on the nodes returns among other values, the value of SAFE,
the SafeKit root installation path, and SAFEVAR, the SafeKit working directory:✓

in Windows
SAFE=C:\safekit if %SYSTEMDRIVE%=C:
SAFEVAR=C:\safekit\var✓

in Linux
SAFE=/opt/safekit"
SAFEVAR=/var/safekit

For details, see section 10.1 page 155.
Editing userconfig.xml of a mirror(/farm) module and its scripts
start_prim/start_both, stop_prim/stop_both is made with:✓the web console at /console/en/configuration/modules/AM/config✓

under the directory SAFE/modules/AM on the node1

Module log and scripts log (that contains module scripts output) for the module on
one node may be analyzed with :✓the web console at /console/en/monitoring/nodes/node1/modules/AM/logs✓

the command executed on node1
safekit logview –m AM for the module log✓

on node1, into files
SAFEVAR/modules/AM/userlog_<year>_<month>_<day>T<time>_<scriptname>.ulog for the scripts logs (output messages of the scripts)

39 A2 38MC 03

69

SafeKit User's Guide

4.1.2

Test license and version

safekit level returns
Host : <hostname>
OS : <OS version>
SafeKit : <SafeKit version>
License : Demo (No license)| Invalid Product | Invalid Host | … Expiration… | <license id> for<hostname>…
or License : Expired license"Demo (No license)" means no SAFE/conf/license.txt file: the product stops
every 3 days
"Invalid Product" means an expired license in SAFE/conf/license.txt"Invalid Host" means no valid hostname in SAFE/conf/license.txt" …Expiration…" means a temporary key"<license id> for <hostname>" means a permanent license
http://www.evidian.com/safekit/requestevalkey.php to get a temporary key of one
month for any OS or any hostname
https://support.evidian.com to get a permanent key based on the hostname andOS70

39 A2 38MC 03

Tests4.1.3

Test SafeKit services and processes running after bootSee also section 9.2 page 143.
Test safeadmin service:
The safeadmin process must appear in the list of running processes
Without this process, no safekit command works and they all return:"Waiting for safeadmin ..........""Error: safeadmin administrator daemon not running"

On Windows, safeadmin is a service and can be started in the Services interface ofWindows
on LINUX, safeadmin is started by service safeadmin start on Linux

Test safewebserver service:safekit boot webstatus displays start-up or not of safewebserver service atboot ("on" or "off", "on" by default)
httpd processes must be in the list of running processes if boot "on"without these processes, the web console is not able to connect to servers as well<module> checkers (userconfig.xml) and distributed command line interface
to start/stop the safewebserver service, run: safekit webserver start|stop

Test safeagent service (Windows only):safekit boot snmpstatus displays start-up or not of safeagent service at boot("on" or "off", "off" by default)
safeagent process must be in the list of running processes if boot "on"to start/stop the safeagent service, run: safekit safeagent start|stop

Test modules:
safekit boot status displays start-up ("on") or not ("off") of modules at bootsafekit state displays state of all configured modules: STOP (mirror or farm),
WAIT (mirror or farm), ALONE (mirror), PRIM (mirror), SECOND (mirror), UP (farm)
check processes of a module: see section 10.2 page 157
safekit module listid displays name of installed modules with their ids: id of amodule must be the same on all servers
go to SAFE/modules/AM/conf (replace AM by the module name); userconfig.xml
file gives the module type, mirror, or farm: <service mode="mirror"> or <service
mode="farm">

39 A2 38MC 03

71

SafeKit User's Guide

4.1.4

Test start of SafeKit web console

connect a web browser to http://<server IP>:9010
the web console home page is displayed

4.2

Tests of a mirror module

4.2.1

Test start of a mirror module on 2 servers

STOP (NotReady)message in the logs of both servers (to read logs, see section 7.3 page 116)
"Action start called by web@<IP>/SYSTEM/root"

the module goes to the stable stateboth servers with in the first log

PRIM (Ready)and

SECOND (Ready)on"Remote state SECOND Ready""Local state PRIM Ready "and in the other log"Local state SECOND Ready ""Remote state PRIM Ready "application is started in the start_prim script of the module on the PRIM serverwith message in the log"Script start_prim"

4.2.2

Test stop of a mirror module on the server

PRIM (Ready)message in the log of the stopped node (to read logs, see section 7.3 page 116)
"Action stop called by web@<IP>/SYSTEM/root"

the stopped node runs the stop_prim script of the module which stops theapplication on the server with message in the log:
"Script stop_prim"

the module becomes

STOP (NotReady) with messages in the log:

"End of stop""Local state STOP NotReady"the module becomeslog:

ALONE (Ready) on the other node with the message in the"Reason of failover: remote stop"

the application is started with the start_prim script on the ALONE node with themessage in the log:
"Script start_prim"

72

39 A2 38MC 03

Tests

4.2.3

Test start of a mirror module on the server

STOP (NotReady)message in the log of the started module (to read logs, see section 7.3 page 116)
"Action start called by web@<IP>/SYSTEM/root"

the

STOP (NotReady) module becomes

SECOND (Ready)the module
ALONE (Ready) on the other server becomescontinues to execute the application

4.2.4

PRIM (Ready) and

Test restart of a mirror module on the server

PRIM (Ready)message in the log of the server where the restart command is passed (to readlogs, see section 7.3 page 116)
"Action restart called by web@<IP>/SYSTEM/root"

the PRIM module becomes(Ready)PRIM (Transient) and then becomesPRIM

the scripts of the module stop_prim/start_prim are executed on the PRIM moduleand restarts locally the application on the server with messages in the log:
"Script stop_prim""Script start_prim"

the other module on the other server stays

4.2.5

SECOND (Ready)

Test swap of a mirror module from one server to the othermessage in the log of the server where the swap command is passed (to read logs,
see section 7.3 page 116)
"Action swap called by web@<IP>/SYSTEM/root""Transition SWAP from SYSTEM""Begin of Swap"

And in the log of the other server, only:"Begin of Swap"

reversing the roles of PRIM and SECOND between both servers
the stop_prim script is first executed on the former PRIM within its log:
"Script stop_prim"

then the start_prim script is executed on the new PRIM server within its log:
"Script start_prim"at the end of swap, module
PRIM (Ready) and module
SECOND (Ready) arereversed on both servers and the application is on the new PRIM server

39 A2 38MC 0373

SafeKit User's Guide

4.2.6

Test virtual IP address of a mirror module

Mirror module in the state
PRIM(Ready) on server node1 andSECOND(Ready) on server node2.
userconfig.xml:<vip><interface_list><interface arpreroute="on"><real_interface><virtual_addr addr="ipvirt"
where="one_side_alias"/></real_interface></interface></interface_list></vip>
1. On an external workstation (or server)in the same LAN, ping both physical IP
addresses + virtual IP address:
ping node1_ip_addressping node2_ip_addressping ipvirt
arp –a2. safekit swap –v AM on the primaryserver (where AM is the module name)
3. On the external workstation (orserver),ping node1_ip_addressping node2_ip_addressping ipvirt
arp –aNote: redo the ping to virtip before
looking at the ARP table because theentry may be marked obsolete andrefreshes only after ping74

1. On server node1, ipconfig or
ifconfig (or ip addr show) returns
ipvirt as an alias on the network
interface.
On the external workstation (orserver), the 3 pings respondOn the external workstation (orserver) in the same LAN, virtip is
mapped to the same MAC address as
node1_ip_address
arp –anode1_ip_address
00-0c-29-0a-5c-fc
node2_ip_address
00-0c-29-26-44-93
ipvirt00-0c-29-0a-5c-fc

2. After the swap,
SECOND (Ready) on
node1 server and
PRIM (Ready) on
node2 server
In the log of new primary, message:"Virtual IP <ipvirt of mirror> set"

3. On node2, ipconfig or ifconfig (orip addr show) returns ipvirt as analias on the network interface
On the external workstation (orserver), the 3 pings respondOn the external workstation (orserver), virtip is mapped to the same
MAC address as node2_ip_address
arp –anode1_ip_address
00-0c-29-0a-5c-fc
node2_ip_address
00-0c-29-26-44-93
ipvirt00-0c-29-26-44-93

39 A2 38MC 03

Tests

4.2.7

Test file replication of a mirror module
1. file1.txt has been replicated on
SECOND (Ready) under /replicated2. Failure because the /replicateddirectory is read-only on the server
SECOND (Ready)

Mirror module in the state
PRIM(Ready) on node1 server and
SECOND(Ready) on node2 server.
userconfig.xml:<rfs><replicated dir="C:\replicated"
mode="read_only" />(or "/replicated"on Linux)</rfs>
1. On the server
PRIM (Ready), go to/replicated and create a file file1.txt
2. On the server
SECOND (Ready), go
to /replicated and try to delete file1.txt
3. Stop the server

PRIM (Ready) andwait for
STOP (NotReady). Then go
to the other server which is
ALONE(Ready) and create a new file file2.txt
4. Restart the server
STOP (NotReady)and wait for
SECOND (Ready).

3. file2.txt is not replicated in /replicatedof the server
STOP (NotReady)
4. file2.txt is reintegrated on therestarted server. During the phase ofreintegration, the server is
SECOND(Transient)In the log of reintegrated server,
message"Updating directory tree from /replicated"

And at the end of /replicatedreintegration, if at least 1 file withmodified data has been reintegrated
from primary server to secondary
server, message"Copied <reintegration statistics>""Reintegration ended (synchronize)"This message gives statistics for the
reintegrated directory: reintegrated
size, number of files, time, andthroughput on the network in KB/sec.
Note: reintegrate a file larger than 100
MB to have reliable statistics
At the end of reintegration, the serveris
SECOND (Ready)39 A2 38MC 03

75

SafeKit User's Guide

4.2.8

Test mirror module shutdown on the server

PRIM (Ready)on Windows, check that the special procedure to stop modules at shutdown has
been applied.
make a shutdown of

PRIM (Ready) server

check in the log of server

SECOND (Ready), message"Reason of failover: remote stop"

the server
SECOND (Ready) becomes
ALONE (Ready); application in thestart_prim script of the module is restarted on the ALONE server with the message
in the log"Script start_prim"on timeout in the SafeKit console, the old server

PRIM (Ready) becomes grey

after reboot of the stopped server, check that the OS shutdown has really called ashutdown of the module"Action shutdown called by SYSTEM"

Check that the application stop_prim script has been executed with the message"Script stop_prim"

And check that the module has been completely stopped before shutting down theserver with the last message"End of stop"

after reboot of stopped server, if the module is started automatically at boot(safekit boot status), message in the log"Action start called at boot time"after a start of the module on the stopped server, the module becomes
SECOND (Ready) on this server and
PRIM (Ready) on the other server

76

39 A2 38MC 03

Tests

4.2.9

Test mirror module power-off on the server

PRIM (Ready)in the log of the server
SECOND (Ready), message for all
heartbeats configured inuserconfig.xml"Resource heartbeat.default set to down byheart""Resource heartbeat.flow set to down by heart""Remote state UNKNOWN grey""Reason of failover: no heartbeat"messages appear within 30 seconds
after the power-off (if no specified
timeout configured for <heart> inuserconfig.xml)
userconfig.xml:<heart><heartbeat name="default" /><heartbeat ident="flow" /></heart>Note: If you want to make a test withdouble simultaneous electrical fault on
both servers, check that <rfsasync="none"> is set in userconfig.xml.
For more information, see section 1.3.6
page 18

the server
SECOND (Ready) becomes
ALONE (Ready); the application in
the start_prim script of the module isrestarted on the ALONE server with themessage in its log"Script start_prim"on timeout in the SafeKit console, theformer server
PRIM (Ready)becomes grey
after reboot of stopped server, if themodule is started automatically atboot (safekit boot status),
message in the log"Action start called at boot time"after reboot, message in the log:
"Previous halt unexpected"

after restart of the module on the
stopped server, the module becomes
SECOND (Ready) on this server and
PRIM (Ready) on the other server

39 A2 38MC 03

77

SafeKit User's Guide

4.2.10

Test split brain with a mirror module
after network isolation of both servers,
all heartbeats are lost. In the logs ofboth servers,

Split brain occurs in situation of network
isolation between two SafeKit servers.
Each server becomes primary ALONE andruns the application. At return of split
brain, a sacrifice must be made by
shutting down the application on one of
the two servers.

"Resource heartbeat.default set to down byheart""Resource heartbeat.flow set to down by heart""Remote state UNKNOWN grey""Local state ALONE Ready "split brain case: both servers are
ALONE (Ready) and run theapplication started in start_prim

Mirror module in the state
PRIM(Ready) andSECOND (Ready)when reconnecting heartbeat
networks, sacrifice of one ALONE
server: the former SECOND server

userconfig.xml:<heart><heartbeat name=”default” /><heartbeat name=”repli” ident="flow"/></heart>log of the former PRIM not sacrificed:"Remote state ALONE Ready""Split brain recovery: staying alone"

log of the former SECOND sacrificed:+"Remote state ALONE Ready""Split brain recovery: exiting alone""Script stop_prim"

on Windows to manage the IP conflict on
the virtual IP address virtip<vip><interface_list><interface check="on"arpreroute="on"><real_interface><virtual_addr addr="192.168.1.10"
where="one_side_alias"/></real_interface></interface></interface_list></vip>
To obtain the split brain, check that there
are no checkers in userconfig.xml thatcan detect the network isolation: no<interface check="on">, no <ping>checker
1. disconnect all heartbeat networks at
the same time (network default andrepli)
2. reconnect networks

4.2.11The server performs a stopstart: stopof the application with stop_prim thenreintegration of replicated files fromthe other servercome back to the stable state
PRIM(Ready) andSECOND (Ready) on
both servers as it was before split
brain
Note: situation of split brain in a mirror
module with file replication is not good.
Indeed, the sacrifice of the former
secondary server causes file reintegrationof this server from the primary one andthe loss of data stored on the secondary
during the split-brain situation.
For this reason, 2 heartbeats on two
physically separate networks arerecommended. Typically, a cable between
the two servers will allow (1) to avoid split
brain with an additional heartbeat networkand (2) set the replication flow on a
separate network

Continue your mirror module tests with checkers

Go to section 4.4 page 86 for tests of checkers.

78

39 A2 38MC 03

Tests

4.3

Tests of a farm module

4.3.1

Test start of a farm module on all servers

STOP (NotReady)message in the logs of all servers (to read logs, see section 7.3 page 116)
"Action start called by web@<IP>/SYSTEM/root"

the module goes to

UP (Ready) on all servers

the application is started in the start_both script of the module on all servers withthe message in the log"Script start_both"

4.3.2

Test stop of a farm module on one server

UP (Ready)message in the log of the stopped server (to read logs, see section 7.3 page 116)
"Action stop called by web@<IP>/SYSTEM/root"

the stopped module runs the stop_both script which stops the application on theserver and with message in the log"Script stop_both"the stopped module becomes

STOP (NotReady) with messages in the log:

"End of stop""Local state STOP NotReady"the other servers stayrestart the module

4.3.3

UP (Ready) and continue to run the application

STOP (NotReady) with the start command

Test restart of a farm module on one server

UP(Ready)

message in the log of the module where the restart command is passed (to readlogs, see section 7.3 page 116)
"Action restart called by web@<IP>/SYSTEM/root"

the restarted module becomes

UP (Transient) then becomes

UP (Ready)the module scripts stop_both/start_both are executed on the server and restart
locally the application with messages in the log"Script stop_both""Script start_both"

39 A2 38MC 03

79

SafeKit User's Guide

4.3.4

Test virtual IP address of a farm module

4.3.4.1

Configuration with vmac_invisible

Farm module in theUP(Ready) state on 2 servers node1
and node2
userconfig.xml with loadbalancing on the safewebserver
service (TCP port 9010):<farm><lan name=”default” /></farm><vip><interface_list><interface><virtual_interface
type="vmac_invisible" ><virtual_addr
addr="virtip" where="alias"/></virtual_interface></interface></interface_list><loadbalancing_list><group name="FarmProto"><rule port="9010"
proto="tcp" filter="on_port"/></group></loadbalancing_list></vip>
On a remote workstation (orserver) in the same LAN, ping of
the 2 physical IP addresses +virtual IP + arp –a

80

In the log of all servers:"Vitual IP <virtip of farm> set"On the 2 servers, ipconfig or ifconfig (or ip
addr show) returns virtip as an alias on the
network interface
On a remote workstation (or server), the pings
respond. And virtip is mapped with the invisible
virtual MAC address:
ping node1_ip_address; ping node2_ip_address ; pingvirtip; arp –a
node1_ip_address
00-0c-29-0a-5c-fc
node2_ip_address
00-0c-29-26-44-93
virtip5a-fe-c0-a8-38-14

Note: by default, the virtual MAC address is aunicast Ethernet address built with 5A:FE(SAFE) and the virtual IP address inhexadecimal

39 A2 38MC 03

Tests

4.3.4.2

Configuration with vmac_directed

Farm module in theUP(Ready) state on 2 servers node1
and node2
userconfig.xml with loadbalancing on the safewebserver
service (TCP port 9010):<farm><lan name=”default” /></farm><vip><interface_list><interface arpreroute=”on”><virtual_interface
type="vmac_directed" ><virtual_addr
addr="virtip" where="alias"/></virtual_interface></interface></interface_list><loadbalancing_list><group name="FarmProto"><rule port="9010"
proto="tcp" filter="on_port"/></group></loadbalancing_list></vip>
On a remote workstation (orserver) in the same LAN, ping of
the 2 physical IP addresses +virtual IP + arp –a

39 A2 38MC 03

In the log of all servers:"Vitual IP <virtip of farm> set"On the 2 servers, ipconfig or ifconfig (or ip
addr show) returns virtip as an alias on the
network interface
On a remote workstation (or server), the pings
respond, and ip1.20 is mapped with the MAC
address of one of the 2 servers:
ping node1_ip_address; ping node2_ip_address; pingvirtip; arp –a
node1_ip_address
00-0c-29-0a-5c-fc
node2_ip_address
00-0c-29-26-44-93
virtip00-0c-29-26-44-93

81

SafeKit User's Guide

4.3.5

Test TCP load balancing on a virtual IP address

Farm module in the state
UP (Ready) on the 2 servers
node1, node2.
Same load balancing configurationin userconfig.xml as the previous
test.

UP (Ready) on the 2 servers: loadbalancing of TCP sessions between node1,
node2 when loading the URL
In the resources of the module, for node1 andnode2: FarmProto 50%

On a remote workstation:

Example of logs with node1 and node2:

1. Connect a browser tohttp://virtip:9010/safekit/mosaic.html, then click on Mosaic
Test. node1, node2 respondIn the logs of node1 and node2:

2. safekit stop –m AM on node2(where AM is the modulename). Reload the URL: node1
responds"farm membership: node1 node2 (group FarmProto)""farm load: 128/256 (group FarmProto)"

128/256: 128 bits on 256 are managed byeach server
safekit –r vip_if_ctrl –l on node1 andnode2:
Bitmap 1:00000000:00000000:00000000:00000000:
ffffffff:ffffffff:ffffffff:ffffffff
Bitmap 2:ffffffff:ffffffff:ffffffff:ffffffff:
00000000:00000000:00000000:0000000

Bits are fairly distributed between bothservers
Special command to check the loadbalancing bitmap for port 9010 oneach node
UP (Ready):safekit –r vip_if_ctrl –l
An entry in the bitmap of 256 bits
must be 1 on a single server.
Furthermore, the 256 bits are fairly
distributed in the bitmaps of allservers
UP (Ready) (if nodefinition of power inside
userconfig.xml)

82

STOP (NotReady) on node2: TCP sessions
served only by node1 when loading the URL
In the log of node1:"farm membership: node1 (group FarmProto)""farm load: 256/256 (group FarmProto)"

256/256: all the bits are managed by node1
safekit –r vip_if_ctrl –l on node1:
Bitmap 1:ffffffff:ffffffff:ffffffff:ffffffff:
ffffffff:ffffffff:ffffffff:ffffffff

All the bits are managed by node 1

39 A2 38MC 03

Tests4.3.6

Test split brain with a farm module

Split brain occurs in case of
network isolation between SafeKit
servers.
Farm module is
UP (Ready) on
the servers node1 and node2.
Same configuration of loadbalancing in userconfig.xml as
the previous test. To get the split
brain, check in userconfig.xml
that there are no checkers thatcan detect isolation: no<interface check="on"> or<ping> checker
On the external workstation:
1. Connect a browser tohttp://virtip:9010/safekit/mos
aic.html, then click on Mosaic
Test. node1 and node2
respondbefore split brain, state
node1 and node2:

UP (Ready) on

In the resources of the module, for node1 andnode2: FarmProto 50%.
In the logs of node1 and node2:"farm membership: node1 node2 (group FarmProto)""farm load: 128/256 (group FarmProto)"

128/256: 128 bits on 256 are managed byeach server
safekit –r vip_if_ctrl –l on node1 andnode2:
Bitmap 1:00000000:00000000:00000000:00000000:
ffffffff:ffffffff:ffffffff:ffffffff
Bitmap 2:ffffffff:ffffffff:ffffffff:ffffffff:
00000000:00000000:00000000:0000000

Bits are fairly distributed between both servers
after isolation of servers, split brain:
In the resources of the module, for node1 andnode2: FarmProto 100%.
In the log of node1:"farm membership: node1 (group FarmProto)""farm load: 256/256 (group FarmProto)"

2. disconnect the network
between node1 and node2.
Depending on the location
where the external console is,node 1 responds or node 2

256/256: all the bits are managed by node 1
safekit –r vip_if_ctrl –l on node1:
Bitmap 1:ffffffff:ffffffff:ffffffff:ffffffff:
ffffffff:ffffffff:ffffffff:ffffffff

in the log of node 2:or"farm membership: node2 (group FarmProto)""farm load: 256/256 (group FarmProto)"

256/256: all the bits are managed by node 2
Bitmap 2:ffffffff:ffffffff:ffffffff:ffffffff:
ffffffff:ffffffff:ffffffff:ffffffff

3. reconnect the network andconnect to URL

Same special command as in theprevious test to check the loadbalancing bitmap for port 9010
on each node
UP (Ready)39 A2 38MC 03

after split brain when network is reconnected
between ip1.1 and ip1.2, the same messages
can be found in the log and the same bitmaps
as those before split brain
Note: the default behavior of farm in situation of
split brain is good. The recommendation is to put
in userconfig.xml a monitoring network <lan></lan> where the virtual IP address is.
Note: In vmac_directed mode, the log messages
and vip_if_ctrl output are different.

83SafeKit User's Guide

4.3.7
Test compatibility of the network with invisible MAC address(vmac_invisible)
4.3.7.1

Network prerequisite

A unicast MAC Ethernet address 5afe-xx-xx-xx-xx is associated withthe virtual IP address of a farm
module. It is never presented by
SafeKit servers as source Ethernet
address (invisible MAC). Switches
cannot locate this address. Whenthey follow a packet to thedestination MAC address 5a-fe-xxxx-xx-xx, they must broadcast thepacket on all ports of the LAN or
VLAN where the virtual IP address
is (flooding). All servers in the farm
therefore receive packets destinedto the virtual MAC address 5a-fexx-xx-xx-xx.

all servers are

UP (Ready)the network monitoring is started on eachserver with a filter on virtip
an external workstation sends a single pingto the virtual IP address with ping –n (or –c)
1 virtip
result: 1 packet "ICMP: Echo: From ipconsole
To virtip" sent and received by all servers
result: there must be as many packets"ICMP: Echo Reply: To ipconsole From virtip"
as there are servers
UP (Ready)Note that this prerequisite does notexist for a mirror module: seesection 4.2.6 page 74

if it is not the case, check if options restrictthe "port flooding" in switches and preventthe broadcast of "ICMP: Echo" to all servers

4.3.7.2

be careful: the "port flooding" restriction inswitches can occur after a certain number offlooding (time, number of KB flooded): theping test must be repeated during several
hours by creating flooding to the virtual IP
address

Server prerequisite

The packets are captured by
Ethernet cards set in promiscuousmode by SafeKit. And the packets
are filtered by the module kernel<vip> according to the loadbalancing bitmap. To make a test,
you need network monitor tool.
Network monitoring on Windows
2003 (CD2):
install "Network Monitor Tools"
in "Management and Monitoring
Tools" (capture only packets insource or destination of theserver)Start / Network Monitor thenCapture Filter / Address Pairs /virtip then Capture / Start then"Stop and View" at the end of
captureNote: to avoid network monitoring tools, anexternal Linux console can be used. The
Linux ping prints duplicate packets comingfrom the 2 servers
UP (Ready):
ping virtip
64 bytes from ip1.20 icmp_seq=1
64 bytes from ip1.20 icmp_seq=1 (DUP!)
64 bytes from ip1.20 icmp_seq=2
64 bytes from ip1.20 icmp_seq=2 (DUP!)...

This test may be carried out for several hours
by storing the output of the ping in a file andthen ensuring that there was (DUP!) all thetime: date > /tmp/ping.txt ; pingvirtip >> /tmp/ping.txt

Network monitoring on Linux:
tcpdump host virtip: capture all
network packets

84

39 A2 38MC 03

Tests

4.3.8

Test farm module shutdown of a server

UP (Ready)on Windows, check that the special procedure to stop modules at shutdown has
been performed
make a shutdown of a

UP (Ready) server

the other servers stay

UP (Ready) and continue to run the application

on timeout in the SafeKit console, the former server

UP (Ready) becomes grey

after reboot, check that shutdown of the OS has called a shutdown of the module"Action shutdown called by SYSTEM"

Check that the stop_both script which stops the application has been executedwith the message"Script stop_both"

And check that the module has been completely stopped before stopping the serverwith the last message"End of stop"

after reboot of the stopped server, if the module is started automatically at boot(safekit boot status), message in the log"Action start called at boot time"after start-up of the module on the stopped server, the module becomesUP(Ready) and it executes the start_both script which restarts the application on
this server with the message in the log"Script start_both"

4.3.9

Test farm module power-off of a server

the other servers stay

UP (Ready)UP (Ready) and continue to run the application

on timeout in the SafeKit console, the former server

UP (Ready) becomes grey

after reboot of the stopped server, if the module is started automatically at boot(safekit boot status), message in the log"Action start called at boot time"after reboot, message in the log"Previous halt unexpected"

after start-up of the module on the stopped server, the module becomes
UP (Ready) and it executes the start_both script which restarts the applicationon this server with the message in the log"Script start_both"4.3.10Continue your farm module tests with checkers

Go to section 4.4 page 86 for tests of checkers.

39 A2 38MC 03

85

SafeKit User's Guide

4.4

Tests of checkers common to mirror and farm

4.4.1

Test <errd>: checker of process with action restart or stopstart

In userconfig.xml:

Kill of process appli.exe on the server in(Ready) state. That is in states PRIM or
ALONE for a mirror module, UP for a farm
module:<errd><proc name="appli.exe" atleast="1"
messages in the log:
action="restart "class="prim "/></errd>"Process appli.exe not running""Action restart|stopstart called by errd"

the module becomes(Transient),
respectively in state PRIM, ALONE or UP

name="appli.exe" atleast="1": atleast one process "appli.exe" must run

in the restart case, the module
becomes(Ready), respectively instate PRIM, ALONE or UP

class=”prim” (mirror module case):checker started on the server in state
PRIM or ALONE (Ready), after
start_prim script (stopped before
stop_prim)in the stopstart case, the module
becomes(Ready), respectively instate SECOND, ALONE or UP

class="both" (farm module case):checker started on all servers
UP(Ready) after start_both script(stopped before stop_both)

action="restart": if appli.exe is
not running, action restart whichapplies only scripts stop_xx; start_xx
action="stopstart": if appli.exe is
not running, action stopstart which
stops completely the module and thenrestarts it

message in the log:
"Action start called automatically"

Note: a stopstart oncauses a failover

PRIM (Ready)

Repeat the test on the same server if it
still runs the application (i.e.,(Ready)in state ALONE or UP):
with the default values of maxloop="3"
and loop_interval="24"(userconfig.xml <service>)
after 4 kills on the same server, themodule becomes
STOP (NotReady)in the log, message before stopping:"Stopping loop"

86

39 A2 38MC 03

Tests

4.4.2
Test <tcp> checker of the local application with action restart or
stopstart
In userconfig.xml:<tcp ident="id" when="prim "><to addr="virtip" port="idport"interval="10"
timeout="5" /></tcp><failover><![CDATA[
tcpid_failure: if (tcp.id == down)then stopstart();]]></failover>

Stop the application listening on port
idport on the server in state(Ready).
That is in states PRIM or ALONE for a
mirror module, UP for a farm module:

messages in the log:
"Resource tcp.id set to down by tcpcheck""Action restart|stopstart from failover ruletcpid_failure "the checker checks that the TCP
application started on port idport
responds to connection requests

the module becomes(Transient),
respectively in state PRIM, ALONE or UP

addr="virtip" port="idport" : TCP
connections tested on IP address virtip
and on TCP port idportin the restart case, the module
becomes(Ready), respectively instate PRIM, ALONE or UP

interval="10" timeout="5" bydefault: test made every 10 seconds
and with a timeout of 5 seconds

in the stopstart case, the module
becomes(Ready), respectively instate SECOND, ALONE or UP.
message in the log:

when="prim" (mirror module case):checker is started on the server in
state(Ready) (i.e., PRIM or
ALONE), after the start_prim script(stopped before stop_prim)
when="both" (farm module case):checker is started on all servers in
state(Ready) UP, after thestart_both script (stopped before
stop_both)

action restart(): Default failover
rule; if the local TCP connection fails,
action restart which runs only scripts
stop_xx ; start_xx
action stopstart(): if the local TCP
connection fails, action stopstart
which stops completely the moduleand then restarts it

39 A2 38MC 03"Action start called automatically"

Note: a stopstart oncauses a failover.

PRIM (Ready)

Repeat the test on the same server if it
still runs the application (i.e.,(Ready)in state ALONE or UP):
with the default values of maxloop="3"loop_interval="24"(userconfig.xml <service>)
after 4 stops of the application on thesame server, the module becomes
STOP (NotReady)in the log, message before stopping:"Stopping loop"

87SafeKit User's Guide

4.4.3

Test <tcp> checker of an external service with action wait

In userconfig.xml:<tcp ident="id" when="pre"><to addr="ip.external" port="idport"interval="10"
timeout="5" /></tcp><failover><![CDATA[
tcpid_failure: if (tcp.id== down) thenwait();]]></failover>

Stop the external TCP service(ip.external, idport), on the server in(Ready) state. That is in state PRIM,
ALONE or SECOND for a mirror module, UP
for a farm module:
messages in the log:
"Resource tcp.id set to down by tcpcheck""Action wait from failover rule tcpid_failure"Note: a wait on
causes a failover

PRIM (Ready)in all cases, the server becomes
WAIT (NotReady) on the server

Restart the external TCP process andservices:messages in the log

the checker checks that the external
TCP service (ip.external, idport)
responds to connection requests
interval="10" timeout="5" bydefault: test made every 10 seconds
and with a timeout of 5 seconds
when="pre": started at the beginningof module start-up after prestart script(and stopped before poststop)

if the TCP connection fails, the checker
sets the resource tcp.id to down. The
failover rule on the TCP checker runs
the stopwait action which stops theapplication and puts the module in thestate WAIT, waiting for tcp.id reset to
up by the checker"Resource tcp.id set to up by tcpcheck""Transition WAKEUP from failover rule
Implicit_WAKEUP"

the module restarts on the server andbecomes(Ready), respectively instate SECOND, ALONE, SECOND or UP

Repeat the test on the same server:
with the default values of maxloop="3"loop_interval="24"(userconfig.xml <service>)
after 4 restarts of on the same server,
the module becomes
STOP(NotReady)in the log, message before stopping:"Stopping loop"

Note: This test allows testing of
connectivity to an external service. But if
the external service is down or isunreachable on all servers, all servers arein state
WAIT (NotReady) and theapplication is unavailable

88

39 A2 38MC 03

Tests4.4.4
Test <interface check="on"> on a local network interface andwith action wait
Unplug the Ethernet cable from ip.0
network on the server in(Ready) state.
That is in state PRIM, ALONE or SECOND fora mirror module, UP for a farm module:
messages in the log:
"Resource intf.ip.default set to down by
intfcheck"

In userconfig.xml:"Action wait from failover rule interface_failure"<vip><interface_list><interface check="on"><!-definition of a virtual IP
addresson the network default--></interface></interface_list></vip>"Transition WAIT_TR from failover ruleinterface_failure"Default failover rule = wait

A checker checks that the Ethernet
cable is connected in the interface of
the ip.0 network where the virtual IP
address is set
If the cable is disconnected, thechecker updates the resource
intf.ip.0 to down. The failover ruleon interface checkers runs the
stopwait action which stops theapplication and puts the module in the
WAIT state waiting for intf.ip.0 reset to
up by the checker.
Note: do not use check="on" on bondingor teaming interface because theseinterfaces bring their own failover
mechanisms from interface to interface

Note: a wait on
causes a failover

PRIM (Ready)in all cases, the module becomes
WAIT (NotReady) on the server

Plug the cable again:
messages in the log"Resource intf.ip.0 set to up by intfcheck""Transition WAKEUP from failover rule
Implicit_WAKEUP"

the module restarts on the server andbecomes(Ready), respectively instate SECOND, ALONE, SECOND or UP

Repeat the test on the same server:
with the default values of maxloop="3"loop_interval="24"(userconfig.xml <service>)
after 4 restarts on the same server,
the module becomes
STOP(NotReady)in the log, message before stopping:"Stopping loop"

Note: disabling the interface (insteadof unplugging the ethernet cable)
leads to
STOP (NotReady). The
reason is that the module cannot start(or restart) without local IP address.

39 A2 38MC 03

89

SafeKit User's Guide

4.4.5

Test <ping> checker with action wait
Break the link between the pinged
external device and the server the serverin (Ready) state. That is in state PRIM,
ALONE or SECOND for a mirror module, UP
for a farm module:
messages in the log:
"Resource ping.id set to down by pingcheck""Action wait from failover rule ping_failure"

In userconfig.xml:<ping ident="id" when="pre"><to addr="ip.device" interval="10"
timeout="5"/></ping>Default failover rule = wait
the checker checks that the external
device (ex.: a router) with address
ip.device responds to pinginterval="10" timeout="5" bydefault: test made every 10 seconds
and with a timeout of 5 seconds
when="pre": started at the beginningof module start-up after prestart script(and stopped before poststop)
if the ping does not respond, thechecker sets the resource ping.id todown. The failover rule on ping checker
runs the stopwait action which stops
the application and puts the module in
the WAIT state, waiting for ping.id
reset to up by the checker.

Note: a wait on
causes a failover

PRIM (Ready)in all cases, the module becomes
WAIT (NotReady) on the server

Restore the network connection:
messages in the log"Resource ping.id set to up by pingcheck""Transition WAKEUP from failover rule
Implicit_WAKEUP"

the module restarts on the server andbecomes(Ready), respectively instate SECOND, ALONE, SECOND or UP

Repeat the test on the same server:
with the default values of maxloop="3"loop_interval="24"(userconfig.xml <service>)
after 4 restarts on the same server,
the module becomes
STOP(NotReady)in the log, message before stopping:"Stopping loop"

Note: this test allows testing of
connectivity from the server to the
network. But if the external device isdown and if the ping fails on all servers,
all servers are in
WAIT (NotReady) andthe application is unavailable.

90

39 A2 38MC 03

Tests

4.4.6

Test <module> checker with action wait
Stop the module othermodule. And start
the module X on all servers:
messages in the log of module X

In userconfig.xml of module X, test ofanother module othermodule:
userconfig.xml of module X:<module name="othermodule"><to addr="ip" interval="10"
timeout="5"/></module>
the checker checks the module
othermodule on its virtual IP address
ip
interval="10" timeout="5" bydefault: test made every 10 seconds
and with a timeout of 5 seconds"Resource module.othermodule_ip set to down
by modulecheck"Action wait from failover rule module_failure"

the module X becomes
WAIT(NotReady) on all servers

Start the module othermodule:
messages in the log of module X"Resource module.othermodule_ip set to up by
modulecheck""Transition WAKEUP from failover rule
Implicit_WAKEUP"

the module X starts on all servers in(Ready)If the module othermodule is not started,the module X stay in the WAIT state
waiting for its restart

Make safekit restart –m othermodule

The module X makes a stopstart when themodule othermodule is restarted

the module X stops and then restarts

Note: if the module X is a mirror moduleusing file replication and because of rule
notuptodate_server, you may
experience a wrong behavior with module
X blocked in a WAIT state, if the stopstartaction happens when X in the transition
SECOND to ALONE

messages in the log of module X:"Action stopstart called by modulecheck"

Repeat the test on the same server:
with the default values of maxloop="3"loop_interval="24"(userconfig.xml <service>)
after 4 restarts on the same server,
the module becomes
STOP(NotReady)in the log, message before stopping:"Stopping loop"

39 A2 38MC 0391

SafeKit User's Guide

4.4.7

Test <custom> checker with action wait

In userconfig.xml:<custom ident="id" when="pre"exec="customscript" ></custom>
scriptSAFE/module/<name>/bin/customscript
is a custom checker: a loop with a test
on a resource
when="pre": custom checker started on
all servers
PRIM, ALONE, SECOND or UP(Ready) after prestart script (stopped
before poststop)
Manage the resource custom.id to performthe action:in the script customscript:
on error: SAFE/safekit set -r custom.id–v down
–i customscript

on success: SAFE/safekit set -r custom.id–v up–i customscript

in userconfig.xml:<failover><![CDATA[
customid_failure: if (custom.id == down) thenwait();]]></failover>

Cause the error evaluated by custom
checker on the server in state(Ready). That is in state PRIM, ALONE orSECOND for a mirror module, UP for a
farm module:
messages in the log:
"Resource custom.id set to down by
customscript""Action wait from failover rulecustomid_failure""Transition WAIT_TR from failover rulecustomid_failure"Note: a wait on
causes a failover

PRIM (Ready)in all cases, the module becomes
WAIT (NotReady) on the server

Fix error tested by custom checker:
messages in the log"Resource custom.id set to up by
customscript""Transition WAKEUP from failover rule
Implicit_WAKEUP"

the module restarts on the serverand becomes(Ready),respectively in state SECOND, ALONE,SECOND or UP

Repeat the test on the same server:

if the custom checker sets the resource
to down, action wait which stops
completely the module and restarts it inwith the default values of
maxloop="3" loop_interval="24"(userconfig.xml <service>)the state
WAIT (NotReady), waiting
for the resource reset to up by thecustom checker

after 4 restarts on the same server,
the module becomes
STOP(NotReady)in the log, message before stopping:"Stopping loop"

92

39 A2 38MC 03

Tests

4.4.8

Test <custom> checker with action restart or stopstart

4.4.8.1

Action through a failover rule

In userconfig.xml:<custom ident="id" when="prim "
exec="customscript" ></custom>
script customscript
SAFE/module/<name>/bin/customsc
ript is a custom checker: loop witha test on the application integrated in
the scripts

Cause the error evaluated by custom checker
on the server in state(Ready). That is instate PRIM, ALONE or SECOND for a mirror
module, UP for a farm module:

messages in the log"Resource custom.id set to down by customscript""Action restart from failover rule customid_failure"when="prim" (mirror module case):custom checker started on the server
PRIM/ALONE (Ready) after
start_prim script (stopped before
stop_prim)
when="both" (farm module case):custom checker started on all servers
UP (Ready) after start_both
script (stopped before stop_both)

Manage the resource custom.id toperform the action:in the script customscript:
on error: safekit set -r custom.id–v down –icustomscript

on success: safekit set -r custom.id–v up –i"Transition RESTART from failover rulecustomid_failure"

the module becomes(Transient),
respectively in state PRIM, ALONE or UP

in the restart case, the module becomes(Ready), respectively state PRIM, ALONEor UP
in the stopstart case, the module becomes(Ready), respectively in state SECOND,
ALONE or UP
message in the log"Action start called automatically"

Note: a stopstart oncauses a failover

PRIM (Ready)customscript

in userconfig.xml:<failover><![CDATA[
customid_failure: if (custom.id ==down) then restart ();]]></failover>or<failover><![CDATA[
customid_failure: if (custom.id ==down) then stopstart ();]]></failover>

39 A2 38MC 03

Repeat the test on the same server if it stillruns the application (i.e.,(Ready) in state
ALONE or UP):
with the default values of maxloop="3"loop_interval="24" (userconfig.xml<service>)
after 4 restarts on the same server, themodule becomes
STOP (NotReady)in the log, message before stopping:"Stopping loop"

93

SafeKit User's Guide

4.4.8.2

Action through a command in the custom checker

In userconfig.xml:<custom ident="id" when="prim "
exec="customscript" ></custom>Cause the error evaluated by custom checker
on the server in state(Ready). That is instate PRIM, ALONE or SECOND for a mirror
module, UP for a farm module:
messages in the log"Action restart called by customscript"

Or"Action stoptart called by customscript"

scriptSAFE/module/<name>/bin/customsc
ript is a custom checker: loop witha test on the application integrated in
the scripts
when="prim" (mirror module case):custom checker started on the server
PRIM or ALONE (Ready) after
start_prim script (stopped before
stop_prim)
when="both" (farm module case):custom checker started on all servers
UP (Ready) after start_both
script (stopped before stop_both)

On error, run command
restart|stopstart:in the script customscript:
on error: safekit restart -i customscript

the module becomes(Transient),
respectively in state PRIM, ALONE or UP
in the restart case, the module becomes(Ready), respectively in state PRIM,
ALONE or UP
in the stopstart case, the module becomes(Ready), respectively in state SECOND,
ALONE or UP
message in the log"Action start called automatically"

Note: a stopstart oncauses a failover

PRIM (Ready)

Repeat the test on the same server if it stillruns the application (i.e.,(Ready) in state
ALONE or UP):
with the default values of maxloop="3"loop_interval="24" (userconfig.xml<service>)or

after 4 restarts on the same server, themodule becomes
STOP (NotReady)safekit stopstart -i customscriptin the log, message before stopping:"Stopping loop"

action restart: run only scripts
stop_xx ; start_xx
action stopstart: stop completely themodule and then restart it

94

Note: on a direct action in the custom checker,
the loop counter is incremented if –i identity
is passed to the command restart or stopstart.
Without identity, SafeKit considers thecommand is as an administrative operation. The
counter is reset and there is no stop after 4
restarts.

39 A2 38MC 03

5. Mirror module administration
5.1 “Operating mode of a mirror module” page 96
5.2 “State automaton of a mirror module (STOP, WAIT, ALONE, PRIM, SECOND NotReady, Transient, Ready)” page 97
5.3 “First start-up of a mirror module (safekit prim command)” page 98
5.4 “Different reintegration cases (use of bitmaps)” page 99
5.5 “Start-up of a mirror module with the up-to-date data
STOP (NotReady) -WAIT (NotReady)” page 100

5.6 “Degraded replication mode (ALONE (Ready) degraded)” page 101

5.7 “Automatic or manual failover” page 103
5.8 “Default primary server (automatic swap after reintegration)” page 105
5.9 “Prim command fails: why? (safekit primforce command)” page 106

To test a mirror module, see section 4.2 page 72
To analyze a problem, see section 7 page 111.

39 A2 38MC 03

95

SafeKit User's Guide

5.1

Operating mode of a mirror module

1. Normal operation

2. Automatic failover

Stable state: primary with secondary.

Stable state: primary without secondary.

On the primary:✓Virtual IP is set✓

Application is running✓

Real-time file replication

On primary stop, automatic failover of thevirtual IP and application.

The secondary is ready to run a failoverand become primary.
3. Failback and reintegration
Transient state: secondary reintegrating.

4. Back to normal operation
Stable state: primary with secondary.

Automatic file synchronization without
application shutdown and updating only
the files that were modified on the primary
while the other node was stopped.

96

39 A2 38MC 03

Mirror module administration

5.2
State automaton of a mirror module (STOP, WAIT, ALONE,
PRIM, SECOND - NotReady, Transient, Ready)39 A2 38MC 03

97

SafeKit User's Guide

5.3

First start-up of a mirror module (safekit prim command)

At first start-up of a mirror module, if both servers are started with the startcommand, both go into

WAIT (NotReady)state with the message "Data may be not

uptodate for replicated directories (wait for the start of the remote server)" in the log.

At first start-up of a mirror module, use the special prim command on the server withthe up-to-date directory, and the second command on the other one. Data is
synchronized from the primary server to the secondary one.
For next start-up, use the start command on both servers.

1. initial state

STOP(NotReady)

STOP(NotReady)

ALONE(Ready)STOP(NotReady)

PRIM(Ready)SECOND(Ready)the mirror module has just been
configured with a new directory to
replicate between node1 and node2
node1 has the up-to-date directory
node2 has an empty directory

2. command prim on node1
use the special prim command to force
node1 to become primary
for following start-ups, always preferstart: see section 5.5 page 100
message in the log:
"Action prim called by web@<IP>/SYSTEM/root"

3. command second on node2
start the other server as secondary
the secondary reintegrates replicateddirectory from primary
message in the log:
"Action second called by web@<IP>/SYSTEM/root"98

39 A2 38MC 03

Mirror module administration

5.4

Different reintegration cases (use of bitmaps)

To optimize file reintegration, different cases are considered:
1. The module must have completed the reintegration (on the first start of themodule, it runs a full reintegration) before enabling the tracking of modification into
bitmaps
2. If the module was cleanly stopped on the server, then at restart of the secondary,
only the modified zones of modified files are reintegrated, according to a set ofmodification tracking bitmaps.
3. If the server crashed (power off) or was incorrectly stopped (exception in nfsbox
replication process), or if files have been modified while SafeKit was stopped, themodification bitmaps are not reliable, and are therefore discarded. All the files
bearing a modification timestamp more recent than the last known synchronization
point minus a grace delay (typically one hour) are reintegrated.
4. A call to the special second fullsync command triggers a full reintegration of all
replicated directories on the secondary when it is restarted.
ALONE(Ready)STOP(NotReady)

ALONE(Ready)SECOND(Transient)

PRIM(Ready)SECOND(Ready)1. secondary server2 has been stopped
data is desynchronized

2. start command on node2
data is reintegrated with bitmap optimization(see above)

3. end of reintegration
data is the same on both servers
only modifications inside files are replicatedwith a real-time synchronous replication

39 A2 38MC 03

99

SafeKit User's Guide

The replication system also keeps track of the last date on which data was synchronizedon each node. This synchronization date, named synctimestamp, is assigned at the endof the reintegration and changes in the
PRIM (Ready) andSECOND (Ready)states.
When the module is stopped on the secondary node and then restarted, thesynctimestamp is one of the reintegration criteria: all files modified around this date arepotentially out of date on the secondary and must be reintegrated. Since SafeKit
7.4.0.50, the synchronization date is also used to implement an additional security.
When the difference between the synchronization date stored on the primary and on thesecondary is greater than 90 seconds, the replicated data is considered unsynchronizedin its entirety. The reintegration is interrupted with the following message in the modulelog:| 2021-08-06 08:40:20.909224 | reintegre | E | Automatic synchronization
cannot be applied due to an abnormal delta between the dates of the lastsynchronization
If the administrator considers that the server is valid, he can force the start in secondarywith full synchronization of the data, by executing the command: safekit second
fullsync -m AM.

5.5

Start-up of a mirror module with the up-to-date data
STOP (NotReady) WAIT (NotReady)

SafeKit determines which server must start as primary or not. SafeKit retains theinformation on the server with the up-to-date replicated directories. To takeadvantage of this feature, use the command start and NOT the command prim
1. initial state

ALONE(Ready)STOP(NotReady)

STOP(NotReady)

STOP(NotReady)server1 is primary ALONE
directories are up-to-date on this server
the module is stopped on node2
node2 has desynchronized replicateddirectories

2. command stop on node1
stop of the server with the up-to-datedirectories

100

39 A2 38MC 03

Mirror module administration

3. command start on node2
the module is put in the WAIT state waiting forthe start of the other server and within its log
of messages:

STOP(NotReady)

WAIT(NotReady)"Data may be not uptodate for replicated directories (waitfor the start of the remote server)""Action wait from failover rule notuptodate_server""If you are sure that this server has valid data, runsafekit prim to force start as primary"in this case, you must start server1 toresynchronize data of server2
if you really want to sacrifice the up-to-date
data and start node2 as primary with the data
not up-to-date: issue a stop command then aprim command on node2

rfs.uptodate="down"

See also 5.9 “Prim command fails: why? (safekit primforce command)” page 106

5.6

Degraded replication mode (ALONE (Ready) degraded)

If the replication process nfsbox fails on the primary server (for instance because ofan unrecoverable replication problem), the application is not swapped on thesecondary server
The primary server goes to the ALONE state in a degraded replication mode.
Degraded is displayed in the web console. A "Resource rfs.degraded set to up
by nfsadmin" message is emitted in the log. safekit state –v –m AM returns
resource rfs.degraded up (replace AM by the module name)
The primary server continues in ALONE state with a nfsbox process which does not
replicate anymore.
You must stop and start the ALONE server to come back to a PRIM – SECOND statewith replication
PRIM(Ready)SECOND(Ready)1. initial state
the mirror is in a stable state:
node1

PRIM (Ready)node2

SECOND (Ready)39 A2 38MC 03

101

SafeKit User's Guide

2. failure of replication process nfsbox on
node1
node1 becomes
ALONE (Ready) degradedwith the message in its log"Resource rfs.degraded set to up by nfsadmin".

ALONE(Ready)WAIT(NotReady)safekit state –v AM returns resource
rfs.degraded=up (where AM is the module
name)node1 ALONE continues to execute theapplication without replication
node2 is in
WAIT (NotReady) waiting forthe replication process with the message in its
log

rfs.degraded="up" rfs.uptodate="down""Action wait from failover rule degraded_server"

and with rfs.uptodate="down"
3. ome back to replication
administrator makes stop command and startcommand on node1 ALONE

PRIM(Ready)SECOND(Ready)the nfsbox replication process is restarted on
node1
node2 reintegrates replicated directories
before becoming
SECOND (Ready)node1 becomes102

PRIM (Ready)39 A2 38MC 03

Mirror module administration

5.7

Automatic or manual failover

Automatic or manual failover on the secondary server is defined in userconfig.xml by<service mode="mirror" failover="on"|"off">. By default, if the parameter is notdefined, failover="on"The failover="off" mode is useful when the failover must be controlled by anadministrator. This mode ensures that an application runs always on the same primaryserver whatever operations are made on the server (reboot, temporary stop of themodule for maintenance...). Only an explicit administrative action (prim command)
may promote the other server as primary.
Note: Failover mode could be set dynamically on a running cluster with the safekit
failover on|off –v AM (replace AM by the module name).
PRIM(Ready)SECOND(Ready)STOP(NotReady)

ALONE(Ready)1. initial state
the mirror is in a stable state:
node1

PRIM (Ready)node2

SECOND (Ready)

2. restart with failover="on"
if node1 former PRIM fails and stops, node2
becomes automatically
ALONE (Ready) (default mode)

39 A2 38MC 03

103

SafeKit User's Guide

3. behavior with failover="off"if node1 former PRIM fails and stops, node2
goes to
WAIT (NotReady) state withmessage in its log"Failover-off configured""Action stopstart called by failover-off""Transition STOPSTART from failover-off""Local state WAIT NotReady "STOP(NotReady)

WAIT(NotReady)the administrator in this situation can restartnode1: the mirror restarts in its former stable
state
node1

PRIM (Ready)node2

SECOND (Ready)the administrator can decide to force node2 to
become primary with the command: stop thenprim on node2
See also section 5.9 page 103

104

39 A2 38MC 03

Mirror module administration

5.8
Default primary server (automatic swap after
reintegration)
After reintegration at failback, a server becomes by default secondary. The
administrator may choose to swap the application back to the reintegrated server at
an appropriate time with the swap command. This is the default behavior whenuserconfig.xml <service> is defined without the defaultprim variable
If the application must automatically swap back to a preferred server after
reintegration, specify a defaultprim server in userconfig.xml: <service
mode="mirror" defaultprim="hostname node1">
STOP(NotReady)

ALONE(Ready)SECOND(Ready)PRIM(Ready)PRIM(Ready)SECOND(Ready)1. initial state
node1 (former PRIM) fails and stops
node2 secondary becomes automaticallyALONE

2. failback without defaultprim
node1 is restarted with command start
it reintegrates replicated directories andthen becomes secondaryan administrator can swap the primary to
node1 with the command swap in a timely
mannerswap stops the application on node2 andrestarts it on node1
3. failback with defaultprim="hostname
node1"
node1 in
STOP (NotReady) at step 1(initial state) is restarted by command start
it reintegrates replicated directories
just after reintegration, an automatic swap
is made on node1 with the message in its
log:"Transition SWAP from defaultprim""Begin of Swap"the application is then automatically stopped
on node2 and restarted on node1
at the end, node1 is PRIM

39 A2 38MC 03

105

SafeKit User's Guide

5.9

Prim command fails: why? (safekit primforce command)A prim command may fail to start a server as primary: after trying a start-up, theserver goes back to
STOP (NotReady).
ALONE(Ready)SECOND(Transient)

1. initial state
node1 ALONE has the up-to-date directory
node2 is in the process of reintegrating files
from node1

Partially synchronized

2. command stop on node2 then on node1

STOP(NotReady)

STOP(NotRead y)stop of node2 during its reintegration: stopof node2 can be made while a file that is half
copied (corrupted file)node1 is also stopped
Partially synchronized
3. command prim on node2
fails with messages in the log describedabove"Data may be inconsistent for replicated directories (stopped
during reintegration)""If you are sure that this server has valid data, run safekit
primforce to force start as primary"in this case, you must start node1 with startcommand or prim command. And to restartnode2 with start command to finish
reintegration of files. While node2 is not in
the state
SECOND (Ready), its data may
be corrupted
if you absolutely want to start as primary on
node2 partially reintegrated and with data
potentially corrupted, use the commandsafekit primforce –m AM on node2(command line only, where AM is themodule name). Message in the log:

STOP
(NotReady)

STOP(NotRead y)Partially synchronized
The command prim fails since thedata may be corrupted"Action primforce called by SYSTEM/root"

Note: The safekit primforce –m AM command forces a full reintegration of
replicated directories on the secondary when it is restarted.

106

39 A2 38MC 03

Farm module administration

6. Farm module administration
6.1 “Operating mode of a farm module” page 107
6.2 “State automaton of a farm module (STOP, WAIT, UP - NotReady, Transient,
Ready) page 108
6.3 “Start-up of a farm module” page 109
To test a farm module, see section 4.3 page 79.
To analyze a problem, see section 7 page 111.

6.1

Operating mode of a farm module

1. Normal operation

2. Automatic failover

Stable state: 2 active nodes.

Stable state: 1 active node.

On all nodes:✓

Virtual IP is set✓

Application is running✓

Network load sharing is distributed
among all nodes

On remote node stop, automatic failoverof the network load sharing.

Each node is ready to run a failover andtake 100% of the load.
3. Back to normal operation
Stable state: 2 active nodes.

39 A2 38MC 03

107

SafeKit User's Guide

6.2
State automaton of a farm module (STOP, WAIT, UP NotReady, Transient, Ready)Note: This is also the state automation of a light module. A light module is identified by<service mode="light"> in userconfig.xml file under SAFE/modules/AM/conf (where
AM is the module name). The light type corresponds to a module that runs on one node
without synchronizing with other nodes (as can-do mirror or farm modules). A light
module includes the start and stop of an application as well as the SafeKit checkers thatcan detect errors.

108

39 A2 38MC 03

Farm module administration

6.3

Start-up of a farm module

Use the start command on each node running the module. An example with a farm of
2 servers is presented below.

1. initial state

STOP(NotReady)

STOP(NotReady)

UP(Ready)UP(Ready)the farm module has just been configured on
node1 and node2

2. command start on node1 and node2
message in the log of both servers:"farm membership: node1 node2 (group FarmProto)""farm load: 128/256 (group FarmProto)""Local state UP Ready"resource of the module instance on both
nodes: FarmProto 50%

39 A2 38MC 03

109

SafeKit User's Guide110

39 A2 38MC 03

7. Troubleshooting
7.1 “Connection issues with the web console” page 111
7.2 “Connection issues with the HTTPS web console” page 113
7.3 “How to read logs and resources of the module?” page 116
7.4 “How to read the commands log of the server?” page 116
7.5 “Stable module
7.6 “Degraded module(Ready) and ” page 117(Ready)and7.7 “Out of service module//(NotReady)” page 117(NotReady) and/(NotReady)” page 117

7.8 “Module

STOP (NotReady): restart the module” page 118

7.9 “Module

WAIT (NotReady): repair the resource="down"” page 119

7.10 “Module oscillating from

(Ready) to(Transient)” page 120

7.11 “Message on stop after maxloop” page 121
7.12 “Module(Ready) but non-operational application” page 122

7.13 “Mirror module

ALONE (Ready) -7.14 “Farm module

UP(Ready)but problem of load balancing in a farm” page 124

WAIT/

STOP (NotReady)” page 123

7.15 “Problem after Boot” page 124
7.16 “Analysis from snapshots of the module” page 125
7.17 “Problem with the size of SafeKit databases” page 128
7.18 “Problem for retrieving the certification authority certificate from an external
PKI” page 129
7.19 “Still in Trouble” page 132

7.1

Connection issues with the web console

If you encounter problems for connecting to the SafeKit web console to SafeKit node,such as no reply or connection error, run the following checks and procedures:
7.1.1 “Browser check” page 112
7.1.2 “Browser state clear” page 112
7.1.3 “Server check” page 112

Then, it may be necessary to reload the console into the browser.

39 A2 38MC 03111

SafeKit User's Guide

7.1.1

Browser check

For the web browser, check:✓that it is a supported browser and its level✓change the proxy settings for direct or indirect connection to the server✓with Microsoft Edge, change the security settings (add the URL into the trusted
zones)✓

clear the browser's state on upgrade as described below✓that the web console and the server are at the same level (backward compatibility
may not be fully preserved)

7.1.2
✓

Browser state clear

Clear the browser cache
A quick way to do this is a keyboard shortcut that works on IE, Firefox, and Chrome.
Open the browser to any web page and hold CTRL and SHIFT while tapping the
DELETE key. (This is NOT CTRL, ALT, DEL). The dialog box will open to clear thebrowser. Set it to clear everything and click Clear Now or Delete at the bottom✓

Clear the browser SSL cache if HTTPS is used
Look at advanced settings for the browser and search for SSL cache.

Finally close all windows for the browser, stop the browser process still running in the
background if necessary, and re-open it fresh to test what wasn't working for you
previously.

7.1.3

Server check

On each SafeKit cluster node check:✓the firewall
If this has not yet been done, run the SAFE/safekit firewallcfg add command
which configures the operating system firewall. For other firewalls, add an exception
to allow connections between the web browser and the server. For details, see section
10.3 page 158.

✓

the web server configuration
HTTP access to the web console requires authentication. If it has not yet been done,
run the SAFE/bin/webservercfg -passwd pwd to initialize (or reinitialize) thisconfiguration with the password of the user admin. For details, see 11.2.1 page 177.

✓

the network and the server availability✓

the safeadmin and safewebserver services
They must be started.

✓

the SafeKit cluster configuration
Run the command safekit cluster confinfo (see section 9.3 page 144). This
command must return on all nodes, the same list of nodes and the same value for theconfiguration signature. If not, reapply the cluster configuration on all nodes (seesection 12.2 page 205).

112

39 A2 38MC 03

Troubleshooting

7.2

Connection issues with the HTTPS web console

If you encounter problems for connecting the secure SafeKit web console to SafeKit
nodes, you can run the following checks and procedures:

7.1 “Connection issues with the web console” page 111
7.2.1 “Check server certificate” page 113
7.2.2 “Check certificates installed in SafeKit” page 115
7.2.3 “Revert to HTTP configuration” page 115

7.2.1

Check server certificates

The SafeKit web console connects to a SafeKit node that is identified by a certificate. To
get the SafeKit node certificate content with Internet Explorer or Chrome, run thefollowing:

1.

Click on the lock next to the URL
to open the security report

2.

Click on the View certificates link.
It opens a window that displays
the certificate content

39 A2 38MC 03

113

SafeKit User's Guide

3.

Check the issuer that must be theappropriate certification authority

4.

Check the validity date and theworkstation date. If necessary,change the workstation date

5.

Check the validity date. If thecertificate is expired, you must renew.
For certificate generated with the
SafeKit PKI, see section 11.3.1.9.1
page 186

6.

Click on Details tab

7.

Select Subject Alternate Name field. Its
content is displayed into the bottom
panel. The location set into the URL forconnecting the SafeKit web console must
be included into this list. Change the URL
if necessary

8.

The address value for the node, set into
the SafeKit cluster configuration, must
be one of the values listed. If it is not,change the cluster configuration as
described in 12.2 page 205.
When using DNS name, you must use
lower case.
With SafeKit <= 7.5.2.9, the server’s
name must be included.

114

39 A2 38MC 03

Troubleshooting

7.2.2

Check certificates installed in SafeKit

You can use the checkcert command for checking all the certificates.
On each SafeKit nodes:
1.

Log as administrator/root and open a command shell window

2.

Change directory to SAFE/web/bin

3.

Run checkcert -t all
It checks all installed certificates and returns a failure if an error is detected

4.

You can check that the server certificate contains some DNS name or IP address
with:checkcert -h ”DNS name value”
checkcert -i ”Numeric IP address value”

The server certificate must contain all DNS names and/or IP addresses used for
HTTPS connection. These ones must also be included into the SafeKit cluster
configuration file.

7.2.3

Revert to HTTP configuration

If the problem cannot be solved, you can revert to the HTTP configuration (where
SAFE=C:\safekit in Windows if System Drive=C: ; and SAFE=/opt/safekit in Linux):
On S1 and S2:
remove the file
SAFE/web/conf/ssl/httpd.webconsolessl.conf
On S1 and S2:
run safekit webserver restart

You must then clear the browser cache as described in 7.1.2 page 112.

39 A2 38MC 03

115

SafeKit User's Guide

7.3

How to read logs and resources of the module?

Module log and Scripts log for the module on
one node may be analyzed with (replace below
node1 by the node name and AM by the modulename):✓

the web console at URI /console/en/monitoring/modules/AM/nodes/node1/logs✓

the command executed on node1
safekit logview –m AM for the module log✓

on node1, into files
SAFEVAR/modules/AM/userlog_<year>_<month>_<day>T<time>_<script name>.ulog
for the scripts log

With the module log, you can understand why
the module is no longer in its stable state(Ready).
With the scripts log, you can see the output
messages of module scripts (start_xxx and
stop_xxx).

You will find a list of SafeKit log
messages in Log Messages Index
page 309.
Messages in the log after anadministrator command are:"Action start called by
web@<IP>/SYSTEM/root""Action stop called by
web@<IP>/SYSTEM/root""Action restart called by
web@<IP>/SYSTEM/root""Action swap called by
web@<IP>/SYSTEM/root""Action stopstart called by
web@<IP>/SYSTEM/root""Action forcestop called by
web@<IP>/SYSTEM/root"
web@<ip>: via the SafeKit console
SYSTEM: command on Windows
root: command on Linux

If "Stopping loop" appears in themodule log, see section 7.11
page 121

Note that a module can leave its stable(Ready)because of an administrator
command: safekit stop | restart | swap |stopstart | forcestop… -m AM

Module status
Resources state of the module on one node
may be analyzed with (replace below node1 by
the node name and AM by the module name):✓

the web console at URI /console/en/monitoring/modules/AM/nodes/node1/resources✓

the command executed on node1
safekit state –m AM -v

state.local, state.remoteusersetting.errd,
usersetting.checker,
usersetting.encryption
Checkersproc.xxx, intf.xxx, custom.xxx
File replication
rfs.uptodate, rfs.degraded,
rfs.reintegre_failed

7.4

How to read the commands log of the server?

There is a log of the safekit commands ran on the server.
Commands log may be displayed using:✓

the command safekit cmdlog
See section 10.9 page 172 for more details.

116

39 A2 38MC 03

Troubleshooting

7.5

Stable module(Ready) and(Ready)A stable mirror module on 2 servers is in the state
PRIM (Ready) SECOND(Ready): the application is running on the PRIM server; on failure, the SECOND server isready to resume the application.
A stable farm module is in the state
application is running on all servers.

7.6

Degraded module

UP (Ready)on all servers of the farm: the(Ready)and/(NotReady)A degraded mirror module is in the state
ALONE (Ready)STOP/ WAIT
(NotReady). There is no recovery server, but the application is running on the ALONE
server.
A degraded farm module is in the state

UP (Ready)on at least one server of thefarm, the other servers being in the stateis running on the UP server.

STOP/

WAIT (NotReady). The application

In the degraded case, there is no emergency procedure to implement. Analysis of thestate
STOP/ WAIT (NotReady)can be done later. However, you can attempt torestart the module in a stable state:see 7.8 “Module

STOP (NotReady): restart the module” page 118

see 7.9 “Module

WAIT (NotReady): repair the resource="down"” page 119

7.7

Out of service module/(NotReady) and/(NotReady)An out of service mirror or farm module is in the state
STOP/ WAIT (NotReady)onall servers. In this case, the application is not operational on any server anymore. You
must restore the situation and restart the module in(Ready)on at least one server:see 7.8 “Module

STOP (NotReady): restart the module” page 118

see 7.9 “Module

WAIT (NotReady): repair the resource="down"” page 119

39 A2 38MC 03

117

SafeKit User's Guide

7.8

ModuleSTOP (NotReady): restart the module

Restart the stopped module (replace below AM by the module name) with:✓the web console via

Monitoring/✓the command safekit start –m AM executed on the node

Check that the module becomeson the node/Start/(Ready).

Analyze results of start in the module and scripts logs (replace below node1 by thenode name and AM by the module name) with:✓the web console at URI /console/en/monitoring/modules/AM/nodes/node1/logs✓

the command safekit logview –m AM on node1, for the module log✓

the files SAFEVAR/modules/AM/userlog_<year>_<month>_<day>T<time>_<scriptname>.ulog on node1, for the scripts log

118

39 A2 38MC 03

Troubleshooting

7.9

ModuleWAIT (NotReady): repair the resource="down"
Messages from wait checkers:
files not up-to-date locally: see section 5
page 95

If the module is in the state
WAIT(NotReady), it waits for the state of a
resource to become up.
You must identify and fix the problem
that caused the resource state to go
down.

"Data may be not uptodate for replicated directories(wait for the start of the remote server)""Action wait from failover rule notuptodate_server""If you are sure that this server has valid data, runsafekit prim to force start as primary"<interface check="on"> checker of a
local network interface

To determine the resource involved,
analyze the module log and resources(see 7.3 page 116).

"Resource intf.ip.0 set to down by intfcheck""Action wait from failover rule interface_failure"

Notes:"Resource ping.id set to down by pingcheck""Action wait from failover rule ping_failure"

A wait checker is started after theprestart script and stopped before
poststop.
The checker is active on all servers
ALONE/PRIM/SECOND/UP (Ready).
The action of the checker upondetecting an error is to set a resource
to down.

<ping> checker of an external IP<module> checker of another module"Resource module.othermodule_ip set to down by
modulecheck""Action wait from failover rule module_failure"<tcp ident="id" when="pre"> checkerof an external TCP service"Resource tcp.id set to down by tcpcheck""Action wait from failover rule tcpid_failure"<custom ident="id" when="pre">customized checker

A failover rule referencing the resource
performs the stopwait action.
The module is locally in state
WAIT (NotReady)while the resource
stays down.
The module exits the
WAIT(NotReady) state as soon as thechecker sets the resource back to up.

"Resource custom.id set to down by customscript""Action wait from failover rule customid_failure"<splitbrain> checker“Resource splitbrain.uptodate set to down by
splitbraincheck"…
"Action wait from failover rule splitbrain_failure"

Files not up-to-date locally due to splitbrain: see section 13.17 page 262

39 A2 38MC 03

119

SafeKit User's Guide

7.10

Module oscillating from

(Ready) to(Transient)If a module oscillates from state(Ready)to state(Transient), it isprobably a victim of a restart or
stopstart checker which detects a
constant error.
By default, after the 4th unsuccessful
restart on a server, the module stops, andthe server stabilizes in
STOP(NotReady).
Use the module log to determine whichchecker is the source of the logs (to readlogs, see section 7.3 page 116).
Notes:A restart or stopstart checker isdefined in userconfig.xml by:✓<errd> in userconfig.xml
checker of processes"Process appli.exe not running""Action restart|stopstart called by errd"<tcp ident="id"when="prim"|"both"> inuserconfig.xml

when="prim" for a mirror module
The checker is started on the node
PRIM/ALONE (Ready)after script
start_prim (stopped before
stop_prim). It checks the application
started in start_prim.

✓

Messages from restart or stopstart
checkers:

TCP checker of the application"Resource tcp.id set to down by tcpcheck""Action restart|stopstart from failover ruletcp_failure"<custom ident="id"when="prim"|"both"> inuserconfig.xml

when="both" for a farm module
The checker is started on all nodes
UP (Ready)after script start_both(stopped before stop_both). It checks
the application started in start_both.

The action of a checker on an error is torestart or stopstart the module. stopstarton
PRIM (Ready)leads to a failover ofthe primary on the other node.

custom checker"Resource custom.id set to down by customscript""Action restart|stopstart from failover rulecustomid_failure"or"Action restart|stopstart called by customscript"

The module is in the state
PRIM/UP(Transient)during the application
restart.
After several oscillations, the module
stops with "Stopping loop" in the module log:
see section 7.11 page 121120

39 A2 38MC 03

Troubleshooting

7.11

Message on stop after maxloop

If an error detected by a checker repeats
itself several times and successively, themodule is stopped on the server in
STOP(NotReady): because the error ispermanent, and the action of the checker
cannot correct it
If in userconfig.xml, there is no
parameter maxloop / loop_interval in<service>, by default, maxloop="3"loop_interval="24"
if the checkers generate more than 3
unsuccessful restarts (restart, stopstart,
stopwait) in less than 24H, then stop ofmodule:STOP(NotReady).

Message on stop after maxloop"Stopping loop"

The counter is reset to 0 if an
administrator executes an action on themodule such as safekit start –m AM(replace AM by the module name) or
safekit stop –m AM (without the option–i <identity>)

39 A2 38MC 03

121

SafeKit User's Guide

7.12

Module(Ready) but non-operational application

If a server has a status of
PRIM(Ready)or
ALONE(Ready)or
UP(Ready), theapplication can be non-operational because of undetected errors on start-up. In thefollowing, replace node1 by the node name and AM by the module name.
Check the output messages of application scripts coming fromstart_prim/start_both and stop_prim/stop_both. They are visible in (replace
below node1 by the node name and AM by the module name) with:✓the web console at URI/console/en/monitoring/modules/AM/nodes/node1/logs✓

the files
SAFEVAR/modules/AM/userlog_<year>_<month>_<day>T<time>_<scriptname>.ulog, on node1, for the scripts log

Check if there are errors during start or stop of the application. Be careful,sometimes the userlog is disabled because it is too large with <userlogging="none"> in userconfig.xml of the module.
Check application scripts start_prim(/both) and stop_prim(/both) of amirror(/farm) and userconfig.xml with:✓the web console at URI /console/en/configuration/modules/AM/config✓

under the directory SAFE/modules/AM on the node1

Execute a restart of the
PRIM/ALONE/UP(Ready)node to stop and restart locallythe application (without failover) with:✓the web console via

Monitoring/on the node/Restart/✓

the command safekit restart –m AM executed on the node (replace AM by
the module name)

If the application is still non-operational, apply a stop
PRIM/ ALONE /UP(Ready)node to stop and the application (stopstart makes a failover if the othernode is Ready) with:122✓

the web console via

Monitoring/on the node/

Stop/

✓

the command safekit stop –m AM executed on the node

39 A2 38MC 03

Troubleshooting

7.13

Mirror module

ALONE (Ready) -WAIT/

STOP (NotReady)If a mirror module stays in state
ALONE(Ready)WAIT(NotReady), check theresource state.remote on each node (to read resources, see section 7.3 page 116). If
this state is UNKNOWN on the two nodes, there is probably a communication problem
between the nodes. This problem may also lead to(NotReady).

ALONE(Ready)-

STOP

Possible root causes are:
Real network problem
Check your network configurations on the two nodes.
Firewall rules on one or the two nodes
For details, see section 10.3 page 158
Not the same SafeKit cluster configuration or cluster cryptographic keys
To communicate, cluster nodes must belong to the same cluster and have the same
configuration (see section 12 page 203):✓

The web console warns if nodes in the cluster nodes list have not an identicalconfiguration✓

The command: safekit cluster confinfo on any nodes of the cluster must
report an identical configuration signature for all nodes of the cluster (see 9.3
page 144)

If the cluster configuration is not identical, re-apply the cluster configuration on all
cluster nodes as described in 3.2.2 page 42.
Not the same module cryptographic keys
When cryptographic has been enabled for the module, the resource
usersetting.encryption is “on” (to read resources, see section 7.3 page 116). If
the nodes do not have the same keys for the module, the nodes will not be able tocommunicate for the internal module communications.
To distribute the same module cryptographic keys, re-apply the module
configuration on all nodes.
See section 10.5 page 164 for details.
Expired cryptographic keys
In SafeKit <= 7.4.0.31, the key for encrypting the module communication has avalidity period of 1 year. When it expires in a mirror module with file replication, thesecondary fails to reintegrate and the module stops with an error message into thelog:reintegre | D | XXX clnttcp_create: socket=7 TLS handshake failed
In SafeKit > 7.4.0.31, the message is:reintegre | D | XXX clnttcp_create: socket=7 TLS handshake failed.
Check server time and module certificate (expiration date, hash)To solve this problem, see 10.5.3.1 page 165

39 A2 38MC 03

123

SafeKit User's Guide

7.14
Farm module
farm

UP(Ready)but problem of load balancing in a

Even though all servers in the farm are

7.14.1

UP(Ready), load balancing is not working.

Reported network load share are not coherent

In a farm module, the sum of the network load share of allnodes must be equal to 100%.

UP(Ready), module

If it’s not the case, there is probably a communication problem between module nodes.
Possible root causes are the same as for a mirror module. See section 7.13 page 123
for possible solutions.
See also section 4.3.6 page 83

7.14.2

virtual IP address does not respond properly

If the virtual IP does not respond properly to all requests for connections:choose a node in the farm that receives and processes connections on the virtual IP
address (established TCP connections):✓in Windows, use the command netstat –an | findstr <virtual IP
address>✓in Linux, use the command netstat –an | grep <virtual IP address>

stop the farm module on all nodes except the one that receives connections andthat remains
UP(Ready) with:✓the web console via

Monitoring/on the node/

Stop/

✓

the command safekit stop -m AM (replace AM by the module name)check that all connections to the virtual IP address are handled by the single server
UP (Ready)For a more detailed analysis on this topic, see:
4.3.4 “Test virtual IP address of a farm module” page 80
4.3.5 “Test TCP load balancing on a virtual IP address” page 82
4.3.7 “Test compatibility of the network with invisible MAC address” page 84

7.15

Problem after Boot

If you encounter a problem after boot, see section 4.1 page 69.
Note that by default, modules are not automatically started at boot. For this, you mustsetup the boot start into the module’s configuration with:✓the web console at /console/en/configuration/modules/AM/config✓

in file SAFE/modules/AM/conf/userconfig.xml on the node1, with the bootattribute of the service tag (see section 13.2.3 page 211)

Then apply the new configuration on all nodes.

124

39 A2 38MC 03

Troubleshooting

7.16

Analysis from snapshots of the module

When the problem is not easily identifiable, it is recommended to take a snapshot of themodule on all nodes as described in section 3.5 page 65. A snapshot is a zip file thatcollects, for one module, the configuration files, dumps, ... Its content allows an offline
and in-depth analysis of the module and node status.
The structure and content of the snapshot varies depending on the version
of SafeKit.

Since SafeKit 8.1, the structure of the snapshot is as follows:
snapshot_nodename_AM
Snapshot for the module AM get from the node
named nodename
AM
Application module nameconfig_year_month_day_hour_mn_sec
Last 3 configurations for the module, including
the current one
dump_year_month_day_hour_mn_sec
Last 3 dumps for the module, including the lastone

for the level 3 support

7.16.1Module configuration files

The module configuration files are saved as follows:
module directory contains the user
configuration files
bin directory
scripts start_xx, stop_xx, …
conf directory
XML configuration userconfig.xml

Check the user configuration file and scripts for troubleshooting with the application
integration into SafeKit

39 A2 38MC 03

125

SafeKit User's Guide

7.16.2Module dump files

The dump contains the state of the module and the SafeKit node as it was at the time ofthe dump.
csv directorylogs and status in csv format
licences directory
SafeKit licenses get from SAFE/conf
directory
userlog directory: module scripts logs
var directory
Extract of the SAFEVAR directory
web directory
web server configuration gets from
SAFE/web/conf directory
Module logs (not verbose and verbose)
Information file
Various information about the node (listand status of installed modules, OSversion, disk, and network configuration,...)
System logs
last.txt and systemevt.txt in LinuxOr

Or

applicationevt.txt and systemevt.txt
in Windows

Commands log for the node

Trace files for level 3 support

Check the license file(s) into licenses directory for troubleshooting with the SafeKit
license check126

39 A2 38MC 03

Troubleshooting

Check the Apache configuration files into web directory for troubleshooting with the
SafeKit web service
Check the module logs, in log.txt and logverbose.txt, for troubleshooting with themodule behavior
Check the module scripts logs
userlog/userlog_<year>_<month>_<day>T<time>_<script name>.ulog fortroubleshooting with application start/stop
If necessary, look at heartplug file for some information on the node and search thesystem logs for events that occurred at the same time as the problem being analyzed
Check the commands log commandlog.txt for troubleshooting with cluster
management or distributed commands
7.16.2.1 var directory
The var directory is mainly for the level 3 support. It is a copy of some part of the
SAFEVAR directory. In the var/cluster directory:
look at the cluster.xml file for checking the cluster configuration
look at the cluster_ip.xml file for checking the DNS name resolution of names into
the cluster configuration
7.16.2.2 csv directory
The logs and reports are also exported into csv format in the csv directory:

Logs and status of the module
Verbose log
Resources status
Resources status history
Logs and status of the node
Commands log
List of installed modules
For the level 3 support
For the level 3 support

Import the csv files into an Excel sheet to facilitate their analysis
To import a file:
1. Create a new sheet
2. From the Data tab, import From Text/CSV

39 A2 38MC 03

127

SafeKit User's Guide

3. In the dialog box, locate and double-click the csv file to import, then click
Import4. Then click on Load

You can use the Excel features to filter rows according to the level of the messages, ...
and load in different sheets the csv of each node.
For the exact date, format cells with Number/Custom jj/mm/aaaa
hh:mm:ss,000

7.17

Problem with the size of SafeKit databases

SafeKit uses SQLite3 storage to save:
The log and the status of the node✓

SAFEVAR/log.db contains the commands log✓

SAFEVAR/resource.db contains the list of installed modules and its history

These are referred to as node databases.

128

39 A2 38MC 03

Troubleshooting

The log and the resources of the module✓

SAFEUSERVAR/log.db contains the module log✓

SAFEUSERVAR/resource.db contains the state of the module resources and its
history

These are referred to as module databases.
The size of the logs and histories increases as events occur on the SafeKit node andmodules. Therefore, they should be purged regularly by deleting the oldest entries. Thisis automatically done thanks to a periodic job (task scheduler in Windows; crontab inLinux) that is controlled by the safeadmin service. The clean of the node databases isalways active. The clean of the module databases is active only when the module isrunning. To check that the jobs are ready:Job for cleaning node databases✓

In Windows, run schtasks /QUERY /TN safelog_clean✓

In Linux, run crontab -u safekit -l
The output of this command must contain the safelog_clean entry

Job for cleaning AM module databases (where AM is the module name)✓

In Windows, run schtasks /QUERY /TN safelog_AM✓

In Linux, run crontab -u safekit -l
The output of this command must contain the safelog_clean_AM entry

The clean-up is implemented by a script located into SAFEBIN (in Linux,
SAFEBIN=/opt/safekit/private/bin; in Windows, SAFE=C:\safekit\private\bin - if
%SYSTEMDRIVE%=C:):dbclean.ps1 in Windows
anddbclean.sh in Linuxdbclean.ps1 AM in Windows
anddbclean.sh AM in Linux

Clean the log and history in the node databases

Clean the log and history in the databases of themodule named AM

If necessary, you can run this script outside the scheduled period to force the databases
clean-up.

7.18
Problem for retrieving the certification authority certificate
from an external PKI
When using an external PKI, you must provide the certificate of the certification
authority CA used to issue server certificates (cacert.crt file containing the chain ofcertificates for the root and intermediates Certification Authorities)
If you have trouble retrieving these files from an external PKI, you can build them using
the procedure described below.

39 A2 38MC 03

129

SafeKit User's Guide

7.18.1

Export CA certificate(s) from public certificates

The following procedure explains how to build from a public certificate, the chain ofcertificates for the root and intermediates Certification Authorities, into the filecombined.cer.
When you have the public certificate (.crt or .cer file in Base-64 encoded X.509 format)
generated by the PKI:
1. Copy the .crt (or .cer) file on a Windows workstation
2. Double click on this file to open it with “Crypto Shell Extensions”
3. Select the “Certification Path” tab to view the tree of certification authorities
4. Select an entry (from top to down except the leaf)

5. Click on “View Certificate”. A new window is opened with details for the selectedcertificate
6. In this new window, select the “Details” tab and click “Copy to File”

7. It opens the Certificate Export Wizard:
a. Click on “Next” to continueb. On the “Export File Format” page, select “Base-64 encoded X.509 (.CER).”,and then click “Next”
130

39 A2 38MC 03

Troubleshootingc. For “File to Export”, “Browse” to the location to which you want to export thecertificate. Fill “File name” with the name of the certificate file. Then, click“Next”
d. Click “Finish” to export the certificate
e. Your certificate is successfully exported

8. Now repeat steps 4-7 for all entries (except the last one) to export all intermediate
CA certificates in the Base-64 encoded X.509(.CER) format. For the example, you
would repeat steps 4-7 on SSSL.com RSA subCA intermediate CA to extract it as its
own certificate.
9. Concatenate all your CA certificates into one file combined.cer
39 A2 38MC 03

131

SafeKit User's Guide

Run the following command with all the CA certificates you extracted earlier:✓

In Windows:
type intermediateCA.cer rootCA.cer > combined.cer✓

In Linux:
cat intermediateCA.cer rootCA.cer >> combined.cer

The resulting combined certificate should look something like the following:This file can be used as the SAFE/web/conf/cacert.crt

7.19

Still in Trouble

See Messages Index page 309
See section 8.5 page 136 for opening a ticket at the call desk132

39 A2 38MC 03

8. Access to Evidian support
8.1 Home page of support site” page 133
8.2 “Permanent license keys” page 134
8.3 “Create an account” page 135
8.4 “Access to your account” page 135
8.5 “Call desk to open a trouble ticket” page 135
8.6 “Download and upload area” page 139
8.7 “Knowledge base” page 140

8.1Home page of support site

https://www.evidian.com/support
Software Keys: get permanent keys
Subscription Request: create an accountDownload: download product or upload snapshots
Call desk: tool for opening a call on problem
Knowledge Base: base of KB
39 A2 38MC 03

133

SafeKit User's Guide

8.2

Permanent license keys

https://www.evidian.com/support/software-keys/
Software Keys: get permanent
keys
Fill-in the form with the delivery
note sent after a purchase order
Take "hostname" and OS of yourservers
To obtain a temporary key forany hostname and any OS, fordetails see section 2.1.5 page 29

134

39 A2 38MC 03

Access to Evidian support

8.3

Create an accounthttps://www.evidian.com/support/
registration/
Subscription Request: create an
account
The procedure must be executed
once with:- Your client identity- Your confidential identity- A unique e-mail address
Note: your identities are sent by
mail if you take an Evidian support
contract
What you will obtain: a user
account and a private password on
the site

8.4

Access to your accounthttps://www.evidian.com/supp
ort/call-desk/
Login on top at right with youridentity and password
Then you have access to allservices of support site

39 A2 38MC 03

135

SafeKit User's Guide

8.5

Call desk to open a trouble ticket

8.5.1

Call desk operations

https://www.evidian.com/support/
call-desk/
Call desk: tool to open a trouble
ticket on problem with 2 main
operations
Create a call
Search for a Call and exchangewith support on a Call

8.5.2

136

1.

Create a call2.

Search and update

3.

Remote access

4.

Report on calls

Create a call

39 A2 38MC 03

Access to Evidian support

In the header, specify the SafeKit version, problem type and priority as well as themodule name and the OS
Summarize the problem and then describe with more details the scenario and the date
and time of the problem
Snapshots of the SafeKit module causing problem are necessary for the analysis. See
next section for attaching snapshots
Create the call by pressing "Submit"

8.5.3

Attach the snapshots

When there is a problem on a SafeKit module, snapshots of the module on all servers arenecessary for analysis
To get snapshots, see section 3.5 page 65
If the snapshots size is smaller than 10 MBytes, you can attach them with the opening of
the call by clicking on "Add"Otherwise, downloading snapshots on the support site may take several minutes. In thiscase indicate in "Remark text" that you download them into your private upload area:
see section 8.6.3 page 140

39 A2 38MC 03

137

SafeKit User's Guide

8.5.4

Answers to a call and exchange with support

All exchanges between the support and the customer are made with "Remarks"
When support adds a remark on a call, the customer is notified by mail. This is the casefor first response of the support after the opening of the call
After consultation of the last remark of support, the customer can add a new remark in
turn
The exchange takes place until the closure of the call by agreement between thecustomer and Evidian support

138

39 A2 38MC 03

Access to Evidian support

8.6

Download and upload area

8.6.1

Two areas of download and upload

https://www.evidian.com/support/downl
oad/
Product download area: area fordownloading SafeKit packages
Private area [client identity]: private
area to upload files

8.6.2

Product download area

Go to <Version
8.2>/Platforms/<Your
platform>/Current versions
Download the SafeKit package
For more information on
installation, documentation,
upgrade, see section 2 page 25

39 A2 38MC 03

139

SafeKit User's Guide

8.6.3

Private upload area

Create a directory
problemfor aUpload snapshots in thisdirectory withFor building snapshots, seesection 3.5 page 65
For attaching snapshots, seesection 8.5.3 page 137

8.7

Knowledge basehttps://support.evidian.com/knowledge_base/Knowledge Base: base of KB
Search for example all articles on the errd
component of SafeKit

140

39 A2 38MC 03

9. Command line interface
9.1 “Distributed commands” page 141
9.2 “Command lines for boot and for shutdown” page 143
9.3 “Command lines to configure and monitor safekit cluster” page 144
9.4 “Command lines to control modules” page 146
9.5 “Command lines to monitor Modules” page 148
9.6 “Command lines to configure Modules” page 149
9.7 “Command lines for support” page 151
The SafeKit command-line interface is provided by the safekit command. To use:
In Windows
1. Open a PowerShell console as administrator
2. Go to the root of the SafeKit installation directory SAFE (by default
SAFE=C:\safekit if %SYSTEMDRIVE%=C:)
cd c:\safekit
3. Run .\safekit.exe <arguments>
In Linux
4. Open a Shell console as root
5. Go to the root of the SafeKit installation directory SAFE (by default
SAFE=/opt/safekit)
cd /opt/safekit
6. Run ./safekit <arguments>

9.1

Distributed commands

Almost all safekit commands can be applied on a list of cluster nodes.
Exceptions are safekit logview, safekit -p and safekit -r commands which can beused only locally.
The distributed command line interface requires the execution of the SafeKit web service
on each node of the list (see section 10.6 page 166).

39 A2 38MC 03

141

SafeKit User's Guide

Execute action on servers specified by the URL list.
URLs must be separated by commas.
Instead of URLs, it is possible to use a comma
separated list of server names as they appear in thecluster.xml file. Associated URLs are automatically built
as https:9453 or http:9010 (depending on
SAFE/web/conf/ssl/ content)
The special syntax –H “*” stands for all the nodes
declared in the cluster.xml admin lan.
safekit -H <url>[,<url,...] <action><arg>

To override protocol and port, use the[<protocol>:<port>] syntax. The ‘:<port>’ part is
optional. Protocol may be ‘http’ or ‘https’. Default port
for http protocol is 9010.
Example: safekit -H
http://192.168.0.2:9010,http://192.168.0.3:9010
module list
safekit –H "*" module list
safekit –H "[http],*" module list
safekit –H "[https:9500],server1,server2"
module list
Deploy the locally installed <module> on the servers
specified -H parameter.
This command performs the following actions:creates <module>.safe from local
SAFE/modules/<module>

safekit[-H <url>[,...]]-E <module>transfers and installs <module>.safe on the list ofservers
if the module was configured locally, configures it on
remote servers
Example: safekit -E farm will export the local farm
module to the list of servers specified in
SAFEVAR/default_cluster.txt (see example above
for syntax of default_cluster.txt)
Deploy the local cluster configuration files on all theservers specified with –H. This command performs thefollowing actions:safekit [-H <url>[,…] -G

Collect the content of the SAFEVAR/cluster directory
Transfer and copy the collected files into the target
servers’ SAFEVAR/cluster directory
Trigger safeadmin configuration reload

142

39 A2 38MC 03

Command line interface

9.2

Command lines for boot and for shutdown

Use the following commands for starting/stopping SafeKit services, configuring services
and modules automatic start/stop on boot/shutdown, stopping all running modules.
In Windows, you may have to apply the procedure described in 10.4 page 163.
safeadmin(Windows)

SafeKit main service mandatory and started automatically atboot. safeadmin can be controlled using the Windows Services
Control Panel applet

service safeadmin
start (Linux)

SafeKit main service mandatory and started automatically atbootsafekit webserver[start | stop |restart]Controls start/stop/restart of the safewebserver service. This
service is used by the web console, module checkers anddistributed command line interface. The command starts the
httpd processes and waits for their start-up

safekit safeagent[start | stop |restart | check]

In Windows : Controls start/stop of the safeagent service that
implements the SafeKit SNMP agent

safekit boot[webon | weboff |webstatus]

Controls the automatic start at boot of the safewebserver
service ("on" or "off"; by default, "on")safekit boot[snmpon | snmpoff| snmpstatus]

In Windows: Controls the automatic start at boot of thesafeagent service ("on" or "off"; by default, "off")
Controls whether the AM module starts automatically at boot ornot ("on" or "off"; by default, "off")
Without the option –m AM, lists the boot status of all modules.

safekit boot [–mAM] [on | off |status]safekit shutdown

39 A2 38MC 03

The boot start of a module can be defined in themodule configuration with the boot attribute of theservice tag in userconfig.xml. This configuration
option makes the safekit boot -m AM on | off
deprecated. However, this is still supported andreplaces the module configuration, provided that theboot attribute is not present or set with the value
ignore.
Stops all running modules

143

SafeKit User's Guide

9.3

Command lines to configure and monitor safekit cluster
Apply the new SafeKit cluster configuration with thecontent of the file passed as argument, cluster.xml orcluster.zip:cluster.xml
configure with new cluster.xml and generate new
cryptographic keys
cluster.zip
configure with the new cluster.xml andcryptographic keys stored into the zip filesafekit cluster config[filepath .xml or .zip][lock | unlock]

When called with no argument, this command keeps thecurrent configuration but generates new cryptographic
keys.
Ex:
safekit cluster config /tmp/newcluster.xml
Use with great care: the new cluster
configuration and cryptographic key mustthen be copied to all cluster nodes tohave the same cluster configuration on all
nodes.
If the command is called with the parameter lock,future safekit cluster config commands will not be
granted until they are called with the unlock
parameter.

safekit cluster confcheckfilepath

144

Check the cluster configuration, with the content of thexml file passed as argument, without applying it

39 A2 38MC 03

Command line interface

Return, for each active cluster node:• the date of last cluster configuration,• the digital signature of last cluster configuration• the state: locked (1) or unlocked (0) status for thecluster configuration
This command allows checking if all node of a cluster
have the same configuration.
Ex:
safekit cluster confinfo

safekit cluster conf info
NodeSignature

Date

Lock

rh6server7

6f1032b11a7b2 … 33e67c 2016-05-20T17:06:45

0

rh7server7

6f1032b11a4e0 … 33e67c 2016-05-20T17:06:45

0

The SafeKit cluster configuration must be
the same on all nodes of a cluster.
Asymmetric cluster configurations are notsupported.
safekit cluster deconfig

Remove the cluster configuration and the cryptographic
key.
Return the global SafeKit modules configuration statesafekit cluster state

For each installed module on each cluster node, thiscommands list:• the node name,• module name,• module mode (farm or mirror)• internal module id number,• date of last module configuration,• digital signature of last configuration
This command list which modules are installed on
which nodes of the cluster. Signature and date of lastconfiguration on each node allow checking that a
module has the same configuration on all nodes, and if
not, which node has the most recent configuration.

safekit cluster genkey

safekit cluster delkey

39 A2 38MC 03

Create cryptographic key for global SafeKit
communication (implemented in the safeadmin
process). The cluster configuration must be
deployed again (with safekit –G) for thiscommand to take effect.
Suppress cryptographic keys for global SafeKit
communication. The cluster configuration must
be applied again (with safekit -G) for thiscommand to take effect.

145

SafeKit User's Guidesafekit –H “[http],*” -G

Redo a name resolution for all names specified incluster.xml and userconfig.xml of modules,
without stopping modules (when possible).
Distributes the local cluster configuration andassociated cryptographic key if it exists, to thetarget nodes specified in the URL list.

safekit –H <url>[,<url>] G

9.4

Ex:
safekit –H
http://192.168.1.1:9010,http://192.168.1.
2:9010 -G

Command lines to control modules

The commands apply to the module named AM, passed as an argument with the -m
option.
safekit start –m AM

Starts the module

safekit waitstart –m AM

Waits for the end of the module startsafekit stop –m AM

Stops the module

safekit waitstop –m AM

Waits for the end of the module stop

safekit waitstate –m AM STOP
| ALONE | UP | PRIM | SECOND

Wait for the required stable state (NotReady or
Ready).
Executes only application stop and start scripts

safekit restart –m AM

For mirror modules, there is no failoveron the other server if the module is PRIM

Mirror modules only
safekit swap [nosync] –m AM

safekit stopstart –m AM

Swaps the roles of primary and secondary nodes.
Use nosync to swap without synchronizing thereplicated directories.
Unlike the safekit restart –m AM command, thesafekit stopstart –m AM command causes acomplete stop of the module followed by a start. If
the module was PRIM, there is a failover of the PRIM
module on the other server
Equivalent to safekit stop –m AM;
safekit start –m AM

146

39 A2 38MC 03

Command line interface

Mirror modules only
safekit prim –m AM

Forces the module to start as primary. It fails if theother server is already primary.
The main use case of this command is described in
section 5.3 page 98
Mirror modules only

safekit second [fullsync] –mAMsafekit forcestop –m AM

Forces the module to start as secondary. It fails if
the other server is not primary.
Use fullsync to force the full synchronization of thereplicated directories.
Forces the module stop even if some resources arefrozenSuspends/resumes the error detection of module
processes defined in <errd> section ofuserconfig.xml

safekit errd suspend –m AM
safekit errd resume –m AM

Useful if you want to stop the application withoutchanging the module state.
The resource variable usersetting.errd reflects thecurrent setting.
Used to stop or start all checkers (interface, TCP, IP,custom, etc.)
Useful for maintenance operation, when man knowsthat some checker will detect a problem becausesome parts of the IT infrastructure will be stopped,and don’t want that Safekit start a failover.

safekit checker off –m AM
safekit checker on –m AM

39 A2 38MC 03

Notes:✓

could be used only on a live module in a stablestate (ALONE, UP, PRIM, SECOND, WAIT)✓

the resource variable usersetting.checker reflects
the current setting✓a side effect of this command is the execution of
the update command.

147

SafeKit User's Guide

Used to dynamically set the failover attribute to on oroff (see section 13.2.3 page 211).
Notes:

safekit failover off –m AM
safekit failover on –m AM✓

could be used only on a mirror live module in astable state (ALONE, PRIM, SECOND,WAIT).

✓

this command must be issued on all machinesbelonging to the same cluster to not have
unexpected results.
the resource variable usersetting.failover reflects
the current setting.

✓

✓

9.5

a side effect of this command is the execution of
the update command.

Command lines to monitor modules

The commands apply to the module named AM, passed as an argument with the -m
option.
safekit level [–m AM]

Indicates the version of SafeKit and the license
With the AM parameter, the "level" script of themodule is called, and its results displayed

safekit state

Displays the status of all modules

safekit state –m AM[–v | -lq]

Displays the status of the AM module
With the verbose option –v, status of all the module
resources are listed: see the usefulness of resourcesin section 7.9 page 119
With the option –lq, the command returns status(and exit code): STOP (0), WAIT (1), ALONE (2),UP (2), PRIM (3), SECOND (4)safekit log –m AM [-s nb][ -A ] [-l en|fr]

Displays the last nb main messages of the AM
module log.
Use -A for displaying all messages (including debugones).
Use -l option for choosing the language, en(glish) or
fr(ench).
Default: –s 300

148

39 A2 38MC 03

Command line interface

safekit logview –m AM [-A][-l en|fr]

View in real time the last main messages of the AM
module log.
Use -A for displaying all messages (including debugones).
Use -l option for choosing the language, en(glish) or
fr(ench).

safekit logview –m AM –s 300[-A ] [-l en|fr]

View in real time the AM module log messages
starting from the last 300 messages

safekit logsave –m AM [-len|fr] [-A] /tmp/f.txt

Save main messages of the AM module log in/tmp/f.txt (absolute path mandatory).
Use -A for saving all messages (including debugones).
Use -l option for choosing the language, en(glish) or
fr(ench).

safekit printi|printe –m AM"message"

Application start/stop scripts can write messages in
the module log with I or E level.

9.6

Command lines to configure modules
Apply changes made in SAFE/modules/AM: userconfig.xml,
start_prim/both or stop_prim/both (mirror/farm)safekit config–m AM

Makes each plug-in defined in userconfig.xml <errd>, <vip>,<rfs>, <user>... considered in the new module configuration
This command could be run on a server in the stable states STOP,ALONE or WAIT (NotReady).
In STOP state all the configuration parameters could be modified.
Some configuration parameters can be changed while the module isrunning in ALONE or WAIT (NotReady) states. This feature is called
dynamic configuration. Parameters that could be dynamicallychanged are reported into section 13 page 209 that describes all
configuration parameters.

safekit module
genkey –m AM

Generates cryptographic keys for the module instances network
exchanges encryption. Considered after the next configuration of
the module.

safekit module
delkey –m AM

Erase cryptographic keys associated with the module. After the
next configuration, module instances network exchanges will be
performed without encryption.

39 A2 38MC 03

149

SafeKit User's Guidesafekit-H<url>[,<url>]-E AM

Distributes the local configuration for the module AM and associatedcryptographic key if it exists, to the target nodes specified in the
URL list.
Ex:
safekit –H
http://192.168.1.1:9010,http://192.168.1.2:9010 –E mirror
Display information on the active and current configuration of themodule AM.
the active configuration is the last configuration successfully
applied. It is in SAFE/private/modules/AM
the current configuration is the one located in
SAFE/modules/AM. It may be different from the active one
when it has been modified and not yet been applied

safekitconfinfo –m AM

This command is useful for checking the configuration of themodule. It displays:
the signature value and a last modification date (Unixtimestamp) for the active configuration
the signature value and last modification date (Unix timestamp)
for the current configuration
When the signature values are different, it means that theconfigurations are not identical and that you may have to apply thecurrent configuration.
You can run this command on all the cluster nodes that implement
the module to check that the configuration of the module isidentical on all nodes.

safekitconfcheck –m AM

Check the module configuration under SAFE/modules/AM without
applying
Installs the AM.safe module file under the AM name[-r] force reinstallation of the module

safekit module
install –m AM[-M id] [-r][AM.safe]

safekit module
package –m AM/…/newAM.safe

150[-M id] forces the installation of the module with the id specified
as module id
AM.safe default location is SAFE/Application_Modules/ andits subdirectories
An absolute path could be used too
If no AM.safe is given, the command search for filemodulename.safe in /Application_Modules/ and its
subdirectories
Packages the AM module in /…/newAM.safe (absolute pathmandatory)
Used by the console to create a backup in
SAFE/Application_Modules/backup/

39 A2 38MC 03

Command line interface

safekit module
uninstall –m AM

Uninstalls the AM module. Deletes the module configuration
directory SAFE/modules/AM

safekit module
list

Lists the names of the installed modules

safekit module
listidLists the names and ids of the installed modules

safekit module
getports –m AM(or –i id)

Lists the communication ports used by the module to communicate
between servers

9.7

Command lines for support
Saves the snapshot of the AM module in/tmp/snapshot_xx.zip (absolute path mandatory)

safekit snapshot –m AM/tmp/snapshot_xx.zip

A snapshot creates a dump and gathers under
SAFEVAR/snapshot/modules/AM the last 3 dumps and last
3 configurations to collect them in a .zip file
To analyze snapshots, see 7.16 page 125
To send snapshots to Evidian support, see 8 page 133
To solve a problem in real time on a server, make a dumpof the AM module

safekit dump –m AM

safekit –r"specialcommand"

39 A2 38MC 03

A dump creates a directory dump
dump_year_month_day_hour_mn_sec on the server side
under SAFEVAR/snapshot/modules/AM. The dump
directory contains the module log and status, as well as
information on the system state and SafeKit processes at
the time of the dump
Calls the special command in SAFEBIN with SafeKit
environment variables set.

151

SafeKit User's Guidesafekit clean [all |log | process |resource] [-m AM]

Clean the logs, the resource file, and the main processesof the module AM.
This command must be used with caution since
it deletes working files and kills processes.

safekit clean log –m AM
Clean the logs (verbose and not verbose logs) of themodule. To be used when these logs are corrupted (e.g.:
errors in log view).
safekit clean resource –m AM
Reinitialize the resource file of the module. To be used
when this file is corrupted (e.g.: errors in resources
display)safekit clean process –m AM
Kill the main processes (heart) of the module. To be used
when the stop and forcestop of the module did notachieve to kill these processes.
safekit clean all –m AM
Default value. Clean log, resource, and process.

9.8

Examples

9.8.1

Cluster configuration with command line

See 12.2.2 page 206.

9.8.2

New module configuration with command line

In the following, replace AM by your module name; replace node1 and node2 by the nameof your cluster nodes set during the SafeKit cluster configuration.
1.

Log as administrator/root and open a command shell window on one node
For instance, log-in node1

2.

Run safekit module install –m AM
SAFE/Application_Modules/generic/mirror.safe
to install a new module named AM, from mirror.safe template

3.

Edit the module configuration and scripts in SAFE/modules/AM/conf and
SAFE/modules/AM/bin

4.

Run safekit module genkey –m AM or safekit module delkey –m AM
to create or delete cryptographic key for the module5.

Run safekit -H "node1,node2" -E AM

152

39 A2 38MC 03

Command line interface

to (re)install the module AM and apply its configuration, which is get from the node
running the command (node1 in this example). It applies it on all listed nodes (node1
and node2).

9.8.3

Module snapshot with command line

The command line the module snapshot is described below. Replace AM by your module
name.
1.

Log as administrator/root and open a command shell window on one node
For instance, log-in node1

2.

Run safekit snapshot -m AM /tmp/snapshot_node1_AM.zip
To save the snapshot of the AM module in /tmp/snapshot_node1_AM.zip (absolute
path mandatory) locally (that is on node1).

Repeat all these commands on the other nodes in the cluster.

39 A2 38MC 03

153

SafeKit User's Guide

154

39 A2 38MC 03

10.Advanced administration
10.1 “SafeKit environment variables and directories” page 155
10.2 “SafeKit processes and services” page 157
10.3 “Firewall settings” page 158
10.4 “Boot and shutdown setup in Windows” page 163
10.5 “Securing module internal communications” page 164
10.6 “SafeKit web service” page 166
10.7 “Mail notification” page 169
10.8 “SNMP monitoring” page 170
10.9 “Commands log of the SafeKit server” page 172

10.1

SafeKit environment variables and directories

10.1.1

GlobalVariable

Description

SAFE(given by safekit –p)

SafeKit installation directory: SAFE=/opt/safekit on
Linux and SAFE=C:\safekit on Windows if
SystemDrive=C:
The license is under SAFE/conf/license.txt

SAFEVAR(given by safekit –p)

SafeKit working files directory:
SAFEVAR=C:\safekit\var on Windows and
SAFEVAR=/var/safekit on LinuxSAFEBIN(given by safekit –p)

SafeKit binary installation directory:
C:\safekit\private\bin on Windows and/opt/safekit/private/bin on Linux. Useful toaccess SafeKit special commands (see 14.4 page 270)

SAFE/Application_Modules

Installable .safe modules directory.
Once a module has been installed, the module islocated under SAFE/modules

39 A2 38MC 03

155

SafeKit User's Guide

10.1.2ModuleVariable

Description

SAFEMODULE

The name of the module. The safekit command no longer
needs the module name parameter (-m AM = -m
SAFEMODULE)

SAFE/modules/AM and
SAFEUSERBIN

Editing a module, named AM, and its scripts is made insidedirectory SAFE/modules/AM. There are userconfig.xml fileand application start and stop scripts start_prim, stop_prim
for a mirror, start_both, stop_both for a farm (online
edition or through the SafeKit console)
After a module configuration, scripts are copied to the runtime
directory SAFE/private/modules/AM/bin: this is the value of
SAFEUSERBIN (do not modify scripts at this place)

SAFEVAR/modules/AM and
SAFEUSERVAR

Module, named A, working files directory(SAFEUSERVAR=SAFEVAR/modules/AM)
Output messages of application scripts are in
SAFEVAR/modules/AM/userlog_year-monthdate_striptname.ulog. To check if there are errors during
start or stop of the application.
Note: the userlog could disabled with <user logging="none">in userconfig.xml
Since SafeKit 8, the file name isuserlog_<year>_<month>_<day>T<time>_<scriptname>.ulog.

SAFEVAR/snapshot/modules/AM

156

Directory of dumps and configurations put in a snapshot of themodule named AM. See section 9.7 page 151 that describes
command lines for support.

39 A2 38MC 03

Advanced administration

The module tree (packaged into a .safe or installed into SAFE/modules/AM) is thefollowing:
AM

Application module nameconf
userconfig.xml

User XML configuration fileuserconfig.xml.template

Internal use only

modulekey.p12

Optional. Internal use only (encryption of
the module internal communications)modulekey.dat

Optional. Internal use only (encryption of
the module internal communications)binprestart

Module script executed on module start

start_prim or start_both

Module script to start the application in
mirror or farm module

stop_prim or stop_both

Module script to stop the application in
mirror or farm module

poststopweb
index.html

Module script executed on module stop

manifest.xml

Obsolete (for the web console < SafeKit
8)Internal use only

Since SafeKit 8, you cannot anymore customize the module quick configuration display(since index.html is obsolete).

10.2

SafeKit processes and services

SafeKit Services

Processes per module

safeadmin (safeadmin
process): main andmandatory service

heart: manages therecovery procedures

vipd: synchronizes a farm ofservers

safewebserver (httpdprocess): service for the
console, for <module>
checkers and the
distributed commands

errd: manages detectionof process deathnfsbox, nfsadmin,
reintegre: file replication andreintegration

39 A2 38MC 03

157

SafeKit User's Guide

SafeKit Services

Processes per module

safeagent (safeagent
process): SafeKit SNMP
agent (optional, windows
only)checkers (ipcheck,intfcheck, …)

See 10.3.3.1 page 159 and 10.3.3.2 page 161 for full details on SafeKit processes nameand ports used.

10.3

Firewall settings

If a firewall is active on the SafeKit server, you must add rules to allow network traffic:
between servers for internal communication (global runtime and module specific)
between servers and workstations running the SafeKit console

10.3.1

Firewall settings in LinuxIf you opted-in for automatic local firewall configuration during SafeKit installation, you
do not have to apply the following procedures.
If you opted-out for automatic local firewall configuration, you must configure the firewall
manually or you may use the safekit firewallcfg command. It inserts (or remove)the firewall rules required by the SafeKit core processes (safeadmin and safewebserver
services) and modules processes to communicate with their peers in the cluster.
Administrators should review conflicts with local policy before applying it.
Add (or delete) the firewalld or iptable firewall rules
for the SafeKit safeadmin and safewebserver
services.
SAFE/safekit firewallcfg add
safekit firewallcfg add
safekit firewallcfg deladd firewall rules for safeadmin andsafewebserver

SAFE/safekit firewallcfg deldelete firewall rules for safeadmin andsafewebserver

158

39 A2 38MC 03

Advanced administration

Add (or delete) the firewalld or iptable firewall rules
for the SafeKit modules.
SAFE/safekit /firewallcfg add AM
add firewall rules for the module named AM
This command must be applied after the
first configuration of the module, and on
next configurations if used ports havechanged (check it with the commandsafekit module getports -m AM).

safekit firewallcfg add AM
safekit firewallcfg del AM

SAFE/safekit firewallcfg del AM
delete firewall rules for the module named AM

10.3.2Firewall settings in Windows

When using the operating system firewall (Microsoft firewall), you may use the safekit
firewallcfg command. It inserts (or remove) the firewall rules required by theprocesses of SafeKit services (safeadmin, safewebserver, safeacaserv, safeagent)and modules processes to communicate with their peers in the cluster.
Administrators should review conflicts with local policy before applying it.
Add (or delete) the Microsoft firewall rules.
safekit firewallcfg add
safekit firewallcfg add
safekit firewallcfg deladd firewall rules for SafeKit core and modules
processes.
safekit firewallcfg deldelete firewall rules for SafeKit core and modules
processes.

10.3.3

Other firewalls

If you use another firewall or want to check rules against local policy, the following lists
processes and ports used by SafeKit services and modules that may be useful toconfigure the firewall.
10.3.3.1 List of processes
10.3.3.1.1

Processes performing local-only network exchanges

Processes for a mirror module✓

errd: manages detection of process death

39 A2 38MC 03

159

SafeKit User's Guide✓

nfsadmin, nfscheck: manage the file replication

Processes for a farm module✓

errd: manages detection of process death✓heart: manages the recovery procedures

10.3.3.1.2

Processes performing external network exchanges

Processes common to all the SafeKit servers, one process by server, started at boot:✓safeadmin service (safeadmin process)main and mandatory administration service✓

safewebserver service (httpd process)web service for the console, for <module> checkers and the distributedcommands✓

safecaserv (httpd process)web service for securing the web console with the SafeKit PKI (optional)✓

In Windows : safeagent service (safeagent process)
SafeKit SNMP v2 agent (optional)

Processes for a mirror module (depending on its configuration):✓heart: manages the recovery procedures✓

arpreroute: manages arp requests (sends ARP packet)✓

nfsbox, reintegre: manage the file replication and reintegration✓

splitbraincheck: manage the splitbrain detection (sends ICMP ping packets)

Processes for a farm module (depending on its configuration):✓

vipd: synchronizes a farm of servers✓

arpreroute: manages arp requests (sends ARP packet)

Processes for a mirror or a farm module depending on checkers configuration:160✓

intfcheck: for checking interface (interface checker configuration
automatically generated when <interface check=on>)✓pingcheck: for pinging an address (<ping> configuration)✓ipcheck: for checking a locally defined ip address (virtual ip checker
automatically generated when <virtual_addr check=on>)✓modulecheck: for checking a SafeKit module (<module> configuration)

39 A2 38MC 03

Advanced administration✓

tcpcheck: for checking a TCP connection (<tcp> configuration)

10.3.3.2 List of ports
The following list ports used by SafeKit services and modules.
10.3.3.2.1

Ports used by services

safeadmin
By default, remote access on UDP port 4800 (to communicate with safeadmin
instances on other SafeKit servers)
For changing this value , see section 12.1.3 page 204.
safewebserver
Local and remote TCP access, by default, on port 9010 for HTTP or port 9453 for
HTTPS. For the ports value definition, see section 10.6 page 166.
This service is accessed locally and from remote SafeKit servers and remote
workstation running the SafeKit console.
safecaserv (optional)
Local and remote access on TCP port 9001 by default. For the port value definition,
see section 11.3.1.9.4 page 189.
This service is accessed locally, and from remote SafeKit servers and remote
workstation running the HTTPS configuration wizard with the SafeKit PKI.
safeagent (Windows only, optional)
Local and remote access on UDP port 3600 by default. For the port value definition,
see section 10.8 page 170.
10.3.3.2.2

Ports used by modules

When a module is configured on a SafeKit server, you can run the command safekit
module getports -m AM to list the external ports used by the module AM. For firewall
configuration, you must configure all SafeKit servers to enable communications targetedat these ports.
The ports values for one module are automatically computed depending on its module id.
Run the command safekit module listid to list all the installed modules with theirname and id.
You can run the command safekit module getports -i ID to list the ports that could
be used by a module that got the id value ID (this command can be run even if themodule is not yet installed, but it will return a superset of the really used port by themodule).
The following gives rules for computing ports values depending on the module id. Whencheckers are configured for the module, you may also need to change the firewall
configuration according to the checkers configuration. You must enable allcommunications on localhost between SafeKit processes.

39 A2 38MC 03

161

SafeKit User's Guide

For a mirror module:✓

Port used by heart
UDP port used for sending heartbeats between SafeKit servers
port=8888 +(id-1)✓

Ports used by rfs (file replication)
TCP port used for replications requests between SafeKit servers
safenfs_port=5600 +(id-1)x4

Example for a mirror module with id 1
safekit module getports -m mirror
List of the ports used by SafeKit
Process
safeadmin
port

Ports
UDP 4800

webconsole
port
TCP 9010
heart
port
UDP 8888
rfs
safenfs_port
TCP 5600
For a farm module✓ Port used by farm
UDP port used for communications between all SafeKit nodes
port 4803 + (id-1)x3
Example for a farm module with id 2
safekit module getports -m farm
List of the ports used by SafeKit
Process
safeadmin
port
webconsole
port
farm
port

Ports
UDP 4800
TCP 9010
UDP 4806

For configured checkers✓

162

Ping checker for mirror or farm module
Change ICMP settings to allow ping at destination to the address defined into
the configuration.

39 A2 38MC 03

Advanced administration

✓

✓

✓

10.4

TCP checker for mirror or farm module
Allow TCP connections at destination to the address defined into the <tcp>configuration if this address is not local.
Module checker
Allow TCP connections at destination to 9010 port of the node running themodule that is checked.
Splitbrain checker
Change ICMP settings to allow ping at destination to the witness defined into
the <splitbrain> configuration.

Boot and shutdown setup in Windows

safeadmin service is configured for automatically starting on boot and stopping on
shutdown. In turn, this service starts modules configured for starting at boot andshutdown all modules.
On some Windows platforms, the safeadmin boot start fails because the network
configuration is not ready, and the modules shutdown does not have time to complete
since the timeout for services shutdown is too short. If you encounter such problems,
apply one of the following procedures.
When using the SNMP agent, adapt the following procedures to set the
manual start of the safeagent service and include its start/stop into SafeKit
start-up (safekitbootstart.cmd) and shutdown (safekitshutdown.cmd)
scripts.

10.4.1

Automatic procedure

You can run the script as follow:
1.

open a PowerShell window as administrator

2.

cd SAFE\private\bin

3.

run addStartupShutdown.cmd

This script sets the manual start for safeadmin service and adds default SafeKit start-up(safekitbootstart.cmd) and shutdown (safekitshutdown.cmd) scripts as part of thecomputer group policy start-up/shutdown scripts. If the script fails, apply the manual
procedure below.

10.4.2

Manual procedure

You must apply the following procedure that uses the Group Policy Object Editor.
1.

set manual start for safeadmin service

2.

start the MMC console with the mmc command line

3.

File - Add/Remove Snap-in Add - "Group Policy Object Editor" – OK

4.

under "Console Root"/"Local Computer Policy"/"Computer Configuration"/"Windows
Settings"/"Scripts (Start-up/Shutdown)", double click on "Start-up". Click on Add
then set for "Script Name:" c:\safekit\private\bin\safekitbootstart.cmd. This
script launches the safeadmin service.

39 A2 38MC 03

163

SafeKit User's Guide

5.

under "Console Root"/"Local Computer Policy"/"Computer Configuration"/"Windows
Settings"/"Scripts (Start-up/Shutdown)", double click on "Shutdown". Click on Add
then set for "Script Name:" c:\safekit\private\bin\safekitshutdown.cmd. This
script shutdowns all running modules.

10.5

Securing module internal communications

You can secure communications for the module between cluster nodes by creating
cryptographic keys associated with the module. By default, these keys are generated by
SafeKit with a “private” certification authority (SafeKit PKI). In SafeKit <= 7.4.0.31, thegenerated key has a validity period of 1 year. See section 10.5.3.1 page 165 for solutions
when the key expires.
Since SafeKit 7.4.0.16, you can also provide your own certificates generated with yourtrusted certification authority (enterprise PKI or commercial PKI). See section 10.5.3.2
page 166 for details.
Since SafeKit 7.4.0.32, the module can be reconfigured with new keys while it is in
ALONE state (dynamic update).
When encryption is not properly configured (e.g.: not the same key on all
cluster nodes of the module), the module internal communications between
nodes are rejected. In this case, the module configuration is not identical on
all nodes. You must apply again the configuration on all nodes.
You can check the configuration by running on each node the commandsafekit confinfo –m AM where AM is the module name (see section 9.6
page 149).
The encryption resource reflects the current communication mode of the module:“on”/”off” when encryption is active/not active. To read resources, see section 7.3 page116. The resource name is usersetting.encryption.

10.5.1

Configuration with the SafeKit Web console

When configuring the module with the SafeKit web console, communication encryption isenabled in the step 3 of the module configuration wizard (see section 3.3.2 page 45).

10.5.2

Configuration with the Command Line Interface

The commands line equivalent for configuring a module, named AM, with cryptographic
key are:1.

Stop the AM module on all nodes

2.

On one node, log as administrator/root and open a command shell window

3.

Run safekit module genkey –m AM

4.

Run safekit –H "server1,server2" -E AM
where server1 and server2 are the nodes that implement the module

164

39 A2 38MC 03

Advanced administration

The commands line equivalent for re-configuring a module without cryptographic key
are:1.

Stop the AM module on all nodes

2.

On one node, log as administrator/root and open a command shell window

3.

Run safekit module delkey –m AM

4.

Run safekit –H "server1,server2" -E AM
where server1 and server2 are the nodes that implement the module

For more details on commands, refer to section 9.6 page 149.

10.5.3

Advanced configuration

10.5.3.1 Advanced configuration with the SafeKit PKIIn SafeKit <= 7.4.0.31, the key for encrypting the module communication has a validity
period of 1 year. When it expires in a mirror module with file replication, the secondary
fails to reintegrate. You must re-configure the module with a new key, as explained in
SK-0084, for reverting to normal behavior. In SafeKit > 7.4.0.31, the validity period has
been set to 20 years.
If you cannot upgrade SafeKit, you can generate new keys with a longer validity period.
For this apply the following procedure:
1.

Stop the AM module on all nodes

2.

On one node, log as administrator/root and open a command shell window

3.

Run safekit module genkey –m AM

4.

Delete the file SAFE/modules/AM/conf/modulekey.p12

5.

Change to the directory SAFE/web/bin

6.

Run ./openssl req -config ../conf/ssl.conf -subj"/O=SafeKiModule/CN=mirror" -new -x509 -sha256 -nodes -days 3650 -newkey
rsa:2048 -keyout pkey.key -out cert.crt
Set the -days value to the validity period you want

7.

Run ./openssl pkcs12 -export -inkey ./pkey.key -in ./cert.crt -name"Module certificate" -out modulekey.p12
This command requires to fill a password. Contact Evidian support to get the correct
value for the password

8.

Delete the files pkey.key and cert.crt

9.

Move the file modulekey.p12 into SAFE/modules/AM/conf

10. Run safekit–H "server1,server2" -E AM

where server1 and server2 are the nodes that implement the module

39 A2 38MC 03

165

SafeKit User's Guide

The module is configured, on the 2 nodes, with the new key and ready to start.

10.5.3.2 Advanced configuration with an external PKI
Since SafeKit 7.4.0.16, you can provide your own key generated with your trusted
certification authority (enterprise PKI or commercial PKI). For this apply the following
procedure:
1.

Stop the AM module on all nodes

2.

On one node, log as administrator/root and open a command shell window

3.

Run safekit module genkey –m AM

4.

Delete the file SAFE/modules/AM/conf/modulekey.p12

5.

Append the X509 certificate in PEM format, for your certification authority (certificateof the CA or certificate bundle of all the certificate authorities) to the file
SAFE/web/conf/cacert.crt

6.

Change to the directory SAFE/web/bin

7.

Generate your certificate with the PKI with the subject set to"/O=SafeKiModule/CN=mirror"

8.

Copy the generated files pkey.key and cert.crt into the directory SAFE/web/bin

9.

Run ./openssl pkcs12 -export -inkey ./pkey.key -in ./cert.crt -name"Module certificate" -out modulekey.p12
This command requires to fill a password. Contact Evidian support to get the correct
value for the password

10. Delete the files pkey.key

and cert.crt

11. Move the file modulekey.p12 into SAFE/modules/AM/conf
12. Run safekit–H "server1,server2" -E AM

where server1 and server2 are the nodes that implement the module
The module is configured, on the 2 nodes, with the new key and ready to start.

10.6

SafeKit web service configuration

SafeKit comes with a web service, safewebserver, which runs on each SafeKit server. It
is a standard Apache web service that is mandatory for running:
the web console (see section 3 page 37)
the distributed command line interface (see 9.1 page 141)
the <module> checkers (see 13.16 page 260)
safewebserver starts automatically at the end of SafeKit package install and on server
reboot. If you do not need the SafeKit web service and want to remove the automaticboot start, refer to section 9.2 page 143.

166

39 A2 38MC 03

Advanced administration

The default configuration is HTTP with file-based authentication, initialized with a single
admin user that got the Admin role. This could be changed via configuration files.

10.6.1

Configuration files

The configuration of an instance of safewebserver on a SafeKit server is contained in the
SAFE/web/conf directory. It consists in standard Apache configuration files (seehttp://httpd.apache.org). The configuration is split into many files, but for most common
configurations, only the main configuration file httpd.conf need to be modified.

After changes, you have to restart the service with the command: safekit
webserver restart (see section 9.2 page 143).
Do not edit .default files since they are backups of delivered configuration
files.
The httpd.conf file consists essentially in a set of Define statements. Comment
character # disables the definition.
The mains Define are:Connection port definition:
Define httpport 9010
Define httpsport 9453
Set the listening port in http and https mode. (See section 10.6.2 page 168 forusage).
User authentication definition:
Define usefile# Define useldap# Define useopenid…
Select which user authentication to use. At most one must be defined. usefile
is the default. (See section 11.4 page 193 for details.)
Apache logging definition:#Define Loglevel info
#Define accesslog
Uncomment these lines to enable the logging for debug purposes. Logging files
httpd.log and access.log are in SAFEVAR.

39 A2 38MC 03

167

SafeKit User's Guide

Session validity period definition:
Define SessionMaxAge 28800
Since SafeKit 8.2.1, the user is automatically logged out after 8 hours ofinactivity (28800 seconds). If necessary, adjust this value.

Other Define are self-documented in the httpd.conf file.
The other configuration files are listed below. Modifying one of them may cause problems
when upgrading SafeKit :Global configuration

httpd_main.conf

File based authentication androle mapping

httpd.webconsolefileauth.conf

Form authentication
configuration

httpd.webconsoleformauth.conf

LDAP/AD authentication
configuration

httpd.webconsoleldap.conf

OpenID Connect authentication
configuration

httpd.webconsoleopenidauth.conf

HTTPS configuration

using a LDAP/AD serverusing an OpenID connect identity provider
httpd.webconsolessl.conf
in SAFE/web/conf/ssl

User authentication configurations may optionally use group.conf (for HTTP) or
sslgroup.conf (for HTTPS) files in SAFE/web/conf for user to role mapping.

10.6.2Connection ports configuration

By default, connect the web console with the URL http://host:9010. The SafeKit web
server will redirect to the appropriate page according to your security settings.
If you need to change the default value:
1. Edit SAFE/web/conf/httpd.conf and change the value of httpport or httpsportvariables.
2. Restart the service using the command safekit webserver restart.

The HTTP and HTTPS configurations cannot be active simultaneously. See 11.3 page 180
for how to configure HTTPS.

168

39 A2 38MC 03

Advanced administration

The port value 9010(HTTP)/9453(HTTPS) is also used by the module checker. Therefore,
if the configuration of a module defines a <module> checker:
1. Edit the module configuration file userconfig.xml
2. Edit the port attribute and assign it to the new port value<check><module name="mirror"><to addr="192.168.1.31" port="9010"/></module></check>
3. Apply the new configuration of the module

10.6.3

HTTP/HTTPS and user authentication configuration

The default configuration is for HTTP.
The default configuration is also set with file-based authentication, initialized with asingle admin user that got the Admin role.
The HTTPS configuration requires the installation of certificates and the definition ofuser authentication.
For a detailed description, see section 11 page 175.
To re-enable the HTTP configuration if it has been changed to HTTPS see 11.2.1.1 page
177.

10.6.4

SafeKit API

Use Swagger UI to visualize and interact with the SafeKit API provided by the SafeKit
web service. For this, connect a browser at the URL http://host:9010/swaggerui/index.html. It may be useful to debug issues with the SafeKit web console and/or API.

10.7

Mail notification

You may need to send a notification, such as an e-mail, when the module is started,
stopped, or run a failover. This is implemented thanks to the scripts of the module.
For mail notification, you have first to choose a command line program to send mail. In
Windows, you can use the Send-MailMessage from the Microsoft Powershell Utility. For
Linux, you can use the mail command.
Notification on the start and the stop of the module
The module scripts prestart/poststop can be used for sending a notification on thestart/stop of the module.
Notification on the failover of the module
The module script transition can be used to send a notification on main local statetransitions of the module running on the local server. For instance, it may be useful
to know when the mirror module is going ALONE (on failover for instance).

39 A2 38MC 03

169

SafeKit User's Guide

For details on module scripts, see 14 page 267.
For a full example with the demonstration module notification.safe, see section 15.14
page 289. Since SafeKit 8, this .safe is delivered with the SafeKit package.

10.8

SNMP monitoring

SafeKit could be monitored by snmp. Since version 8, snmp monitoring implementationdiffers in Windows and Linux : In Windows, SafeKit use its own snmp agent service,when in Linux, the operating system’s snmp agent is used.

10.8.1

SNMP monitoring in Windows

For using the SafeKit SNMP agent safeagent, you must:
1. configure it to start on boot, with the commandsafekit boot [snmpon |snmpoff | snmpstatus]

Controls the automatic start at boot of thesafeagent service ("on" or "off"; by default, "off")

2. add the corresponding firewall rule
When using the operating system firewall, the firewall has already been configured forsafeagent if you have applied the command:
SAFE/safekit firewallcfg add

3. start it with the commandsafekit safeagent [start |stop | restart | check]Controls start/stop of the safeagent service that
implements the SafeKit SNMP agent.

The configuration of the safeagent is defined in the self-documented
SAFE/snmp/conf/snmpd.conf file. It is a standard net-snmp configuration file as
described in http://net-snmp.sourceforge.net. By default, the service is listening on UDP
agentaddress port 3600 and accepts read request from the public community and write
requests from the private community. Read requests are used to get module status andwrite requests to run actions on the module.
You can change the default configuration according to your needs. When you modify
snmpd.conf, you must manually change the firewall rule and restart the service to load
the new configuration with: safekit safeagent restart

10.8.2

SNMP monitoring in Linux

Since version 8.0, Safekit did not come with its own snmp agent anymore, so thefollowing safekit commands are obsoleted in Linux: safeagent install, safeagent start,
safeagent stop, boot snmpon, boot snmpoff, boot snmpstatus.
Instead, it is possible to configure the standard snmpd Linux agent to access safekit mib:
170

39 A2 38MC 03

Advanced administration

1. Install net-snmp
dnf install net-snmp net-snmp-utils
2. If selinux is in enforced mode, you have to set snmpd in permissive mode for snmp
by :semanage permissive -a snmpd_t
3. If firewall is active, you have to open the snmp ports with:
firewall-cmd --permanent --add-service snmp
firewall-cmd --reload
4. Edit /etc/snmp/snmpd.conf
Add the following lines :pass .1.3.6.1.4.1.107.175.10 /opt/safekit/snmp/bin/snmpsafekit
view systemview included .1.3.6.1.4.1.107.175.10
Note : the “view systemview” line set the access rights. You could have to adapt it toyour general snmpd configuration.
5. Enable and Start the snmp agent
systemctl enable snmpd
systemctl start snmpd

10.8.3The SafeKit MIB

The SafeKit MIB is common to Windows and Linux implementation. It is delivered in
SAFE/snmp/mibs/safekit.mib .
The SafeKit MIB is accessed with the following identifier (OID, prefix of SafeKit SNMP
variables): = enterprises.bull.safe.safekit (1.3.6.1.4.1.107.175.10).
The SafeKit MIB defines:
The module table: skModuleTable
The index on the module table is the ID of the application module as returned by thecommand safekit module listid.
Through the MIB, you can read and display the status of an application module on aserver (STOP, WAIT, ALONE, UP, PRIM, SECOND) or you can take an action on themodule (start, stop, restart, swap, stopstart, prim, second).
For example, the status of the module with ID 1 is read by an SNMP get to the variable:
enterprises.bull.safe.safekit.skModuleTable.skModuleEntry.skModuleCurrentSt
ate.1 = stop (0)Use the snmpwalk command to check all MIB entries.

The resource table: skResourceTable
Each element defines a resource as for instance the one corresponding to the network
interface checker "intf.192.168.0.0" and its status (unknown, init, up, down).
Example: SNMP get request toenterprises.bull.safe.safekit.skResourceTable.skResourceEntry.skResourceNam
e.1.2 means name of resource 2 in application module 1.

39 A2 38MC 03

171

SafeKit User's Guide

10.9

Commands log of the SafeKit server

There is a log of the safekit commands ran on the server. It allows auditing the actions
performed on the server to help support for instance. The log records all the safekit
commands that are run and that modify the system such as a module install andconfiguration, a module start/stop, the safekit webserver start/stop, …
The command log is stored in the SAFEVAR/log.db file in SQLite3 format. For viewing its
content:
run the command safekit cmdlog
orclick on the commands log tab into the web console
Below is the raw extract of this log:
| 2021-07-27 14:37:33.205122 |safekit |mirror| 2021-07-27 14:37:33.400513 |cluster |cluster state| 2021-07-27 14:37:33.405597 |cluster |state change on node centos7-test3| 2021-07-27 14:37:34.193280 ||| 2021-07-27 14:37:34.718292 |cluster |cluster state| 2021-07-27 14:37:34.722080 |cluster |state change on node centos7-test4| 2021-07-27 14:37:37.510971 ||| 2021-07-27 14:38:05.092924 |
safekit |mirror -u web@10.0.0.103| 2021-07-27 14:38:05.109368 ||mirror |6883 | START | config -m

mirror |0 |I | update

mirror |0 |I | module|
mirror |6883 |
0 |

END | 0I | update

mirror |0 |I | module|
mirror ||6871 |
END | 07017 | START | prim -m
7017 |

END | 0Each field has the following meaning:✓

The 1st field in the log entry is the date and time of the message✓

The next one is the type of the action✓

The next one is the module name when the action is not global✓

The next one is the pid of the process that runs the command. It is used as theidentifier of the log entry✓

The next ones are START when the command starts and the command’s arguments;
or END when the command has finished with the return value.

10.10

SafeKit log messages in system journal

Since SafeKit 8, SafeKit modules log messages are sent to system log too. To view them:
In Windows, open a PowerShell window and run
Get-EventLog -Logname Application -Source Evidian.SafeKit that returns:

172

39 A2 38MC 03

Advanced administration

47086 Nov 23 11:27 Information Evidian.SafeKit
1073873154 mirror |heart | Remote state UNKNOWN Unknown...
47085 Nov 23 11:27 Information Evidian.SafeKit
1073873154 mirror |heart | Resource heartbeat.flow set to down by heart...
47084 Nov 23 11:26 Information Evidian.SafeKit
1073873154 mirror |heart | Local state ALONE Ready...
47082 Nov 23 11:26 Warning
Evidian.SafeKit
2147614977 mirror |heartplug | Action alone called by heart : remote stop...
47081 Nov 23 11:25 Information Evidian.SafeKit
1073873154 mirror |heart | Remote state PRIM Ready...
47080 Nov 23 11:25 Information Evidian.SafeKit
1073873154 mirror |heart | Local state SECOND Ready...
47079 Nov 23 11:25 Information Evidian.SafeKit
1073873154 mirror |rfsplug | Reintegration ended (default)...
In Linux, open a shell window and run
journalctl -r -t safekit that returns:
Nov 23 15:22:43 localhost.localdomain safekit[3689940]: mirror | heart | Local
state ALONE ReadyNov 23 15:22:43 localhost.localdomain safekit[3689940]: mirror | heart | Local
state PRIM ReadyNov 23 15:16:48 localhost.localdomain safekit[3689940]: mirror | heart | Local
state ALONE ReadyNov 23 15:16:48 localhost.localdomain safekit[3690096]: mirror | userplug |Script start_prim > userlog_2023-11-23T151648_start_prim.ulog
Nov 23 15:16:48 localhost.localdomain safekit[3690066]: mirror | rfsplug |Uptodate replicated file system
Nov 23 15:16:24 localhost.localdomain safekit[3689940]: mirror | heart | Remotestate UNKNOWN Unknown

39 A2 38MC 03

173

SafeKit User's Guide

174

39 A2 38MC 03

11.Securing the SafeKit web service
11.1 “Overview” page 175
11.2 “HTTP setup” page 177
11.3 “HTTPS setup” page 180
11.4 “User authentication setup” page 193

11.1

Overview

The SafeKit web service is mainly used by:
the web console (see section 3 page 37)
the distributed command line interface (see 9.1 page 141)
SafeKit provides different setups for this web service to enhance the security of the
SafeKit web console and distributed commands.

ProtocolAuthenticationRole management

✓

HTTP

✓

None (http only)

✓

Admin

✓

HTTPS

✓

File based✓

Control✓

LDAP/AD✓

Monitor✓

OpenID Connect

The most secure setups are based on HTTPS and user authentication. SafeKit provides a“private” certification authority (the SafeKit PKI). This allows SafeKit to be quickly
secured without the need for an external PKI (enterprise PKI or commercial PKI) thatprovides trusted certification authority.
SafeKit offers also optional role management based on 3 roles:
Admin role

This role grants all administrative rights by allowing access to
Configuration andMonitoring in the navigation sidebar

Control role

This role grants monitoring and control rights by allowing access only to
Monitoring in the navigation sidebar

Monitor role

This role grants only monitoring rights, prohibiting actions on modules(start, stop…) in
Monitoring in the navigation sidebar.

39 A2 38MC 03

175

SafeKit User's Guide

11.1.1

Default setup

The default setup is the following:
Setup

Authentication

ProtocolDefault✓

HTTP

Role management✓

File-based authentication(username/password stored in an
Apache file)✓

Initialization with a single user
named admin with the Admin role

To configure, see 11.2.1 page 177

11.1.2Predefined setups

The predefined setups are as follows:
Setup

Unsecure

Authentication

Protocol✓HTTP

Role management✓

No authentication✓

Same role for all users

For troubleshooting purpose only.
To configure, see 11.2.2 page 179

File-based

✓

HTTP

✓

HTTPS

To configure HTTPS with:✓username/password stored in a
local Apache file✓

Optional role management stored ina local Apache file

To configure, see 11.4.1 page 194

the SafeKit PKI, see 11.3.1
page 181
an external PKI, see 11.3.2
page 189

LDAP/AD

✓

HTTP

✓

LDAP/AD authentication

✓

HTTPS

✓

Optional role management

To configure HTTPS with:

To configure, see 11.4.2 page 196

the SafeKit PKI, see 11.3.1
page 181
an external PKI, see 11.3.2
page 189

176

39 A2 38MC 03

Securing the SafeKit web service

OpenID
Connect

✓

HTTP

✓

OpenID Connect authentication

✓

HTTPS

✓

Optional role management

To configure HTTPS with:

To configure, see 11.4.3 page 199

the SafeKit PKI, see 11.3.1
page 181
an external PKI, see 11.3.2
page 189

On Linux, for all files added under SAFE/web/conf, change their rights
with:chown safekit:safekit SAFE/web/conf/<filename>chmod 0440 SAFE/web/conf/<filename>

11.2

HTTP setup

By default, after the SafeKit install, the web service is configured for HTTP with file-based
authentication that must be initialized.
This default configuration can be extended as described in 11.2.1 page 177.
It can also be replaced by the unsecure setup described in 11.2.2 page 179 or anyone ofthe predefined setups.

11.2.1

Default setup

The default setup relies on HTTP with file-based authentication. It requires some
initialization described below. It is a mandatory step.
This default configuration can be extended:✓to add users and assign them a role as described in 11.4.1.1 page 194✓

to switch to HTTPS with:
the SafeKit PKI described in 11.3.1 page 181
an external PKI described in 11.3.2 page 189

After the installation of SafeKit, the configuration and restart of the web service is notnecessary since this is the default configuration and the web service has been started
with it.
11.2.1.1 Reset to default HTTP Setup
If you have changed the default user authentication configuration and want to revert toit, see 11.4.1 page 194.

39 A2 38MC 03

177

SafeKit User's Guide

If you want to revert to HTTP from HTTPS, on all SafeKit servers:
Remove SAFE/web/conf/ssl/httpd.webconsolessl.conf
Run safekit webserver restart(where SAFE=C:\safekit in Windows if System Drive=C:Linux)and SAFE=/opt/safekit in

11.2.1.2 Initialization for the web console and distributed command
SafeKit provides a script to get the web console and distributed commands up andrunning quickly.
In Linux, this script can be automatically called during the install of SafeKit; in Windows,
it must be manually executed. In both cases, you will have to give the password value,<pwd> for the admin user.
On S1 and S2:
On Windows, open a PowerShell window as
administrator and run (SAFE=C:\safekit if%SYSTEMDRIVE%=C:)
webservercfg -passwd<pwd>

SAFE/private/bin/webservercfg.ps1 -passwd<pwd>
On Linux, open a shell window as root and run(SAFE=/opt/safekit)
SAFE/private/bin/webservercfg -passwd <pwd>
You must set the same password on all nodes.

The password must be identical on all the nodes of the cluster. Otherwise,
web console and distributed commands will fail with authentication errors.

Once this initialization is done on all the cluster nodes:
you can authenticate in the web console with the name admin and the password you
provided. The role is Admin by default (unless you change the default behavior byproviding the group.conf file as described in in 11.4.1.1 page 194)
On authentication failure in the web console, you may need to reinitialize the admin
password. For this, run again webservercfg -passwd <pwd> on all nodes.
you can run distributed commands. It is based on a dedicated user rcmdadmin with
the Admin role. It is managed in a different, private user file that you do not have tochange.

178

39 A2 38MC 03

Securing the SafeKit web service

On authentication failure for distributed commands, you may need to reset
rcmdadmin password. To reset only this one, without changing the admin password,
run webservercfg -rcmdpasswd <pwd> on all nodes.
11.2.1.3 Test the web console and distributed command
The setup is complete; you can now test that it is operational.
Test the web console
1.

Start a browser on the user’s workstation

2.

Connect it to the default URL http://host:9010 (where host is the name or Ip
address of one of the SafeKit nodes)

3.

In the login page, enter admin as user’s name and the password you gave on
initialization (the value for <pwd>)

4.

The loaded page authorizes accesses that corresponds to the Admin role bydefault

Test the distributed command
1. Connect on S1 or S2 as administrator/root
2. Open a system console (PowerShell, shell, …)
3. Change directory to SAFE
4. Run safekit -H "*" levelthat should return the level for all nodes

11.2.2

Unsecure setup based on identical role for all

It is based on the configuration of a single role that is applied to all users without
requiring authentication. This solution can only be implemented in HTTP and isincompatible with user authentication methods. It is intended to be used fortroubleshooting only.
11.2.2.1 Configure and restart the web service
To configure where SAFE=C:\safekit in Windows if System Drive=C: ; andSAFE=/opt/safekit in Linux):
On S1 and S2:edit SAFE/web/conf/httpd.conf filecomment all authentication variants (usefile, useldap,
useopenid)#Define usefile…
#Define useldap…
#Define useopenid

39 A2 38MC 03

179

SafeKit User's Guide

select the desired role by uncommenting the associated line; ifboth lines are commented, the default role is Monitor.
Define httpadmin#Define httpcontrol✓

httpadmin for Admin role✓

httpcontrol for Control roleOn S1 and S2, disable HTTPS if you had configured it:
remove the file
SAFE/web/conf/ssl/httpd.webconsolessl.conf
On S1 and S2:
run safekit webserver restart

11.2.2.2 Test the web console and distributed command
The setup is complete; you can now test that it is operational.
Test the web console
1.

Start a browser on the user’s workstation

2.

Connect it to the default URL http://host:9010 (where host is the name or Ip
address of one of the SafeKit nodes)

3.

The loaded page authorizes only the actions corresponding to the selected role

Test the distributed command
1. Connect on S1 or S2 as administrator/root
2. Open a system console (PowerShell, shell, …)
3. Change directory to SAFE
4. Run safekit -H "*" levelthat should return the level for all nodes

11.3

HTTPS setup

The HTTPS web service relies on the existence of a set of certificates listed below:
The certificate of the Certification Authority CA used to issue
the server certificate for S1 and S2

180

39 A2 38MC 03

Securing the SafeKit web service

The server certificate of S1 and S2 used to assert the nodes’
identity

Apply one of the following 2 procedures to configure HTTPS and associated certificates:
11.3.1 “HTTPS setup using the SafeKit PKI” page 181
Go to this section to quickly setup HTTPS with the SafeKit “private” certification
authority.
11.3.2 “HTTPS setup using an external PKI” page 189
Go to this section to setup HTTPS with an external PKI (enterprise PKI or commercial
PKI) that provides trusted certification authority.
At the end of HTTPS setup, you must implement one of the authentication methods
described in 11.4 page 193.

11.3.1HTTPS setup using the SafeKit PKI
Verify that the system clock is set to the current date and time on all
SafeKit nodes and workstations that will run the HTTPS SafeKit web
console. Certificates are timestamped, and a time difference between
systems may have an impact on certificate validity.

11.3.1.1 Choose the Certificate Authority serverFirst, choose one SafeKit node to act as the Certificate Authority server. The selected
node will be hereafter called the CA server. The other cluster nodes are called non-CA
server. Then go through all the next subsections to activate the HTTPS configuration
with the SafeKit PKI.

11.3.1.2 Start the CA web service on the CA serverOn the CA server:
1.

Log as administrator/root and open a command shell window

2.

Change to the directory SAFE/web/bin

3.

Run the command ./startcaserv
When prompted, enter a password to protect the access to this service for the
CA_admin user (for instance, PasW0rD). This command starts the safecaserv service.

39 A2 38MC 03

181

SafeKit User's Guide

Remember this password since it will be required to connect to this service
in next steps.
The CA web service running on the first server is also accessed by theadditional non-CA servers.
Since the service listens to TCP port 9001, make sure TCP port 9001 is not
used, and is allowed in the firewall configuration. On Linux, the TCP 9001
port is automatically opened in local firewall by the startcaserv command.
In Windows, the safekit firewallcfg command opens safecaserv
service communications.
11.3.1.3 Generate Certificates on the CA server
During this step, the environment for generating certificates is set up: certificate
authority, local server and client certificates are created; and server-side certificates are
installed in their expected location.
On the CA server:
1. Log as administrator/root and open a command shell window
2. Change to the directory SAFE/web/bin
3. List server DNS names and IP addresses
By default, the server certificate includes all the locally defined IP addresses and DNS
names. They are listed into the files: SAFE/web/conf/ipv4.json and
SAFE/web/conf/ipv6.json and SAFE/web/conf/ipnames.json.
For building these files, run the command:
In Linux./getipandnames
This command relies on the host command delivered with the bind-utils
package. Install it if necessary or manually fill the DNS names into the file
SAFE/web/conf/ipnames.json.
In Windows./getipandnames.ps1
If the service will be accessed using another DNS name or IP address, edit thecorresponding file to insert the new value before executing the initssl
command. This is required for instance in the clouds using NAT, where theserver has a public address mapped on a private address.
4. Run the command:./initssl sca
This command :182

39 A2 38MC 03

Securing the SafeKit web service✓

Create a CA certificate conf/ca/certs/cacert.crt and its associated key
conf/ca/private/cacert.key✓

Create server certificate conf/ca/certs/server_<HOSTNAME>.crt and its
corresponding key conf/ca/private/server_<HOSTNAME>.key✓

Install the CA certificate, server certificate and key in the conf directoryThis command creates a Certificate Authority certificate with the default
subject name (that is “SafeKit Local Certificate Authority”). To customize
the subject name, run the command with an extra parameter:./initssl sca “/O=My Company/OU=My Entity/CN=My Company
Private Certificate Authority”

11.3.1.4 Generate certificates on non-CA server
During this step, on non-CA servers, local certificate requests are created, signed
certificates are retrieved from the CA server, and finally certificates are installed at their
expected locations.
Apply the following procedure sequentially on each non-CA servers:
1. Log on as administrator/root and open a command shell window
2. Change to the directory SAFE/web/bin
3. List server DNS names and IP addresses
By default, the server certificate includes all the locally defined IP addresses and DNS
names. They are listed into the files: SAFE/web/conf/ipv4.json,
SAFE/web/conf/ipv6.json and SAFE/web/conf/ipnames.json. For building thesefiles, run the command:
In Linux./getipandnames
This command relies on the host command delivered with the bind-utils
package. Install it if necessary or manually fill the DNS names into the file
SAFE/web/conf/ipnames.json.
In Windows./getipandnames.ps1
If the service will be accessed using another DNS name or IP address, edit thecorresponding file to insert the new value before executing the initssl
command. This is required for instance in the clouds using NAT, where theserver has a public address mapped on a private address.
4. Run the command:./initssl req https://CAserverIP:9001 CA_admin
where CAserverIP is the DNS name or IP address of the CA server.
Then enter, each time it is required, the password you specified when you started the
CA web service on the CA server (for instance, PasW0rD)Or

39 A2 38MC 03

183

SafeKit User's Guide./initssl req https://CAserverIP:9001 CA_admin:PasW0rD

If necessary, set the environment variables HTTPS_PROXY and HTTP_PROXY toadequate values.

If you get the error "Certificate is not yet valid", it means the system clockof the server is not synchronized with the system clock of the CA server.
You should synchronize your server clocks and re-run the initssl
command if the time difference is not acceptable.

11.3.1.5 Enable HTTPS on CA server and non-CA server
To enable HTTPS, on all SafeKit servers:
copy SAFE/web/conf/httpd.webconsolessl.conf to
SAFE/web/conf/ssl/httpd.webconsolessl.conf
On Linux run :chown safekit:safekit SAFE/web/conf/ssl/httpd.webconsolessl.conf
chmod 0440 SAFE/web/conf/ssl/httpd.webconsolessl.conf
run safekit webserver restart(where SAFE=C:\safekit in Windows if System Drive=C:Linux)and SAFE=/opt/safekit in

11.3.1.6 Configure the firewall on CA server and non-CA server
When the SafeKit web service runs in HTTPS mode, it is safe to allow network
communication with this server and configure the firewall. For this, apply the instructions
described in 10.3 page 158.
11.3.1.7 Set the HTTPS SafeKit Web console
If the CA certificate has not been imported, the browser issues security alerts when theuser connects to the web console with his client certificate. If the import has not alreadybeen done, apply the procedure below in Windows:
1. Log-in the user’s workstation
2. Download from the CA server the CA certificates (cacert.crt file) located into
SAFE/web/conf/ca/certs.

184

39 A2 38MC 03

Securing the SafeKit web service

3. Click on the downloaded
cacert.crt file for opening thecertificate window. Then click on
Install Certificate button

4. It opens the Certificate Import
Wizard. Select Current User andclick on the Next button

5. Browse stores to selectthe Trusted Root
Certification Authorities
store. Then click on Next
button6. Then complete thecertificate import.

39 A2 38MC 03

185

SafeKit User's Guide

11.3.1.8 Stop the CA web service on CA serverOnce all SafeKit servers have been configured, it is recommended to bring the CA web
service (safecaserv service) offline on the CA server, to limit the risk of accidental or
malicious access.
For stopping the SafeKit CA web service with the command line:1.

Log as administrator/root and open a command shell window

2.

Change to the directory SAFE/web/bin

3.

Run the command ./stopcaserv
On Windows, this command also removes the service entry to prevent any
accidental start of the service afterwards. On Linux, the 9001 port is
automatically closed on local firewall.

When all foreseeable certificate generation and installation is done, it is a good practice
to make sure files unnecessary at production time are not accessible. This step is notmandatory.
The files that constitute the CA, i.e., the SAFE/web/conf/ca file tree (especially theprivate keys stored under SAFE/web/conf/ca/private/*.keys) should be stored for
future use on a removable storage media and removed from the server. Store theremovable media in a secure place (i.e., a vault). This also applies to the files locatedunder the SAFE/web/conf/ca directory of non-CA servers. The CA files should berestored into the same location before using the CA again (for example, if adding a new
SafeKit cluster node).
11.3.1.9 SafeKit PKI advanced configuration
11.3.1.9.1

Renewing certificates

Every certificate has an expiration date. The default expiration date of the CA certificate
is set to 20 years after the CA installation date. The default expiration date of the server
certificates is set to 20 years after the certificate request date.
Expired server certificates will trigger warnings when the browser connects to the server.
Expired CA certificates cannot be used to validate issued certificates.
It is possible to renew certificates using the original certificate requests and the private
keys stored under the SAFE/web/conf/ca directory tree. You may also create a newcertificate request using the existing private key. The procedure to do so is beyond thescope of this document, see openssl (or your certificate authority) documentation.
Creating a new set of certificates (and private keys) will have the side effect of renewing
all certificates. To create a new set of certificates:
1. Erase the web/conf/ca directory on all SafeKit servers related to the CA, including
the CA SafeKit server itself
2. Suppress existing certificates from the client machines certificate stores
3. Apply the full procedures described in 11.3 page 180
11.3.1.9.2

Revoking certificates

It is possible to modify the SafeKit web service configuration to use a CRL containing therevoked certificates list. Setting up such a configuration is beyond the scope of thisdocument. Refer to the Apache and openssl documentation.

186

39 A2 38MC 03

Securing the SafeKit web service

Creating a new set of certificates and replacing the old set with the new one will have theside effect of effectively revoking the previous certificate set, since the CA certificate isdifferent.
11.3.1.9.3

Commands for certificate generation

These commands are located, and must be run from, the SAFE/web/bin directory.
All paths below are relative to SAFE/web directory.

initssl sca [<subject>]Parameters<Subject>: the optional CA certificate subject, that identify in human readable form theowner of the CA.
Examplesinitssl ca "/O=My Company/OU=My Unit/CN=My Company Private Certificate
Authority"
Description
This command :Create a CA certificate conf/ca/certs/cacert.crt and its associated key
conf/ca/private/cacert.key
Create server certificate conf/ca/certs/server_<HOSTNAME>.crt and its
corresponding key conf/ca/private/server_<HOSTNAME>.key
Install the CA certificate, server certificate and key in the conf directoryIt’s initialize a conf/ca file tree needed for the SafeKit PKI related commands.

Note that the best practice is to protect private keys with apassword, but it needs more complex configuration on theserver and is beyond the scope of this document. See theApache and OpenSSL documentation for more information.
initssl rca
Description
As initssl sca, but reuse the existing CA infrastructure to reissue the server certificate andkey (re)install the CA certificate , server certificate and key in the conf directory

initssl req <url> <user>[:<password>]]Parameters<url>: URL of the CA service. (https://CA_server:9001)

39 A2 38MC 03

187

SafeKit User's Guide<user>,<password>: user and password used to authenticate against the CA web
service.
<user> preconfigured value is CA_admin. <password> is the one entered by theadministrator at the start of CA web service. If these optional field are not present,
the password will be asked interactively several times, when needed.
Exampleinitssl req https://192.168.0.1:9001 CA_admin:PasW0rD
Description
This command :Creates a certificate request for a server certificate that includes all the locally defined
IP addresses and DNS names. The certificate request is stored inconf/ca/private/server_<hostname>.csr. The corresponding key is stored inconf/ca/private/server_<hostname>.key.
Creates a certificate request for a client certificate with the Admin role (to be used by
the distributed commands). The certificate request is stored inconf/ca/private/user_Admin_<hostname>.csr. The corresponding key is stored inconf/ca/private/user_Admin_<hostname>.key.
Retrieves the CA certificate from the CA server
Retrieves signed certificates corresponding to the certificate requests above, from the
CA server (using provided login)
Installs certificates and keys in the conf directory
Checks certificates are OK
If no <url> is given, the command stops after having generated the certificate requests
corresponding to:
The local server, in the conf/ca/private/server_<hostname>.csr
An Admin role client certificate, inconf/ca/private/user_Admin_<hostname>.csr
Those certificate requests are stored in a base64 encoded file ready to be submitted
to an external certificate authority such as Microsoft Active Directory Certificate
Services (refer to the Microsoft documentation on how to submit a base64 encodedcertificate request file).
makeusercert <name> <role>
Parameters<name> is the subject's CN name of the certificate, usually the subject's username.
<role> is subject's role as a console user. The valid value is Admin or Control orMonitor.
Examplesmakeusercert administrator Admin
makeusercert manager Control

188

39 A2 38MC 03

Securing the SafeKit web service

makeusercert operator Monitor
Description
Creates a client certificate request (and certificate + pkcs12 file containing certificate andkey if started on the CA SafeKit server) for the <name> and <role>.
When the pkcs12 file is generated, the command asks twice for a password to protect the
file. The generated unencrypted private key is stored into
conf/ca/private/user_<role>_<name>.key file. If applicable, the generated certificate
and pkcs12 files are stored into conf/ca/certs/user_<role>_<name>.crt andconf/ca/private/user_<role>_<name>.p12 files respectively.
Client certificates could be used as an authentication method on an HTTPS server. They
are transmitted to the web service by the browser and verified on the server as part of
the HTTPS connection handshake. A certificate corresponding to the desired role must be
installed in the browser certificate store before the SafeKit web console can be used.
11.3.1.9.4

SafeKit CA web service

The SafeKit CA web service configuration is stored in
SAFE/web/conf/httpd.caserv.conf file.
This service implements limited PKI.
CA certificates are accessible at the https://CAserverIP>:9001/certs/<certificatename>.crt URL.
For example, the CA certificate is accessible athttps://CAserverrIP>:9001/certs/cacert.crt.
Certificate signature requests are processed by posting a form at the URL:
https://<CA server IP>:9001/caserv .
The form takes the following parameters:
action = signrequest
name = <certificate name>servercsr = <file content of the server certificate request>
Orusercsr = <file content of the client certificate request>

11.3.2HTTPS setup using an external PKI

Apply steps below to setup HTTPS with your trusted certification authority (yourenterprise PKI or commercial PKI).
11.3.2.1 Get and install server certificates
11.3.2.1.1

Get certificate files

You must get server certificates from the PKI with the expected format.

39 A2 38MC 03

189

SafeKit User's Guide

The certificate of the Certification Authority CA used to issue theserver certificates

The server certificate to assert the S1 identity.

The server certificate to assert the S2 identity.

X509 certificate file in PEM format
The subfield CN (Common Name) into the subject field, or the
Subject Alternative Name field of the certificate, must contain :s1.crts2.crt✓S1 name(s) and/or IP address(es) for s1.crt✓

S2 names and/or IP address(es) for s2.crt

Be aware that you must provide all names and/or IP
addresses, for S1 and S2, which are used for HTTPS
connections:✓those included into the SafeKit cluster configuration file✓

Those used in the browser URL to load the web console
from a cluster node, and which are not present into thecluster configuration

See the example in 11.3.2.1.3 page 191.
s1.keys2.key11.3.2.1.2

The private, *unencrypted* key corresponding to the certificatess1.crt and s2.crt

Install files in SafeKit

Install the certificates as follow (where SAFE=C:\safekit in Windows if System
Drive=C: ; and SAFE=/opt/safekit in Linux):
On S1:
copy s1.crt to SAFE/web/conf/server.crt
s1.crts1.keycopy s1.key to SAFE/web/conf/server.key
On S2:
copy s2.crt to SAFE/web/conf/server.crt

s2.crt190

39 A2 38MC 03

Securing the SafeKit web service

s2.keycopy s2.key to SAFE/web/conf/server.key

On Linux, on S1 and S2, run:chown safekit:safekit SAFE/web/conf/server.crt SAFE/web/conf/server.key
chmod 0440 SAFE/web/conf/server.crt SAFE/web/conf/server.key

You can check the installed certificates with:
cd SAFE/web/bincheckcert -t server
It returns a failure if an error is detected.
You can check that the certificate contains some DNS name or IP address with:checkcert -h ”DNS name value”
checkcert -i ”Numeric IP address value”
11.3.2.1.3

ExampleConsider the following architecture:

The corresponding SafeKit cluster configuration file, SAFEVAR/cluster/cluster.xml
must contain these values into addr field:<?xml version="1.0"?><cluster><lans><lan name="default"><node name="s1" addr="10.0.0.10"/><node name="s2" addr="10.0.0.11"/></lan><lan name="private"><node name="s1" addr="10.1.0.10"/><node name="s2" addr="10.1.0.11"/></lan></lans></cluster>
The server certificates must contain the same values (DNS names and/or IP addresses)
as those in the cluster configuration and the values used to connect the web console. If
not, the SafeKit web console and distributed commands will not work properly.
To check that the certificate file is correct:
1. Copy the .crt (or .cer) file on a Windows workstation

39 A2 38MC 03

191

SafeKit User's Guide

2. Double click on this file to open it with Crypto Shell Extensions
3. Click on the Details tab
4. Verify the Subject Alternative Name fieldIf you prefer the command line interface, you can run on each the SafeKit node:
SAFE/web/bin/openssl.exe x509 -text -noout -in SAFE/web/conf/server.crt
and look for the value after Subject Alternative Name

11.3.2.2 Get and install the CA certificate
11.3.2.2.1

Get certificate file

You must get these certificates from the PKI with the expected format.
The Certification Authority CA certificate
used to issue the server certificates.
X509 certificate file in PEM format
cacert.crt

The chain of certificates for the root
and intermediates CA

Server certificates for S1and S2

If you have trouble retrieving this file from the PKI, you can build it using the proceduredescribed in 7.18 page 129.
11.3.2.2.2

Install file in SafeKit

Install certificates files as follow (where SAFE=C:\safekit in Windows if System
Drive=C: ; and SAFE=/opt/safekit in Linux):

192

39 A2 38MC 03

Securing the SafeKit web service

On S1 and S2:
copy cacert.crt to SAFE/web/conf/cacert.crt
On Linux, run:cacert.crt

chown safekit:safekit SAFE/web/conf/cacert.crt
chmod 0440 SAFE/web/conf/cacert.crt

You can check the installed certificates with:
cd SAFE/web/bincheckcert -t CA
It returns a failure if an error is detected.
You must also check that the cacert.crt contains the chain of certificates for the root
and intermediates Certification Authorities.
11.3.2.3 Configure and restart the web service
To enable HTTPS, on all servers :copy SAFE/web/conf/httpd.webconsolessl.conf to
SAFE/web/conf/ssl/httpd.webconsolessl.conf
On Linux, run :chown safekit:safekit SAFE/web/conf/ssl/httpd.webconsolessl.conf
chmod 0440 SAFE/web/conf/ssl/httpd.webconsolessl.conf
run safekit webserver restart(where SAFE=C:\safekit in Windows if System Drive=C: ; and SAFE=/opt/safekit
in Linux)
11.3.2.4 Change the firewall rules
You can run the safekit firewallcfg command to change the firewall rules. It set
SafeKit rules into the operating system default firewall (in Windows, Microsoft Windows
Firewall ; in Linux, firewalld or iptables).
On S1 and S2:
Firewallrun SAFE/safekit firewallcfg add

Don’t run this command if you want to configure the firewall yourself or if you use a
different firewall than the system one. For the list of SafeKit processes and ports, see
10.3 page 158.

11.4

User authentication setup

Setup one of the following user authentication methods:
11.4.1 “File-based authentication setup” page 194

39 A2 38MC 03

193

SafeKit User's Guide

11.4.2 “LDAP/AD authentication setup” page 196
11.4.3 “OpenID authentication setup” page 199
At the end of this setup, you can start using the secure SafeKit web console.

11.4.1File-based authentication setup

File-based authentication setup can be applied in HTTP or HTTPS. It relies on thefollowing files:

User file configuration that defines authorized users

Optional file to restrict the user’s role.
If the group.conf file is not present, all authenticated users will havethe Admin role.

11.4.1.1 Manage users and groups
The users and groups must be identical on S1 and S2, as well as passwords. It is definedby the files user.conf and group.conf into SAFE/web/conf directory (SAFE=C:\safekit
in Windows if System Drive=C: ; and SAFE=/opt/safekit in Linux).
During the default setup initialization, described in 11.2.1 page 177, theuser named admin has been created and thus is present into user.conf.
You can decide to remove this user if you create others.
Create a new user
Users are created with the SAFE/web/bin/htpasswd command.
For instance, to add the new user manager and set its password managerpassword,run:
SAFE/web/bin/htpasswd -bB SAFE/web/conf/user.conf manager managerpassword
The new user is inserted into SAFE/web/conf/user.conf the file.
admin:$2y$05$oPquL6Z2Y78QcXpHIako.O58Z6lWfa5A86XD.eCbEnbRcguJln9Cemanager:$apr1$U2GLivF5$x39WKmSpq6BGmLybESgNV1
operator1:$apr1$DetdwaZz$hy5pQzpUlPny3qsXrIS/z1
operator2:$apr1$ICiZv2ru$wRkc3BclBhXzc/4llofoc1

Assign the role of the users

194

39 A2 38MC 03

Securing the SafeKit web service

By default, all users have the Admin role. If you want to assign distinct roles to different
users, you must create the SAFE/web/conf/group.conf file and assign user’s role. The
group file can contain the 3 groups Admin, Control, Monitor. Users in these groups will
have the corresponding roles.
Each line of the group file must contain the group name followed by acolon, followed by the member users name separated by spaces. See theexample above.
For instance, assign the Control role to the new user manager:
Admin : admin
Control : manager
Monitor : operator1 operator2

If you enable the role management, you must insert the user admin into
group.conf. Otherwise, this user will no longer be operational.
Delete a user, …
Use htpasswd -? for all user management commands (add/delete, ...).

11.4.1.2 Install files
Install the files as follow (where SAFE=C:\safekit in Windows if System Drive=C: ;and SAFE=/opt/safekit in Linux):
On S1 and S2:
copy user.conf to SAFE/web/conf/user.conf

On S1 and S2 if groups are set:
copy group.conf to SAFE/web/conf/group.conf
On Linux, on S1 and S2, run:chown safekit:safekit SAFE/web/conf/user.conf SAFE/web/conf/group.conf
chmod 0440 SAFE/web/conf/user.conf SAFE/web/conf/group.conf

These files must be identical on all nodes.
11.4.1.3 Configure and restart the web service
To configure the file-based authentication (where SAFE=C:\safekit in Windows if
System Drive=C: ; and SAFE=/opt/safekit in Linux):

39 A2 38MC 03

195

SafeKit User's GuideOn S1 and S2:edit SAFE/web/conf/httpd.conf file
if necessary uncomment usefile
Define usefile
On S1 and S2:
run safekit webserver restartThis is the default content of httpd.conf.
11.4.1.4 Test the web console and distributed command
The setup is complete; you can now test that it is operational.
Test the web console
1.

Start a browser on the user’s workstation

2.

Connect it to the default URL http://host:9010 (where host is the name or Ip
address of one of the SafeKit nodes). If HTTPS is configured, there is an automatic
redirection to https://host:9453

3.

In the login page, specify in the user’s name and password
With the SafeKit default configuration, you can log-in with the user admin bygiving the password you assigned during initialization.

4.

The loaded page only allows access authorized by the user's role. If the groups
have not been defined, all users have the Admin role.

Test the distributed command
1. Connect on S1 or S2 as administrator/root
2. Open a system console (PowerShell, shell, …)
3. Change directory to SAFE
4. Run safekit -H "*" levelthat should return the level for all nodes

11.4.2

LDAP/AD authentication setup

LDAP/AD authentication setup can be applied in HTTP or HTTPS. It requires:
LDAP/Active Directory account configuration used toassert the user identity

196

39 A2 38MC 03

Securing the SafeKit web service

Optional LDAP/Active Directory group configuration to
restrict the user’s role.
When groups are not defined, all authenticated users
have the Admin role.

On some Linux distributions (such as RedHat 8 and CentOS 8), the web
server start fails when it is configured with LDAP/AD authentication. In thiscase, apply the solution described in SK-0092.

Apply the steps described below after verifying that S1 and S2 can connect to the LDAP
controller domain port (default is 389).
11.4.2.1 Manage users and groups
If necessary, ask your LDAP administrator to create users of the SafeKit web console.
If you want to define user’s role, ask your LDAP administrator to create groups for
Admin, Control, Monitor roles and assign users to groups. When groups are not defined,
all users will have the Admin role.
11.4.2.2 Configure and restart the web service
To configure the LDAP/AD authentication (where SAFE=C:\safekit in Windows if%SYSTEMDRIVE%=C: ; and SAFE=/opt/safekit in Linux):
On S1 and S2:
Initialize the authentication for the distributed command. This may have alreadybeen done if you initialized the default configuration after SafeKit installation.
Otherwise:
Run webservercfg -rcmdpasswd pwd
where pwd is the password for the private user rcmdadmin. You don’t need tomemorize it.
On S1 and S2:edit SAFE/web/conf/httpd.conf fileuncomment useldap
Define useldap
Locate the following lines and replace bold values according to your LDAP/AD
service configuration:
Define binddn "CN=bindCN,OU=bindOU1,OU=bindOU2,DC=domain,DC=fq,DC=dn"
Define bindpwd "Password0"
Define searchurl "ldap://ldaporad.fq.dn:389/OU=searchou, DC=domain, DC=fq,
DC=dn?sAMAccountName, memberOf?sub?(objectClass=*)"39 A2 38MC 03

197

SafeKit User's Guide

the binddn and bindpwd variables must contain the credentials of an accountwith search rights on the directory
the searchurl variable defines the RFC2255 search URL to authenticate theuser
CN: common name
OU: organization unit
DC: domain component (one field for each part of the FQDN)
If the group configuration is not enabled, all authenticated users will have theAdmin role.
On S1 and S2
To enable group management:
edit SAFE/web/conf/httpd.conf fileuncomment the following lines and replace bold values according to your
LDAP/AD service configuration:
Define admingroup"CN=Group1CN,OU=Group1OU1,OU=Group1OU2,DC=domain,DC=fq,DC=dn"
Define controlgroup"CN=Group2CN,OU=Group2OU1,OU=Group2OU2,DC=domain,DC=fq,DC=dn"
Define monitorgroup"CN=Group3CN,OU=Group3OU1,OU=Group3OU2,DC=domain,DC=fq,DC=dn"
Users set into the LDAP/AD groups associated to admingroup, controlgroup andmonitorgroup, will respectively have Admin, Control and Monitor roles.
For more sophisticated authentication, read Apache web service documentation(see http://httpd.apache.org).
On S1 and S2:
run safekit webserver restart

11.4.2.3 Test the web console and distributed command
The setup is complete; you can now test that it is operational.
Test the web console
1.

Start a browser on the user’s workstation

2.

Connect it to the default URL http://host:9010 (where host is the name or Ip
address of one of the SafeKit nodes). If HTTPS is configured, there is an automatic
redirection to https://host:9453

5.

In the login page, specify in the user’s name and password

6.

The loaded page only allows access authorized by the user's role. If the groups
have not been defined, all users have the Admin role.

198

39 A2 38MC 03

Securing the SafeKit web service

Test the distributed command
1. Connect on S1 or S2 as administrator/root
2. Open a system console (PowerShell, shell, …)
3. Change directory to SAFE
4. Run safekit -H "*" levelthat should return the level for all nodes

11.4.3

OpenID authentication setup

OpenID authentication relies on the mod_auth_openidc Apache module. It requires:
OpenID Identity provider client application registration
and account configuration used to assert the user
identity
Optional OpenID claims configuration to restrict theuser’s role.
When claims are not defined, all authenticated users
have the Admin role.

On some Linux distributions you may need to install the
mod_auth_openidc module from the distribution repository.

Apply the steps described below after verifying that S1 and S2 can connect to the
OpenID Identity Provider. You may need to setup a proxy configuration, see relevant
httpd.conf section and mod_auth_openidc documentation for details.
11.4.3.1 Manage app, users and groups
If necessary, ask your OpenID administrator to create users of the SafeKit web console.
Ask your OpenID administrator to register the webconsole App into the OpenID provider(OP) and retrieve the assigned credentials (ClientID and ClientSecret) values (you willneed those values during the httpd.conf configuration step below).
Set the app’s redirect uri to Erreur ! Référence de lien hypertexte non valide.
FQDN>:9453/openid or Erreur ! Référence de lien hypertexte non valide.
FQDN>:9010. If you plan to connect to more than one server, enter the url of eachconnection server.
If you want to define user’s role on the Identity Provider, ask your OpenID administrator
to create groups or roles for Admin, Control, Monitor roles and assign users to the

39 A2 38MC 03

199

SafeKit User's Guide

created groups or roles, then fill in the AdminClaim, ControlClaim and MonitorClaimvariables in httpd.conf with the corresponding claims. When the above is not defined,
all authenticated users will have the Admin role.
You may also define the groups on the SafeKit Web Server by filling in the group.conf
file as in the File-based authentication case (see “Assign the role of the users” in section
11.4.1.1 page 194).
11.4.3.2 Configure and restart the web service
To configure the OpenID authentication (where SAFE=C:\safekit in Windows if%SYSTEMDRIVE%=C: ; and SAFE=/opt/safekit in Linux):
On S1 and S2:
Initialize the authentication for the distributed command. This may have alreadybeen done if you initialized the default configuration after SafeKit installation.
Otherwise:
Run webservercfg -rcmdpasswd pwd
where pwd is the password for the private user rcmdadmin. You don’t need tomemorize it.
On S1 and S2:edit SAFE/web/conf/httpd.conf fileuncomment useopenid
Define useopenid
Locate the following lines and replace values according to your OpenID service
configuration:
OIDCProviderMetadataURL <Your OpenId provider metadata URL>OIDCClientID <Your OpenID client ID>
OIDCClientSecret <Your OpenID client secret>
OIDCRemoteUserClaim <The Claim in ID token that identifies the user, if notset, defaults to sub>## openid connect scope request; this defines which claims are returned by
the IDP.
OIDCScope "openid email"✓the OIDCClientID and OIDCClientSecret variables must contain thecredentials of the registered app in the OpenID Identity Provider.

✓

the OICDScope variable defines the scopes needed to return the RemoteUser
and optionally roles claims. openid should always be specified.

If neither the AdminClaim, ControlClaim and MonitorClaim configuration nor thegroup.conf configuration is enabled, all authenticated users will have the Admin
role.

200

39 A2 38MC 03

Securing the SafeKit web service

On S1 and S2
To enable role claim management:
edit SAFE/web/conf/httpd.conf fileuncomment the following lines and replace the values according to your OpenID
service configuration:# Define AdminClaim roles:SKAdmin# Define ControlClaim roles:SKControl# Define MonitorClaim roles:SKMonitor
Users’ tokens bearing the claims defined by the AdminClaim, ControlClaim andMonitorClaim, will respectively have Admin, Control and Monitor roles.
For more details, see the mod_auth_openidc documentation (GitHub OpenIDC/mod_auth_openidc: OpenID Certified™ OpenID Connect Relying Party
implementation for Apache HTTP Server 2.x).
On S1 and S2:
run safekit webserver restart

11.4.3.3 Test the web console and distributed command
The setup is complete; you can now test that it is operational.
Test the web console
3.

Start a browser on the user’s workstation

4.

Connect it to the default URL http://host:9010 (where host is the name or Ip
address of one of the SafeKit nodes). If HTTPS is configured, there is an automatic
redirection to https://host:9453

5.

In the login page, specify in the user’s name and password

6.

The loaded page only allows access authorized by the user's role. If the groups
have not been defined, all users have the Admin role.

Test the distributed command
1. Connect on S1 or S2 as administrator/root
2. Open a system console (PowerShell, shell, …)
3. Change directory to SAFE
4. Run safekit -H "*" levelthat should return the level for all nodes

39 A2 38MC 03

201

SafeKit User's Guide

202

39 A2 38MC 03

12.Cluster.xml for the SafeKit cluster
configuration
12.1 “Cluster.xml file” page 20312.2 “SafeKit cluster Configuration” page 205
SafeKit uses the configuration file cluster.xml. This file defines all the servers thatmake up the SafeKit cluster as well as the IP address (or name) of these servers on the
networks used to communicate with the cluster nodes. These are global cluster andmodule internal communications; these communications are encrypted. This network isalso used for executing the global safekit command (with argument -H).
You must define at least one network that includes all nodes in the cluster. It is
recommended to define several networks to tolerate at least one network failure.

12.1

Cluster.xml file

Each network (lan) has a logical name that will be used in the configuration of themodules to name the monitoring networks:
into the heartbeat section for a mirror module (for details, see 13.3 page 213)
into the lan section for a farm module (for details, see 13.4 page 215)
The node name is the one that is used by the SafeKit administration service (safeadmin)for uniquely identifying a SafeKit node. You must always use the same name fordesigning a given server on different networks. This name is also used by the SafeKit
web console when displaying the node’s name.

12.1.1Cluster.xml example

In the example below, two networks are defined. The network named private can bededicated to file replication.
<cluster><lans><lan name="default"><node name="node1" addr="192.168.1.67"/><node name="node2" addr="192.168.1.68"/><node name="node3" addr="192.168.1.69"/><node name="node4" addr="192.168.1.70"/></lan><lan name="repli"><node name="node1" addr="10.0.0.1"/><node name="node2" addr="10.0.0.2"/><node name="node3" addr="10.0.0.3"/><node name="node4" addr="10.0.0.4"/></lan></lans></cluster>

39 A2 38MC 03

203

SafeKit User's Guide

In the example below, a unique network is used, but in a Network address translation(NAT) configuration. For each node two addresses must be defined: the local one laddr(defined on local interface) and the external one addr (as seen by other servers).
<cluster><lans><lan name="default"><node name="node1" addr="server1.dns.name" laddr="10.0.0.1"/><node name="node2" addr="server2.dns.name" laddr="10.0.0.2"/></lan></lans></cluster>All nodes must be able to communicate to the others via the NATted addresses.

12.1.2Cluster.xml syntax<cluster><lans [port="4800"]><lan name="lan_name" [command="on|off"] ><node name="node_name" addr="IP1_address"|"IP1_name"[ laddr="local_IP1_address" ]/><node name="node_name" addr="IP2_address"|"IP2_name"[ laddr="local_IP2_address" ] />…
</lan>…
</lans></cluster>

12.1.3<lans>, <lan>, <node> attributes<lans

Begin the definition of the cluster nodes andnetwork topology.

[port="xxxx"]

Defines the UDP port with which the membershipprotocol is exchanged.
Default: 4800

[pulse=”xxxx”]

Defines the period of the membership protocol
messages emission. Longer pulse makes themembership protocol use less bandwidth but react
more slowly.

[mlost_count=”xx”]

Defines the number of periods elapsed without
message before electing a new leader.

[slost_count=”xx”]

Defines the number of periods elapsed without
messages before declaring a follower node offline.

<lan

Definition of a LAN (i.e., IPv4 broadcast domain,
IPv6 link) on which the membership protocol will betransmitted. At least one LAN must be defined.
Define one such tag per used LAN.

204

39 A2 38MC 03

Cluster.xml for the SafeKit cluster configuration

name="lan name"Single logical name for the lan.
This name is used into module configuration toname networks used by the module.

command="on"|"off"Set command="on" to use this network for running
distributed commands on the cluster. In this case,this <lan> section must include all nodes in thecluster. You can set only one <lan> section withcommand="on".
When this attribute is not set, it is the first <lan>section that is used for running distributedcommands on the cluster.
Default: off<nodeDefinition of one node in the SafeKit cluster. Define
as many <node> tags as there are nodes in thecluster (at least 2).

name="node name"Single logical name to the SafeKit server.
You must always use the same name for designinga given server on different lans.

addr="IP_address"|"IP_name"

IPv4 or IPv6 address, or name of the node as it is
known by other nodes on this LAN (IP address
recommended to be independent from a DNS
server). On NAT configuration, it must be theexternal address.
When defining an IPv6 address, use literal format:
the address is enclosed in square brackets (e.g.
[2001::7334])

laddr="local_IP_address"

Local IP address on this LAN. To be used only on
NAT configurations, where local address is different
from external one.
IPv4 address or literal IPv6 address.

In SafeKit < 8.2, the cluster configuration had attributes console andframework on <lan> tag. These attributes were necessary for the legacy web
console and are obsolete with the new one. If presents, these attributes are
ignored in SafeKit 8.2.

12.2

SafeKit cluster Configuration

12.2.1

Configuration with the SafeKit web console

The SafeKit web console provides a configuration wizard for editing the cluster.xml fileand applying the configuration on all the cluster nodes.
39 A2 38MC 03

205

SafeKit User's Guide✓

The cluster configuration requires to log in the web console witha user having Admin role✓

If the cluster is not configured, the web console automatically
opens the Cluster configuration wizard✓

When the cluster is configured, the current cluster configuration
is loaded from the connection node specified in the browser URL

Open the cluster configuration wizard:✓

Directly via the URL http://host:9010/console/en/configuration/cluster/config

Or✓

Navigate in the console

In this example, the console is loaded from 10.0.0.107, which corresponds to node1 inthe existing cluster. This is the connection node.

•(1) Click on

Configuration in the navigation sidebar•(2) Click on Cluster configuration tab•(3) Click on

Configure the cluster button

For details on the cluster configuration wizard, see section 3.2.1 page 39.

12.2.2

Configuration with command line

For the full description of commands, refer to 9.3 page 144.
The commands for configuring the cluster with a new cryptographic key are:1.

206

safekit cluster config [<filepath>]

39 A2 38MC 03

Cluster.xml for the SafeKit cluster configuration

where filepath is the path for the new cluster.xml
when filepath is not set, the current configuration is kept and only encryption
key is generated
2.

safekit –H "*" -G
it applies the local configuration, defined into cluster.xml, on all cluster nodes

The commands line for re-configuring without cryptographic key are:1.

safekit cluster delkey

2.

safekit –H "*" -G

The commands for re-generating the cryptographic key are:1.

safekit cluster genkey

2.

safekit –H "*" -G

12.2.3

Configuration changes

When changing the cluster configuration, the new configuration must be applied on all
cluster nodes. When the configuration is applied only on a subset of the nodes present
into the cluster configuration, only this subset will be able to communicate with eachother. This is also the case when the cryptographic key is not identical on all nodes. Thiscan have the effect of disrupting the operation of the modules installed on servers.
For a correct behavior, you must re-apply the configuration on all the nodes that belong
to the cluster as described above.
You can check the configuration by running the command safekit cluster
confinfo on each node (see section 9.3 page 144). When the configuration
is operational, this command must return on all nodes, the same list ofnodes and the same value for the configuration signature.
Changing the cluster configuration could have important impact on module configurations
since the lan names set into the cluster configuration are used into the module’s
configuration. Any change in the cluster configuration, will trigger modules updates:
each module will reload its configuration to adapt the changes. Such changes could leadto module stop in case of incompatibility (for example if a lan used by a module isremoved from the cluster configuration). So, great care must be taken when modifying
cluster configuration when modules are running.

39 A2 38MC 03

207

SafeKit User's Guide

208

39 A2 38MC 03

13.Userconfig.xml for a module configuration
13.1 “Macro definition (<macro> tag)” page 210
13.2 “Farm or mirror module (<service> tag)” page 210
13.3 “Heartbeats (<heart>, <heartbeat > tags)” page 213
13.4 “Farm topology (<farm>, <lan> tags)” page 215
13.5 “Virtual IP address (<vip> tag)” page 217
13.6 “File replication (<rfs>, <replicated> tags)” page 225
13.7 “Enable module scripts (<user>, <var> tags)” page 243
13.8 “Virtual hostname (<vhost>, <virtualhostname> tags)” page 244
13.9 “Process or service death detection (<errd>, <proc> tags)” page 245
13.10 “Checkers (<check> tag)” page 251
13.11 “TCP checker (<tcp> tags)” page 253
13.12 “Ping checker (<ping> tags)” page 254
13.13 “Interface checker (<intf> tags)” page 25613.14 “IP checker (<ip> tags)” page 257
13.15 “Custom checker (<custom> tags)” page 258
13.16 “Module checker (<module> tags)” page 260
13.17 “Splitbrain checker (<splitbrain> tag)” page 262
13.18 “Failover machine (<failover> tag)” page 263
Each time you modify userconfig.xml, the configuration must be applied to all thenodes of the cluster onto which the module is installed, to become the active
configuration. Apply the new configuration, modified on node1, on all nodes with (replacenode1, node2 by the nodes name and AM by the module name) :✓the web console by navigating to
Configure the module/

Configuration/Modules configuration/✓or the web console by directly entering the URI/console/en/configuration/modules/AM/config/✓

or the command safekit config -H “node1,node2” –m AM executed on node1

Example of userconfig.xml:<safe><!-- Insert below <macro> <service> tags --></safe>39 A2 38MC 03

209

SafeKit User's Guide

With the web console, the module must be stopped before applying theconfiguration.
With command line, it is possible to apply a new configuration while the module
is running, but only in
ALONE (Ready) or
WAIT (NotReady) states. Thisfeature is called dynamic configuration. Only a restricted subset of parameters
could be changed. If the new configuration cannot be deployed, an error
message is displayed. The attributes that can be dynamically modified arereported hereafter.

13.1

Macro definition (<macro> tag)

13.1.1<macro> example<macro name="ADDR1" value="aa.bb.com"/>
An example of macros usage is given in 15.4 page 277.

13.1.2<macro> syntax<macroname="identifier"value="value"/>

13.1.3<macro> attributes<macroname="identifier"

A character string that identifies the macro.

value="value"

The value that will replace each occurrence of%identifier% in the rest of userconfig.xml.

/>

The syntax %identifier% can also be used in userconfig.xml to represent thevalue of an environment variable named identifier. In case of conflict, it is
the macro value that is expanded.

13.2

Farm or mirror module (<service> tag)

13.2.1<service> example

Example for a mirror module:<service mode="mirror" defaultprim="alone" maxloop="3" loop_interval="24"
failover="on"><!-- Insert below <hearbeat> <rfs> <vip> <user> <vhost> <errd> <check><failover> tags --></service>
Example for a farm module:
210

39 A2 38MC 03

Userconfig.xml for a module configuration<service mode="farm" maxloop="3" loop_interval="24"><!-- Insert below <farm> <vip> <user> <vhost> <errd> <check> <failover> tags -></service>
See examples of <service> definition for a mirror module in 15.1 page 274 and, for a
farm module, in 15.2 page 275.

13.2.2<service> syntax<service mode="mirror"|"farm"|"light"[boot="off"|"on"|"auto"|"ignore"][boot_delay="0"][failover="on"|"off"][defaultprim="alone"|"server_name"|"lastprim"][maxloop="3"] [loop_interval="24"][automatic_reboot="off"|"on"]></service>
Only boot, maxloop, loop_interval and automatic_reboot
attributes can be changed with a dynamic configuration.

13.2.3<service> attributes<service

Top level section of userconfig.xml

mode="mirror"|"farm"|"light"

The mirror keyword sets the module behavior to mirror
architecture mode. The synchronization protocol between the 2servers is defined in section 13.3 page 213.
See mirror.safe application module for an example.
The farm keyword sets the module behavior to farm architecturemode. The definition of the synchronization protocol between
servers is described in section 13.4 page 215.
See farm.safe application module for an example.
The light keyword sets the module behavior to the minimum
needed for one server with software error detection and local
restart only[boot="on"|"off"|"auto"|"ignore"]If set to on, the module is automatically started at boot time.
If set to off, the module is not started at boot time.
If set to auto, the module is automatically started at boot time, if
it was started before the reboot.
Before SafeKit 7.5, the configuration to start the module at bootwas done with the command safekit boot -m AM on | off(which had to be executed on each node). If you prefer to continue
using this command, remove the boot attribute or set it to ignore(the default). The module will not be started at boot time unlessthe safekit boot -m AM on command is executed.

39 A2 38MC 03

211

SafeKit User's Guide

The state of the boot configuration is visible in theusersetting.boot resource. The status of resources is visible in
web console/ Control/Select the node/Resources tab/; with thecommand safekit state -m AM -v
Default value: ignore[boot_delay="0"]

The delay, in seconds, before starting the module at boot.
Default value: 0 (no delay)[failover="on"|"off"]For mirror module only.
If set to on, an automatic failover on the secondary server istriggered if the primary fails or stops.
If set to off, when the primary server fails or stops, the secondary
server waits (no automatic failover is triggered). Only the primcommand can start the secondary server as primary. See
description in 0 page 103
Default value: on[defaultprim="alone"|"server_name"|"lastprim"]

For mirror module only.
defaultprim specifies which server among two servers is thedefault primary server for an application module.
This option is useful when a module is ALONE on a server and themodule is started on the other server.
With defaultprim="alone", the ALONE module becomes PRIM
while the module on the other server becomes SECOND. Valuerecommended avoiding swap of application after reintegration.
With defaultprim="server_name", when the module is runningon two servers, the primary server among the two servers is theone set in defaultprim. This value can be useful for active/active or
N-1 architectures see section 1.5.1 page 20 or section 1.5.2 page
20.
With defaultprim="lastprim", the restarted module becomes
PRIM if it was PRIM before its last stop.
Default value: alone[maxloop="3"]

Number of successive error detections before stop.
This attribute defines the maximum number of "restart" or"stopstart" sequences that can be automatically triggered by
failure detectors before the module locally stops.
The counter is reset to its initial value at the expiration of theloop_interval timeout and upon safekit start, restart,swap, stopstart… administrative commands execution.
Note that a safekit command sent by a detector passes the -iidentity parameter and decrements the counter, whereas
administrator issued commands do not.
For more information, see 13.18.4 page 264.

212

39 A2 38MC 03

Userconfig.xml for a module configuration

This attribute’s value can be changed with a dynamic
configuration.
The maxloop is represented by the resource
heart.stopstartloop. Its current value corresponds to the dateon which the counter was initialized (in the form of a Unix Epochtimestamp); and its assignment date corresponds either to its
initialization or to a stopstart, restart. View the resource history
to see each increment of the loop counter.
Default value: 3[loop_interval="24"]

Time interval during which maxloop applies.
If set to 0, the maxloop counter becomes inactive.
Default value: 24 hours.
This attribute’s value can be changed with a dynamic
configuration.

[automatic_reboot="off"|"on"]If set to on, "stopstart" triggers a reboot instead of stopping andrestarting the module.
Default value: off
This attribute’s value can be changed with a dynamic
configuration.

13.3

Heartbeats (<heart>, <heartbeat > tags)

Heartbeats must be used only for mirror architecture. For farm architecture, see section
13.4 page 215.
The basic mechanism for synchronizing two servers and detecting server failures is theheartbeat, which is a monitoring data flow on a network shared by a pair of servers.
Normally, there are as many heartbeats as there are networks shared by the twoservers. In normal operation, the two servers exchange their states (PRIM, SECOND, theresource states) through the heartbeat mechanism and synchronizes their application
start and stop procedures.
If all heartbeats are lost, it is interpreted as if the other server was down, and the local
server switches to the ALONE state. Although not mandatory, it is better to have twoheartbeat channels on two different networks for synchronizing the two servers to avoid
the split-brain case.

13.3.1<heart> example<heart><heartbeat name=”default” ident="Hb1" />39 A2 38MC 03

213

SafeKit User's Guide<heartbeat name=”net2” ident="Hb2" /></heart>13.3.2<heart> syntax<heart[port="xxxx"] [pulse="700"] [timeout="30000"][permanent_arp="on"]><heartbeat[port="xxxx"] [pulse="700"] [timeout="30000"] name=”network” [ident="name"]>[<!-- syntax for SafeKit < 7.2 --><server addr="IP1_address"|"IP1_name" /><server addr="IP2_address"|"IP2_name" />]</hearbeat>…
</heart>
The <heart> tag and full subtree can be changed with adynamic configuration.

13.3.3<heart>, <heartbeat > attributes<heart[port="xxxx"]

UDP port on which all the heartbeats are exchanged. Default:
depends on the id of the application module. Returned by thesafekit module getports command.

[pulse="700"]

The delay, in milliseconds, between two heartbeat packets.
Default value: 700 ms[timeout="30000"]

Timeout value for heartbeat loss detection.
Default value: 30 000 ms<heartbeat

Definition of one heartbeat. There are as many <heartbeat> tags
as there are networks used to probe servers’ mutual connectivity.
At least one heartbeat must be defined.

[port="xxxx"]

Redefines the UDP port for the heartbeat. Default value is thesame as the one defined in <heart> tag.

[pulse="700"]

Redefines the delay in milliseconds between two heartbeat
packets. Default value is the same as the one defined in <heart>tag.

[timeout="30000"]Redefines the timeout value for heartbeat loss detection. Default
value is the same as the one defined in <heart> tag.

214

39 A2 38MC 03

Userconfig.xml for a module configuration

name="network"

Network named used by the heartbeat. network must be thename of a network set into the SafeKit cluster configuration (fordetails, see 12 page 203).
This attribute is mandatory in new config syntax (since SafeKit
7.2).

[ident="name"]

Set how the heartbeat will be labelled in the web console and ininternal “resources”, i.e.: The internal resource heartbeat.namecan be used in the failover machine described in 13.18 page 263.
If no ident attribute is present the value of the name attribute will
be used.
ident="flow" is a reserved name associated with aheartbeat declared on a replication flow. If you set aheartbeat with ident="flow", automatically the
replication flow will be set on the same network.
If you set ident="flow" without <rfs> configuration,
the module start blocks in WAIT state.

[permanent_arp="on"|"off"]

Regularly, heart sets a permanent ARP entry for the ip addresses
associated with the heartbeats.
On some Linux systems, it may cause heart to freeze. Set thisparameter to off in this case and manually set permanent arp
for the remote server on boot. On Linux, this can be done byinserting the following line into a script that is executed at boot:arp -s hostname hw_addr
Default value: on[<server addr="IP1_address />]Definition of the server address in the heartbeat.
The <server> tag is a legacy syntax used in previous SafeKit
version (before SafeKit 7.2). It’s supported for compatibility
reason but must not be used for new modules.
In the same userconfig.xml, you must not use thesyntax for SafeKit 7.1 and the one for SafeKit 7.2.

13.4

Farm topology (<farm>, <lan> tags)

The basic mechanism to synchronize a farm of servers is a group communication protocol
which automatically detects the available members of the farm. Normally, themembership protocol is configured on all networks connecting the N servers.

13.4.1<farm> example<farm><lan name=”default” /><lan name=”net2” /></farm>39 A2 38MC 03

215

SafeKit User's Guide

For examples of <farm> configuration, see section 15.5 page 278.

13.4.2<farm> syntax<farm [port="xxxx"]><lan name=”network” >[<!-- syntax for SafeKit < 7.2 --><node name="server1" addr="IP1_address" /><node name="server2" addr="IP2_address" />]</lan>…
</farm>The <farm> tag and subtree cannot be changed with a dynamic
configuration.

13.4.3<farm>, <lan> attributes<farm

Begin the definition of a farm topology.

[port="xxxx"]

UDP port with which the membership protocol is exchanged.
Default: depends on the id of the application module. Returnedby the command safekit module getports.

[pulse=”xxxx”]

The period of the membership protocol messages emission.
Longer pulse makes the membership protocol use less bandwidth
but reacts more slowly.

[mlost_count=”xx”]

Number of periods elapsed without message before electing a
new leader.

[slost_count=”xx”]

Number of periods elapsed without messages before declaring afollower node offline.

<lan

Definition of a LAN (i.e., IPv4 broadcast domain, IPv6 link) on
which the membership protocol will be transmitted. At least one
LAN must be defined. Define one such tag per used LAN.

name="network"

Define the name of network used. network must be the name ofa network set into the SafeKit cluster configuration (see 12 page
203).
This attribute is mandatory in new config syntax (since SafeKit
7.2).

[<nodename=”identity”
addr="IP1_address" />]Address and name of the node on this lan. The node tag is alegacy syntax used in previous SafeKit version (before SafeKit7.2). It’s supported for compatibility reason but must not be used
for new modules.
In the same userconfig.xml, you must not use thesyntax for SafeKit 7.1 and the one for SafeKit 7.2.

216

39 A2 38MC 03

Userconfig.xml for a module configuration

13.5

Virtual IP address (<vip> tag)
If you install and run several application modules on the same server, thevirtual IP addresses must be different for each application module.

13.5.1<vip> example in farm architecture

The following example configures load balancing to port 80 and virtual IP address
between nodes in an on-premises cluster:<vip><interface_list><interface check="on" arpreroute="on" arpinterval="60" arpelapse="10"><virtual_interface type="vmac_directed"><virtual_addr addr="192.168.1.222" where="alias" check="on"/></virtual_interface></interface></interface_list><loadbalancing_list><group name="FarmProto"><rule port="80" proto="tcp" filter="on_port"/></group></loadbalancing_list></vip>
See also the example in section 15.2 page 275.

13.5.2<vip> example in mirror architecture

The following example configures the virtual IP address on the primary node of an onpremises cluster:<vip><interface_list><interface check="off" arpreroute="on"><real_interface><virtual_addr addr="192.168.1.222" where="one_side_alias"
check="on"/></real_interface></interface></interface_list></vip>
See also the example in 15.1 page 274.

13.5.3

Alternative to <vip> for servers in different networks

The configuration of a virtual IP address with a <vip> section in userconfig.xml
requires servers in the same IP network (network rerouting and load balancing made atlevel 2).
If servers are in different IP networks, the <vip> section cannot be configured. In thiscase, an alternative is to configure the virtual IP in a load balancer. The load balancer
routes packets to the physical IP addresses of servers by testing an URL status named
health check and managed by SafeKit.

39 A2 38MC 03

217

SafeKit User's Guide

So, SafeKit provides a health check for SafeKit modules. For this, configure the healthcheck in the load balancer with:HTTP protocol
port 9010, the SafeKit web service port
URL /var/modules/AM/ready.txt, where AM is the module nameIn a mirror module, the health check:returns OK, that means that the instance is healthy, when the module state is
PRIM(Ready) or
ALONE(Ready)returns NOT FOUND, that means that the instance is unhealthy, in all other states
In a farm module, the health check:returns OK, that means that the instance is healthy, when the farm module stateis
UP (Ready)returns NOT FOUND, that means that the instance is out of service, in all otherstatesAnother alternative is that you implement a special DNS configuration and a DNS
rerouting command inserted in the SafeKit restart scripts.

13.5.4<vip> syntax

13.5.4.1 Virtual IP loadbalancing in farm architecture<vip [tcpreset="off"|"on"]><interface_list><interface[check="off"|"on"][arpreroute="off"|"on"][arpinterval="60"][arpelapse="1200"]><virtual_interface[type="”vmac_directed”|”vmac_invisible”][addr="xx:xx:xx:xx:xx"]><virtual_addr
addr="virtual_IP_name"|"virtual_IP_address"[where="alias"][check="off"|"on"][connections="off"|"on"]/>…
</virtual_interface></interface></interface_list><loadbalancing_list><group name="group_name"<cluster><host name="node_name" power="integer" />218

39 A2 38MC 03

Userconfig.xml for a module configuration…
</cluster><rule[virtual_addr="*"|"virtual_IP_name"|"virtual_IP_address"][port="*"|"value"]
proto="udp"|"tcp"filter="on_addr"|"on_port"|"on_ipid"/>…
</group>…
</loadbalancing_list></vip>The <vip> tag and subtree cannot be changed with adynamic configuration.

13.5.4.2 Virtual IP failover in mirror architecture
For on-premises SafeKit cluster:<vip [tcpreset="off"|"on"]><interface_list><interface[check="off"|"on"][arpreroute="off"|"on"][arpinterval="60"][arpelapse="1200"]><real_interface><virtual_addr
addr="virtual_IP_name"|"virtual_IP_address"
where="one_side_alias"[check="off"|"on"][connections="off"|"on"]/>…
</real_interface></interface>…
</interface_list></vip>

13.5.5 <vip><interface_list>, <interface>, <virtual_interface>,<real_interface>, <virtual_addr> attributes<vip

39 A2 38MC 03

219

SafeKit User's Guide[tcpreset="off"|"on"]Before unconfiguring the virtual IP address, allconnections with the virtual IP address as IP source arereset. The reset is disabled when set to off.
Default value: on<interface_list><interface

Definition of an interface with virtual IP addresses.
Define as many <interface> sections as there are
network interfaces to configure.

[check="off"|"on"]Set an interface checker on the interface to stop theservice and put it in the WAIT state when the interface isdown. The name of the interface checker isintf.<network_IP_mask> (intf.192.168.0.0).
Default value: on
For more information, see 13.13 page 256.

[arpreroute="off"|"on"]Automatically broadcast gratuitous ARP on virtual IP
addresses defined in <real_interface> section.
Default value: off.

[arpinterval="60"]Time in seconds between two gratuitous ARP.
Default value: 60 s

[arpelapse="1200"]

Time during which gratuitous ARP are sent.
Default value: 1200 s[name="interface name"]

Linux only.
You can specify the name of the network interface on
which the virtual IP addresses will be set.
Ex.: name="bond0"
Default: no value, SafeKit detects the network
interface with virtual IP addresses set on it.

13.5.5.1 <virtual_interface>, <virtual_addr> attributes in farm architecture
Use with farm modules for virtual IP load-balancing:<virtual_interface

Definition of virtual IP addresses configured on an
Ethernet interface.

type=

vmac_directed: advertise the MAC address of one of
the servers as the associated mac address, as withnormal traffic. No promiscuous mode needed. For
details, see 13.5.7.3 page 224.

"vmac_directed"|"vmac_invisible"

vmac_invisible: virtual MAC address never visible in
Ethernet headers to allow broadcasting of switch.

220

39 A2 38MC 03

Userconfig.xml for a module configuration

Needs promiscuous mode. For details, see 13.5.7.2
page 224
Note: can be used for a mirror module with a need oftransparent rerouting.
[addr="xx:xx:xx:xx:xx"]

Unicast virtual MAC address value.
If not set, default is the concatenation of "5A:FE"(Safe) and the first configured virtual IP address inhexadecimal. Ignored in vmac_directed mode.

<virtual_addr

Definition of one Virtual IP address. Set as many
<virtual_addr> sections as there are virtual IP
addresses on the interface.

addr="virtual_IP_name"|"virtual_IP_address"

Name or address of the virtual IP (prefer an IP address
to be independent from the name server).
IPv4 or IPv6 address.

where="alias"

Configuration for farm module: the virtual IP address isdefined on all servers as an alias IP address.
Load balancing rules apply only for this type of virtualIP addresses.
Note : when VMAC is used with a mirror module, sethere where="one_side_alias"[check="off"|"on"]Defines an ip checker on the virtual IP address tostopstart the module when the virtual IP is deleted orin conflict. The name of the ip checker is ip.<addr
value> (ip.192.168.1.99).
Default value: on
For more information, see 13.14 page 257[connections="off"|"on"]Enables counting of the number of active connections
on the virtual address. This count is stored in the
resource named connections.<virtual addr value>(for example: connections.192.168.1.99) which is
assigned every 10 seconds. This value is provided as a
guideline only.
Default value: off

netmask="defaultnetmask"

Linux and IPV4 only. By default, the netmask of the
network interface on which the virtual IP address is
set.
Set the netmask if there are several netmasks on theinterface.

</virtual_interface>

39 A2 38MC 03

221SafeKit User's Guide

13.5.5.2 <real_interface>, <virtual_addr> attributes in mirror architecture
Use with mirror modules for virtual IP failover:<real_interface>

Definition of virtual IP addresses associated with thereal MAC address of the interface.

<virtual_addr

Definition of one virtual IP address. Set as many
virtual_addr sections as there are virtual IP
addresses on the interface.

addr="virtual_IP_name"|"virtual_IP_address"

Name or address of the virtual IP (prefer an IP address
to be independent from the name server).

where="one_side_alias"

The Virtual IP address will be aliased on the server onwhich the module becomes PRIM or ALONE.

[check="off"|"on"]Defines an ip checker on the virtual IP address tostopstart the module when the virtual IP is deleted orin conflict. The name of the ip checker is ip.<addr
value> (ip.192.168.1.99).

IPv4 or IPv6 address.

Default value: on
For more information, see 13.14 page 257.
[connections="off"|"on"]Enables counting of the number of active connections
on the virtual address. This count is stored in the
resource named connections.<virtual addr value>(for example: connections.192.168.1.99) which is
assigned every 10 seconds. This value is provided as a
guideline only.
Default value: off

netmask="defaultnetmask"

Linux and IPV4 only. By default, the netmask of the
network interface on which the virtual IP address is
set.
Set the netmask if there are several netmasks on theinterface.

</real_interface>

13.5.6<loadbalancing_list>, <group>, <cluster>, <host> attributes

For load-balancing examples, see 15.5 page 278.
Use with farm module.
<loadbalancing_list>

222

39 A2 38MC 03

Userconfig.xml for a module configuration<groupDefinition of a load balancing group. Define as many
sections as there are groups. An example is given in
15.5.3 page 279.

name="group_name"Name of the load balancing group.

<clusterDefinition of the server set on which the load current
group balancing will be applied. If no <cluster> section
is defined, the rules apply to all servers of the farm.

<host

Definition of one node in the cluster. Define as many
hosts sections as there are nodes configured for themodule.

name = "node_name"Define the name of the host. node_name must be thename of a node name set into the SafeKit cluster
configuration (see 12 page 203).

power = "value"

Relative weight to apply to the current node in this loadbalancing group’s cluster. Can be equal to 0, whichmeans no traffic will be dispatched to this node. See
section 13.5.7.4 page 224 for more information.

</cluster><rule

Definition of a load balancing rule for the group. Define
as many sections as there are load balancing rules forthis group.

[virtual_addr="*" |"virtual_IP_address"|"virtual_IP_name"]

Virtual IP name or address scope of the rule.

[port="*"|"value"]

TCP or UDP port to which the load balancing rule applies.

By default, all virtual IP addresses: *By default, all ports: *proto="udp" | "tcp" |"arp"

proto="udp"
Load balancing rule applies to the UDP protocol.
proto="tcp"
Load balancing rule applies to the TCP protocol.
proto="arp"Load balancing rule applies to the IP<->MAC resolution
protocol (arp or neighbour discovery)

filter="on_addr" |"on_port" |"on_ipid"

filter="on_addr"
Load balancing criteria is the source IP address (client,
far end of the connection) (see 15.5.1 page 278).
filter="on_port"
Load balancing criteria is the source port (client, far endof the connection) (see 15.5.1 page 278).
filter="on_ipid"
Load balancing is made on the client ip_id at input.

39 A2 38MC 03

223

SafeKit User's Guide

Useful for UDP. No sense for TCP and for IPv6 addresses(see example in 15.5.2 page 279).

13.5.7<vip> Load balancing description

13.5.7.1 <vip> prerequisites
See network prerequisites described in 2.3.2 page 31.
13.5.7.2 What is the vmac_invisible type?
When type=“vmac_invisible”, a virtual MAC address is mapped on the virtual IP
address with a unicast MAC Ethernet address on several network nodes. When a network
device tries to resolve the virtual IP address into its corresponding MAC address, theSafeKit servers respond with the virtual MAC address. However, SafeKit servers use its
physical MAC address to communicate. To “see” the packets sent to the virtual MAC
address the interface is set to promiscuous mode. So, the virtual MAC address is invisible
to layer 2 network devices. Ethernet switches therefore forward virtual MAC address
directed packets to all the ports in the same vlan as the source, reaching all the serversof the farm. A kernel module running on each farm server is responsible for filtering out
the packets that should not be processed by a given farm node, according to the loadbalancing rules defined.
With the virtual MAC address technology, the failover time is null. There is no network
rerouting after a failure: all network equipment keeps their mapping virtual IP address,
virtual MAC address.
To test a virtual MAC address in your network, see 4.3.7 page 84
13.5.7.3 What is the vmac_directed type?
When type=“vmac_directed”, there is in fact no virtual MAC address. Farm servers
reply to virtual IP resolution requests with their own physical MAC address. A kernel
module running on each farm server is responsible for filtering and dispatching thepackets to their designated target farm node according to the load balancing rules
defined. In vmac_directed mode there is a short failover time for clients that haveresolved the virtual IP address as the MAC address of the failed server. This iscomparable to what happens in “real interface” mode. Clients that have another farmserver’s MAC address in their cache are not affected.
To help minimize failover time in ipv4, set the arpreroute attribute to “on” on thecorresponding “<interface>” tag, and tune the arpelapse and arpinterval attributes to thedesired values. Ipv6 does not need arpreroute, it has a built-in mechanism that takes
care of the failover.
13.5.7.4 How does load balancing work?
On all the servers of the farm, the load balancing algorithm filters received packets
according to the identity of the sender. The criteria to check is defined by configuration inuserconfig.xml: client IP address, client port… (i.e.: level 3 load balancing), or
requestor address (arp rules, i.e., level 2 load balancing). The criteria are hashed into avalue representing the server on which the packet is to be accepted.

224

39 A2 38MC 03

Userconfig.xml for a module configuration

When a server fails, the membership protocol reconfigures the filters to re-balance thetraffic of the failed server on the available servers.
Each server can have a power (=1, 2…) and then takes more or less traffic. The power isimplemented by the number of bits set to 1 in the hash table (a bitmap of 256 bits).
A bitmap example is given in 4.3.5 page 82.

13.6

File replication (<rfs>, <replicated> tags)

For mirror modules only.
In Linux, you must set the same value for uid/gid on the two nodes for replicating filepermissions. When replicating a filesystem mount point, you must apply a special
procedure described in 13.6.4.2 page 234.
In Windows, it is strongly recommended to enable the USN journal on the drive that
contains the replicated directory as described in 13.6.4.3 page 235.
If you install and run several application modules on the same server, thereplicated directories must be different for each application module.

13.6.1<rfs> example

Example in Windows:<rfs async="second"><replicated dir="c:\safedir" mode="read_only"/></rfs>
Example in Linux:<rfs async="second"><replicated dir="/safedir" mode="read_only"/></rfs>
See also the example in 15.4 page 278.

13.6.2<rfs> syntax<rfs[acl="on"|"off"][async="second"|"none"][iotimeout="nb seconds"][roflags="0x10"|"0x10000"][locktimeout="100"][sendtimeout="30"][nbrei="3"][ruzone_blocksize="8388608"][namespacepolicy="0"|"1"|"3"|"4"][reitimeout="150"][reicommit="0"][reidetail="on"|"off"]

39 A2 38MC 03

225

SafeKit User's Guide[allocthreshold="0"][nbremconn ="1"][checktime="220000"][checkintv="120"][nfsbox_options="cross"|"nocross"][scripts="off"][reiallowedbw=”20000”][syncdelta=”nb minutes”][syncat=”synchronization scheduling”]>[<flow name=”network” >[<!-- syntax for SafeKit < 7.2 --><server addr="IP_address_1" /><server addr="IP_address_2" />]</flow>]<replicated dir="absolute path of a directory"[mode="read_only"]><tocheck path="relative path of a file or subdir" /><notreplicated path="relative path of a file or subdir" /><notreplicated regexpath="regular expression on relative path of a file orsubdir" />…
</replicated></rfs>
Only async, nbrei, reitimeout and reidetail
attributes of <rfs> tag can be changed with a dynamic
configuration. The <flow> tag, describing the replication
flow, can also be changed dynamically.

13.6.3<rfs>, <replicated> attributes<rfs[mountoversuffix= "suffix"]Linux only.
During the module configuration, the replicated directory "/a/dir"is renamed "/a/dirsuffix". The directory /a/dir is created and itis:a mount point to /a/dirsuffix when the module is started
a link to "/a/dirsuffix" when the module is stopped
By default, suffix value is “_For_SafeKit_Replication”.

If there is a hard failure, then the symbolic link will not
be restored. In this case, you must restore the symbolic
link manually.

226

39 A2 38MC 03

Userconfig.xml for a module configuration

Restriction
You cannot explicitly specify a root file system as a
replicated directory (because of the directory renamethat is not allowed across a file system). The workaround is to manipulate the fstab file as described ina KB on https://support.evidian.com.

When the module is started, NEVER ACCESS files in"/a/dirsuffix", otherwise the modifications will not be
replicated, and the system will become inconsistent.
ALWAYS ACCESS replicated files through "/a/dir".
[acl="on" | "off"]

Setting acl to on activate the replication of ACL on files anddirectories.
Default value: off
Restrictions for Windows
ACL replication will not work if the SYSTEM account does
not have the "Full control" access right on all thereplicated forest.
File ACLs are replicated literally (as SID values),
therefore ACL granted to locally defined users andgroups will be meaningless on the remote system.
File encryption and file compression attributes are notsupported.

[async="second" |"none"]

Setting async mode to second is a way to improve file replication
performances: modification operations are cached on the secondary
server and the acknowledgements are sent more quickly to theprimary server.
Setting async mode to none ensures more robustness: modification
operations are put on disk of the secondary before sending
acknowledgement to the primary.
With async="second", in case of double failure at the same time ofboth PRIM and SECOND servers, if the PRIM server cannot restart,
then the SECOND server does not have up-to-date data on its disk.
There is data loss if the SECOND server is forced to start as primarywith the prim command.
Default value: second
This attribute’s value can be changed with a dynamic
configuration.

[packetsize]

Linux only.
Maximum size in bytes for NFS replication packets. It must be lower
than the maximum size allowed by the NFS server of both servers.

39 A2 38MC 03

227

SafeKit User's Guide

When it is set into the configuration, it is used as mount options for
rsize and wsize.
By default, the size is the one of the NFS server.
[reipacketsize="
8388608"]

Maximum size in bytes of reintegration packets.
In Linux, this value must be less or equal to packetsize.
Default value in Linux: value of packetsize if it is set into theconfiguration and is lower than 8388608; else 8388608
Default value in Windows: 8388608 bytes[ruzone_blocksize="8388608"]

Size of a zone for the modification bitmap of a file.
It must be a multiple of reipacketsize attribute.
Default value: value of reipacketsize if it is set into theconfiguration; else 8388608[iotimeout]

Windows only.
IO time out in seconds in the Windows file system filter. If an IO
cannot be replicated and if the timeout expires in the filter, then the
PRIM server becomes ALONE.
If not set, the default value is dynamically calculated.

[roflags="0x10"|"0x10000"]

Windows only.
To ensure the consistency of the data replicated on the 2 servers,
the modification of the replicated directories/files must only takeplace on the PRIM server. If changes are made on the SECOND
server, they are notified in the module log with the identification of
the process responsible so that the administrator can correct thisanomaly. This is the behavior with roflags="0x10".
Since SafeKit 7.4.0.31, the module can also be stopped on the
SECOND server by setting roflags="0x10000".
Default value: 0x10

[locktimeout="100"]

Timeout in seconds for replication requests. If a request cannot be
served within this timeout, the PRIM server becomes ALONE.
Default value: 100 seconds[sendtimeout="100"]

Since SafeKit > 7.4.0.5
Timeout in seconds for sending TCP packets to the remote node. If
a packet cannot be sent within this timeout, the PRIM server
becomes ALONE. Increase this value in case of low networks.
Default value: 30 seconds
In SafeKit 7.4.0.5, the default value was 12O seconds.

[nbrei="3"]228

Number of reintegration threads running in parallel forresynchronizing files.

39 A2 38MC 03

Userconfig.xml for a module configuration

Default value: 3This attribute’s value can be changed with a dynamic
configuration.
[namespacepolicy="0"|"1"|"3"|"4"]

In Windows, with namespacepolicy="1", zone reintegration after
reboot when the module has been properly stopped is not active.
To enable it in Windows, set namespacepolicy="3". It activates
the USN change journal on the volume containing the replicateddirectories (see fsutil usn command for creating USN change
journal on a volume). Even with this configuration, full reintegration
is used instead of zone reintegration when:
the USN change journal associated with the volume has been
deleted/recreated for administration reasons
discontinuity in the USN journal is detected
When zone synchronization is not possible (on the first reintegrationor when zones are not available), the files that need to besynchronized are fully copied. If this reintegration does notcomplete, the next one will copy again these files. To avoid this, set
namespacepolicy="4". This option also enables USN journal
checking in Windows.
Set namespacepolicy="0" to deactivate the zone reintegration on
Windows or Linux.
Default value: 4 since SafeKit > 7.4.0.5 (not supported in previous
releases)[reitimeout="150"]

Timeout in seconds for reintegration requests. The timeout can beincreased to avoid reintegration failure on heavy load of the primaryserver.
Default value: 150 seconds
This attribute’s value can be changed with a dynamic
configuration.

[reicommit="0"]Linux only.
Set reicommit="nb blocks" to commit every (nb blocks)*
reipacketsize when reintegrating one file (in addition to thecommit at the end of the copy). This can help to succeed
reintegration of big files but slows down reintegration time.
Default value: 0 that means no intermediate commit[reidetail="on"|"off"]

39 A2 38MC 03

Detailed logging for reintegration.
Default value: off229

SafeKit User's Guide

This attribute’s value can be changed with a dynamic
configuration.
[allocthreshold="0"]

Windows only.
Size in Gb to apply the allocation policy before reintegration.
When allocthreshold> 0, enable fast allocation of disk space forfiles to be synchronized on the secondary node. This feature avoidsa timeout when the primary writes at the end of the file, when thefile is large (> 200 Gb) and not yet completely copied.
Since SafeKit 7.4.0.64, the allocation policy has changed and isapplied for:
Newly created files (files that did not exist on the secondary
when the reintegration starts)
Files with size on the primary >= allocthreshold (sizein Go)
Full synchronization:•

on first reintegration•

on start with full synchronization (safekit second
fullsync)•

when synchronization by zones is disabled(namespacepolicy=”0”)

Default value: 0 (that disables the feature)[nbremconn="1"]

Number of TCP connections between the primary and the secondary
nodes.
This value may be increased to improve the replication andsynchronization throughput when the network has high latency (incloud for instance).
Default value: 1[checktime="220000"]

Linux only.
Timeout in milliseconds for the null request that checks the local
replicated file system. Run the safekit stopstart command when
the timeout is reached.
Default value: 220 000 milliseconds[checkintv="120"]

Linux only.
Interval in seconds between two null requests.
Default value: 120 seconds

nfsbox_options="
cross"|"nocross"

230

Windows only.

39 A2 38MC 03

Userconfig.xml for a module configuration

It specifies the policy to apply when a reparse point of type
MOUNT_POINT is present in the replicated directory tree. This policy
applies to all replicated directories.
MOUNT_POINT reparse points in NTFS can represent two types ofobjects: an NTFS mount point (for example the D:\ directory) oran NTFS "directory junction" (a form of "symbolic link" to anotherpart of the file system namespace).
When nfsbox_options="cross", the MOUNT_POINT reparse point
object itself is not replicated/reintegrated. It is evaluated, and thereintegration/replication process the target content as it would do
for the content of a standard directory. This is useful for instancewhen a replicated directory is a mount point (e.g., replicating a"drive letter" root). This is the default configuration value.
When nfsbox_options="nocross", the MOUNT_POINT reparse point
object itself is replicated/reintegrated, but not evaluated.
Reintegration does not descend into the target of the reparse point.
This is useful for instance when a replicated directory tree contains
NTFS "junctions" that point to another part of the replicated tree(e.g., when replicating a PostgreSQL database, as PostgreSQL is
known to need such objects).
Default value: cross[scripts="on" | "off"]

scripts="on" activates _rfs_* script callbacks used to implement
external data replication management (see Linux drbd.safe module
for more information)
Default value: off[reiallowedbw=”20000”]

When defined, this attribute specifies the maximum bandwidth thatthe reintegration phase may use (for instance 20000 KB/s), in kilo
bytes per second (KB/s).
Due to implementation trade-off, a +/-10% fluctuation of theeffectively used bandwidth is to be expected.
The replication bandwidth is not affected by thisparameter.
By default, the attribute is not defined, and the bandwidth used by
the reintegration is not limited[syncdelta=”nb
minutes”]

When <=1, the attribute is ignored and the default failover andstart policy is applied: only an up-to-date server can start asprimary or run a failover.
When >1, it changes the default failover and start policy. The not
up-to-date server can become primary but only if the elapsed time,in minutes, since the last synchronization is lower than the
syncdelta value (see 13.6.4.4 page 236).
Default value: 0 minutes

39 A2 38MC 03

231

SafeKit User's Guide[syncat="synchro
nizationscheduling"]

Default: real-time replication and automatic synchronization (noscheduling)
Use syncat for scheduling the synchronization of replicateddirectories on the secondary node (see 13.6.4.10 page 242). The
module must be started for enabling this feature. Once
synchronized, the module blocks in the WAIT (NotReady) state until
the next synchronization.
The scheduling is based on native job scheduler:
On Unix, the job is defined in the safekit user’s crontab
On Windows, the job is defined as a system task
You must configure syncat with the syntax of the native job
scheduler. For instance, for synchronizing daily, after midnight:
in Windows
syncat="/SC DAILY /ST 00:01:00"
in Unix
syncat="01 0 * * *"See crontab documentation in Unix and schtasks.exe
documentation in Windows, for the full syntax ofscheduled date and time.

Since SafeKit configuration is just a front end to thejob scheduler, when scheduling is not working, pleasecheck first for syntax errors.
[<flow name=”network”>[<serveraddr="IP_1" /><server
addr="IP_2" /> ]</flow>]

Obsolete configuration preserved for backwards compatibility.
When this section is not defined, the replication flow uses the same
network as the heartbeat with ident="flow" if there is one, if not it
uses the first heartbeat (see 13.3 page 213).
If you define this section, be coherent with heartbeat
ident="flow", if there is one, because default failover rules apply
to this heartbeat (see 13.18.5 page 265).
This <flow> tag subtree can be changed with a dynamic
configuration for setting a new replication flow for
instance.
The name attribute of <flow> define the network used for
replication flow. It must present in global cluster configuration (see12 page 203).
The <server> tag is a legacy syntax used in previous SafeKit
version (before 7.2). It’s supported for compatibility reason but
must not be used for new modules.

232

39 A2 38MC 03

Userconfig.xml for a module configuration

In the same userconfig.xml, you must not use thesyntax for SafeKit 7.1 and the one for SafeKit 7.2.
<replicatedBegin the definition of replicated directories.
Set as many lines as there are replicated directories.

dir="/abs_path"

Absolute path of a directory to replicate.

[mode="read_only"]

Read-only access rights on the secondary machine for replicateddirectories to avoid corruption<notreplicatedpath="relative"/>

Relative path of a file or sub-directory in a replicated directory. The
file (or sub-directory) is not replicated. Set as many lines as there
are non-replicated files or sub-directories.

<notreplicatedregexpath="regul
ar expression"/>

Regular expression on the name of entries under the replicateddirectory :Replicate all except entries matching the regular expression
For example, to avoid replicating entries with the extension .tmp
or .bak in the /safedir directory or its sub-directories :<replicated dir="/safedir"><notreplicated regexpath=".*\.tmp$" /><notreplicated regexpath=".*\.bak$" /></replicated>
Note that /safedir/conf/config.tmp.swap is replicated.

Replicate only those entries in the directory that match theregular expression after the !
For example, to replicate only entries with the extension .mdf or.ldf in the /safedir directory or its sub-directories :<replicated dir="/safedir"><notreplicated regexpath="!.*\.mdf$" /><notreplicated regexpath="!.*\.ldf$" /></replicated>
Rename between not replicated and replicated files is
not supported.
The regex engine is POSIX Extended regex (see POSIX
documentation):✓in Windows, case insensitive mode✓in Linux, case sensitive mode
As regular expressions are defined inside the XML fileuserconfig.xml, special characters interpreted by
XML like '<' or '>' cannot be used in regularexpressions.

39 A2 38MC 03

233

SafeKit User's Guide<tocheck
path="relative"/>

13.6.4

Relative path of a file or sub-directory in a replicated directory.
Checks the presence of the file or sub-directory before starting thereplication mechanism. Avoids errors such as starting replication on
an empty file system. Set as many lines as there are files or subdirectories to check.

<rfs> description

13.6.4.1 <rfs> prerequisites
See file replication prerequisites described in 2.2.4 page 30.
13.6.4.2 <rfs> LinuxOn Linux, interception of data is based on a local NFS mount. And the replication flow
between servers is based on NFS v3 / TCP protocol.
The NFS mount of replicated directories from remote Unix clients is not supported. The
NFS mount of other directories can be made with standard commands.

Procedure for replicating a mount point
When replicating a mount point in Linux, the module configuration fails with the error:
Error: Device or resource busy
In the following, we take the example of PostgreSQL module that set as replicateddirectories /var/lib/pgsql/var and /var/lib/pgsql/data. The userconfig.xml of themodule contains:<rfs … ><replicated dir="/var/lib/pgsql/var" mode="read_only" /><replicated dir="/var/lib/pgsql/data" mode="read_only" /></rfs>
These directories are mount points as shown by the result of the command df –H. It
returns for instance:/dev/mapper/vg01-lv_pgs_var … /var/lib/pgsql/var/dev/mapper/vg02-lv_pgs_data … /var/lib/pgsql/data
You must apply the following procedure for configuring the module to replicate these
directories.
It is the same procedure for all mounts points that must be replicated.

umount the file systems by running the commands:
umount /var/lib/pgsql/var
umount /var/lib/pgsql/data
configure the module by running the command:/opt/safekit/safekit config –m postgresql
The configuration should succeed (no errors)
check the symbolic links created by running the command ls -l /var/lib. It
returns:

234

39 A2 38MC 03

Userconfig.xml for a module configuration

lrwxrwxrwx 1 root var -> var_For_SafeKit_Replication
lrwxrwxrwx 1 root data -> data_For_SafeKit_Replication
edit /etc/fstab and change the two lines:/dev/mapper/vg01-lv_pgs_var /var/lib/pgsql/var ext4…
/dev/mapper/vg02-lv_pgs_data /var/lib/pgsql/data ext4…
with/dev/mapper/vg01-lv_pgs_var/var/lib/pgsql/var_For_SafeKit_Replication ext4…
/dev/mapper/vg02-lv_pgs_data/var/lib/pgsql/data_For_SafeKit_Replication ext4..
mount the file systems by running the commands:
mount /var/lib/pgsql/var_For_SafeKit_Replication
mount /var/lib/pgsql/data_For_SafeKit_Replication

Apply this procedure on both nodes if replicated directories are mount point on
both nodes. Once applied, you can use the module as usual: i.e., safekit start
stop etc ...

To protect the start of the module on a non-mounted and empty directory, you
can insert in userconfig.xml the checking of a file inside the replicateddirectory. Example for /var/lib/pgsql/var (do the same for/var/lib/pgsql/data with a file inside this directory which is always present):<replicated dir="/var/lib/pgsql/var" mode="read_only"><tocheck path="postgresql.conf" /></replicated>
If you want to unconfigure the module (or uninstall whole SafeKit package), you mustreverse this procedure by:umount the file systems with:umount /var/lib/pgsql/var_For_SafeKit_Replication
umount /var/lib/pgsql/data_For_SafeKit_Replication
de-configure the module with /opt/safekit/safekit deconfig -m postgresql
edit /etc/fstab to undo previous editing
mount the file systems with:mount /var/lib/pgsql/var
mount /var/lib/pgsql/data
13.6.4.3 <rfs> Windows
On Windows, interception of data is based on a file system filter. And the replication flow
between servers is based on NFS v3 / TCP protocol.
The <rfs> filter may not work correctly with some anti-viruses.

39 A2 38MC 03

235

SafeKit User's Guide

On Windows, you can mount remotely a replicated directory from a workstation. If you
want to mount with the virtual name instead of the digital virtual IP address, you must
set the two following registry keys on the server side:[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa]"DisableLoopbackCheck"=dword:00000001[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters] "DisableStrictNameChecking"=dword:00000001
In Windows, to enable zone reintegration after server reboot, when the module has beensuccessfully stopped, the <rfs> component uses the NTFS USN log to verify that the
information recorded on the zones is still valid after the reboot. When the control
succeeds, the zone reintegration can be applied to the file; otherwise, the file must be
fully copied.
By default, only the system drive has a USN log active. If the replicated directories arelocated on a different drive than the system drive, you must create the log (with fsutil
usn command). See SK-0066 for an example.
13.6.4.4 <rfs> replication and failover
With its file-replication function, mirror architecture is particularly suitable for providing
high availability for back-end applications with critical data to protect against failure. The
reason is that the secondary server data is strongly synchronized with the primary server
data. A synchronized server is considered as up-to-date and only an up-to-date server
can start as primary or run a failover.
If the application availability is more critical than the application data, this default policy
can be relaxed by allowing a server to become primary if the time elapsed since the lastsynchronization is below a configurable delay. This is configured by setting the
syncdelta attribute of the <rfs> tag:
syncdelta <= 1
The attribute is ignored and the default failover and start policy is applied. The
default value is 0.
syncdelta > 1
When the last up-to-date server is not responding, the not up-to-date server canbecome primary but only if the elapsed time since the last synchronization islower than the syncdelta value (in minutes).
This feature is implemented with:
rfs.synced resource
When syncdelta is > 1, the rfs.synced resource is managed. This resource is UP
if the replicated data are consistent and if the elapsed time, in minute since thelast synchronization is lower than the syncdelta value.
syncedcheck checker
When syncdelta is > 1, this checker is running. It sets the value for the
rfs.synced resource.
rfs_forceuptodate failover rule
When syncdelta is > 1, the following failover rule is valid:

236

39 A2 38MC 03

Userconfig.xml for a module configuration

rfs_forceuptodate:if (heartbeat.* == down && cluster() == down &&rfs.synced == up && rfs.uptodate == down) then rfs.uptodate=up;
This rule leads to the primary start of the server when the up-to-date server is not
responding and if the server is isolated and can be considered as synchronizedaccording to syncdelta value.
13.6.4.5 <rfs> replication verification
You can check for the module, named AM, that files are identical on the primary and thesecondary, by running the following command on the SECOND server: safekit
rfsverify –m AM. Run safekit rfsverify –m AM > log to redirect the command
output into the file named log.
This output of the command is a log like that of the reintegration in which the files to becopied (therefore different) are indicated. When on the primary, there is activity on thereplicated directories, an anomaly may be detected while there is no difference between
the files in the following cases:
on Windows because modifications are made on disk before being replicatedwith async="second" (default) because reads can bypass the asynchronous
writes.
To check if there is really an inconsistency, you must re-run the command on thesecondary server making sure that there is no more activity on the primary.
On Windows, some files are systematically seen as erroneous by the verifier while there
is no difference. This occurs when files are modified with SetvalidData: files areextended without resetting the new extension and the reads return random data fromthe disk.
It is strongly recommended to run this command only when there are noaccesses to the replicated directories on the primary.

13.6.4.6 <rfs> file changes since the last synchronization
Before starting a secondary server, it may be useful to evaluate the number of files anddata that have been changed on the primary server since the secondary server has
stopped. This feature is provided by running the following command on the ALONE server:
safekit rfsdiff –m AM. Run safekit rfsdiff –m AM > log to redirect the command
output into the file named log.
This command runs on-line checks of regular files content of the module AM. It scans theentire replicated tree and displays the number of files that have been modified as well as
the size that need to be copied. It also displays estimation for the synchronization
duration. This is only estimation since only regular files are scanned and some othermodifications may occur until the synchronization is run by the secondary server.
This command must be used with caution on a production server since it leads to an
overhead on the server (for reading trees and files with locking). On Windows, rename offiles can fail during the evaluation.
It is strongly recommended to run this command only when there are noaccesses to the replicated directories.

39 A2 38MC 03

237

SafeKit User's Guide

13.6.4.7 <rfs> replication and reintegration bandwidth
The replication component monitors, on the PRIM server, the bandwidth used byreplication and reintegration write requests.
Two resources (rfs.rep_bandwidth and rfs.rei_bandwidth) reflect the average
bandwidth used by replication and reintegration respectively during the last 3 seconds,
expressed in kilo bytes per second (KB/s).
If the replication load is IO intensive, the reintegration phase may saturate the network
link and significantly slow down the application. In such a case, the <rfs> reiallowedbw
attribute may be used to limit the bandwidth taken by the reintegration phase (see13.6.3 page 226). Please note that limiting the reintegration bandwidth will make thereintegration phase longer.
There are also 2 resources that reflect the network bandwidth (in in Kbytes/sec) used
between nfsbox processes, that run on each node to implement replication andreintegration:
rfs.netout_bandwidth is the network output bandwidth
rfs.netin_bandwidth is the network input bandwidth
You can observe the value of rfs.netout_bandwidth on the primary or
rfs.netin_bandwidth on the secondary to know the modification rate at the time ofobservation (write, create, delete, …). The history of the resource values gives an
overview of its evolution over time.
The value of the bandwidth depends on the application, system, and network activity. Its
measurement is available for information purposes only.
13.6.4.8 <rfs> synchronization by date
SafeKit 7.2 offers a new command safekit secondforce –d date –m AM that forces themodule AM to start as secondary after copying only files modified after the specified date.
This command must be used with cautions since the synchronization will
not copy files modified before the specified date. It is the administrator’s
responsibility to ensure that these files are consistent and up-to-date.

The date is in the format of YYYY-MM-DD[Z] or "YYYY-MM-DD hh:mm:ss[Z]" or YYYYMM-DDThh:mm:ss[Z], where:•

YYYY-MM-DD indicates the year, month, and day•

hh:mm:ss indicates the hours, minutes, and seconds•

Z indicates that the time is in UTC time zone; when not set the time is in local
time zone

For instance:•

safekit secondforce -d 2016-03-01 –m AM for copying only files modified after
the 1st of March 2016•

safekit secondforce -d "2016-03-01 12:00:00" –m AM for copying only files
modified after the 1st of March 2016 at 12h, local time zone

238

39 A2 38MC 03

Userconfig.xml for a module configuration•

safekit secondforce -d 2016-03-01T12:00:00Z –m AM for copying only files
modified after the 1st of March 2016 at 12h, UTC time zone

This command may be useful in the following case:•the module is stopped on the primary server and a backup of the replicated data
is done (on a removable drive for instance)•

the module is stopped on the secondary server and the replicated data is restoredfrom the backup. It may be the first start-up or the repair of the secondary
server.

•the module is started on the primary server that becomes ALONE•

the module is started on the secondary with the command safekit secondforce–d date –m AM where the date is the backup dateIn this case, only the files modified since the backup date will be copied (full copy),
instead of the full copy of all files.
In Windows, the file modification date on the secondary server ischanged when the file is copied by the synchronization process.
Therefore, safekit secondforce –d date –m AM, where date is prior
to the last reintegration on this server, has no interest.
13.6.4.9 <rfs> external synchronization
On the first synchronization, all replicated files are fully copied from the primary node to
the secondary node. During the following synchronizations, necessary when thesecondary node comes back, only zones modified, during the secondary downtime, offiles that have been modified on the primary node during the secondary node downtime.
When the replicated directories are voluminous, the first synchronization can take a lot oftime especially if the network is slow. For this reason, since SafeKit> 7.3.0.11, SafeKit
provides a new feature to synchronize a large amount of data that must be used inconjunction with a backup tool.
On the primary node, simply back up the replicated directories and pass thesynchronization policy to the external mode. The backup is transported (using an
external drive for instance) and restored to the secondary node, which is also configured
to perform external synchronization. When the module is started on the secondary node,
it copies only the file areas that were modified on the primary node since the backup
The external synchronization relies on a new SafeKit command safekit rfssync thatmust be applied on both nodes to set the synchronization policy to external. This
command requires as arguments:•

the role of the node (prim | second)•a unique identifier (uid)External synchronization procedure
The external synchronization procedure, described below, is the procedure to be followedin the case of a cold backup of the replicated directories. In this case, the applicationmust be stopped, and any modification of the replicated directories is prohibited until the

39 A2 38MC 03

239

SafeKit User's Guide

module and the application are started, inmust be strictly adhered to.

ALONE(Ready). The order of operations

The external synchronization procedure, described below, is the procedure to be followedin the case of a hot backup of replicated directories. In this case, the module is
ALONE(Ready); the application is started and changes to the contents of the replicateddirectories are allowed. The order of operations must be strictly adhered to.

240

39 A2 38MC 03

Userconfig.xml for a module configuration

safekit rfssync commandsafekit rfssync
external prim <uid>[–m AM]

Set the synchronization policy to external. It is identified
by the value of uid (at max 24 char).

safekit rfssync
external second <uid>[–m AM]

Set the synchronization policy to external. It is identified
by the value of uid (at max 24 char).

safekit rfssync –dprim <uid> [–m AM]
safekit rfssync –dsecond <uid> [–m AM]

Disable the replicated directories change detection between
the cold backup/restore and the start of the module.

safekit rfssync full[–m AM]

Set the synchronization policy to full. This will copy all
files in their entirety on the next synchronization.

safekit rfssync

Display the current synchronization policy

The node is the primary one, the source for synchronizing
data.

The node is the secondary one, the destination forsynchronizing data

Use this option with caution since the external
synchronization may not properly detect allchanges to be copied.

Internals
The synchronization policy is represented by module’s resources:
usersetting.rfssyncmode, usersetting.rfssyncrole, usersetting.rfssyncuid andrfs.rfssync:
usersetting.rfssyncmode=”default”(usersetting.rfssyncrole=”default”, usersetting.rfssyncuid=”default”)
These values are associated with the standard synchronization policy, which isapplied by default. It consists of copying only the modified areas of the files.
When this policy cannot be applied, the modified files are copied in their entirety.
usersetting.rfssyncmode=”full”(usersetting.rfssyncrole=”default”, usersetting.rfssyncuid=”default”)
These values are associated with the full synchronization policy. It is applied:othe first time the module is started after its first configurationoon safekit commands (safekit second fullsync ; safekit rfssync
full ; safekit primforce ; safekit config ; safekit deconfig)oon change of pairing for the module

39 A2 38MC 03

241

SafeKit User's Guide

The full synchronization policy will copy all files in their entirety on the next
synchronization.
usersetting.rfssyncmode=”external”, usersetting.rfssyncrole=”prim |second” and usersetting.rfssyncuid=”uid”
These values are associated with the external synchronization policy assignedwith the commands safekit rfssync external prim uid and safekit rfssync
external second uid. The next synchronization will apply the external
synchronization policy.
rfs.rfssync=”up | down”
This resource is only up when the synchronization policy, defined by the previous
resources, can be applied.
When the synchronization policy is not the default policy, the synchronization policy
automatically returns to the default mode after successful synchronization.
In some cases, external synchronization cannot be applied, and the secondary nodestops with an error specified in the module log. In this situation, you must either:complete the external synchronization procedure if this has not been done in its
entirety on the 2 nodes
fully reapply the external synchronization procedure on the 2 nodes
revert to the full synchronization policy (safekit rfssync full command)
apply the synchronization by date, using the date of the backup (see 13.6.4.8
page 238). Unlike external synchronization, synchronization by date will copy thefiles, modified on the primary node, in their entirety (instead of just modified
parts).
13.6.4.10<rfs> scheduled synchronization

By default, SafeKit provides real-time file replication and automatic synchronization. On
heavy loaded server or high latency network, you may want to let the secondary node
weakly synchronized. For this, you can use the syncat attribute for scheduling
replicated directories synchronization on the secondary node. The module must be
started for enabling this feature. Once synchronized, the module blocks in the WAIT
(NotReady) state until the next synchronization schedule. It is implemented with:the resource rfs.syncat that is set to up on the scheduled dates and set todown after the data synchronization
the failover rule rfs_syncat_wait that blocks the module into the state WAIT
(NotReady) until the rfs.syncat resource is up
If you want to manually force the synchronization, you can run the command: safekit
set –r rfs.syncat –v up –m AM while the module is in the WAIT (NotReady) state.
With syncat, you just have to configure the scheduled time for the synchronization withthe syntax of the native job scheduler: crontab in Linux and schtasks.exe in Windows(see 13.6.3 page 226).

242

39 A2 38MC 03

Userconfig.xml for a module configuration

13.7

Enable module scripts (<user>, <var> tags)

This section describes only the configuration options available for <user> tag. Refer to 14
page 267 for a full description of module scripts.

13.7.1<user> example<user logging="userlog" ><var name="VARENV" value="V1" /></user>See also the mirror module example in 15.1 page 274.

13.7.2<user> syntax<user[nicestoptimeout="300"][forcestoptimeout="300"][logging="userlog"|"none"][userlogsize="2048"]><var name="ENVIRONMENT_VARIABLE_1" value="VALUE_1" />…
</user>
The <user> tag and full subtree can be changed with adynamic configuration.

13.7.3<user>, <var> attributes<user[nicestoptimeout="300"]

Timeout delay in seconds to execute the stop_xx script.

[forcestoptimeout="300"]Timeout delay in seconds to execute the stop_xx –force
script.

Default value: 300 seconds

Default value: 300 seconds[logging="userlog"|"none"]

stdout and stderr messages of the application started inscripts.
When logging="userlog", messages are redirected into thelog
SAFEVAR/modules/AM/userlog_<year>_<month>_<day>T<time>_<script name>.ulogwhere AM is the module name
(SAFEVAR=C:\safekit\var on Windows and
SAFEVAR=/var/safekit on LINUX).
When logging="none", messages are not logged.

39 A2 38MC 03

243

SafeKit User's Guide

Default value: userlog[userlogsize="2048"]

Limit in KB of the size of the userlog
On module start, the file is truncated to 0 if the size hasreached this limit.
Default value: 2048 KB<var
name="ENV_VARIABLE_1"
value="VALUE_1" />The environment variable and its value are exported before
the execution of module scripts. Define as many var sections
as there are environment variables to export.

13.8

Virtual hostname (<vhost>, <virtualhostname> tags)

13.8.1<vhost> example<vhost><virtualhostname name="vhostname" envfile="vhostenv"/></vhost>See also the example in 15.6 page 280.

13.8.2<vhost> syntax<vhost><virtualhostname
name="virtual_hostname"envfile="path_of_a_file"[when="prim"|"second"|"both"]/></vhost>The <vhost> tag and subtree cannot be changed with adynamic configuration.

13.8.3<vhost>, <virtualhostname> attributes<vhost><virtualhostname
name="virtual_hostname"244

Definition of the virtual hostname.

39 A2 38MC 03

Userconfig.xml for a module configuration

envfile="path_of_envfile"

Path of the environment file automatically generated
by SafeKit during configuration command
If the path of the file is relative, the file will be
generated in the runtime environment of theapplication module i.e.: SAFEUSERBINThis generated environment file is used in module
scripts to set the virtual hostname before startingand stopping the application. See the module
template vhost.safe delivered with Linux andWindows package.

[when="prim"|"second"|"both"]Define when the virtual hostname must be returnedto the application instead of the physical one.
Default value: prim means when the server is
primary (PRIM or ALONE).

/></vhost>13.8.4<vhost> description

Some applications need to see the same hostname on all SafeKit servers (typically,
because it is stored in a replicated file). With the virtual hostname, these applications see
the virtual name whereas other applications see the physical name.
See 15.6 page 280 for a complete example.
On Linux
Implementation is based on the LD_PRELOAD environment variable: gethostname anduname functions are overloaded.
On Windows
Implementation is based on the CLUSTER_NETWORK_NAME_ environment variable: thequery API (GetComputerName, GetComputerNameEx, gethostname) functions take thisvariable into account. To use vhost for a service, use the command vhostservice<service> [<file>] before/after the service start/stop.

13.9

Process or service death detection (<errd>, <proc> tags)<errd> section requires <user/> section.

13.9.1<errd> example

13.9.1.1 Process monitoring
Linux and Windows, myproc is the command name of the process to monitor:<errd><proc name="myproc" atleast="1" action="restart" class="prim"/>

39 A2 38MC 03

245

SafeKit User's Guide</errd>Linux only (since SafeKit > 7.2.0.29), oracle_.* is a regular expression on thecommand name of the process to monitor:<errd><proc name="oracle" nameregex="oracle_.*" atleast="1" action="restart"class="prim"/></errd>See also the example in 15.7 page 282.
13.9.1.2 Service monitoring
myservice is the name of the Windows service (since safekit > 7.3) or Linux systemd
service (since safekit > 7.4.0.19) to monitor:<errd><proc name="myservice" service="yes" atleast="1" action="restart" class="prim" /></errd>13.9.2<errd> syntax<errd[polltimer="10"]><proc name="command name and/or resource name for the monitored process (orservice in Windows)"[service="no|yes"][nameregex=="regular expression on the command name"][argregex="regular expression on process arguments, including command
name"]
atleast="1"
action="stopstart"|"restart"|"stop"|"executable_name"class="prim"|"both|"pre"|"second"|"sec"|"othername"][start_after="nb polling cycles"][atmax="-1"]/>…
</errd>The <errd> tag and full subtree can be changed with adynamic configuration.

13.9.3<errd>, <proc> attributes<errd
polltimer="30"

Time delay, in seconds, between two polls of the list ofprocesses.
Default value: 30 seconds

<proc

246

Definition of a process to monitor. Set as many proc
sections as there are processes.

39 A2 38MC 03

Userconfig.xml for a module configuration

A resource is associated with each <proc>, it is named
proc.<value of the attribute name> (e. gproc.process_name). The resource is up when themonitoring condition is true; else down if false.
name="command_name"name is the command name of the process to monitor. It
is also the name of the resource associated with themonitored process.
At max 15 characters in Linux (the command name canbe truncated); 63 in Windows.
Example: on Linux, name="vi" and on Windows
name="notepad.exe".
Windows only. The name is automaticallyconverted to lower case.
See 13.9.4 page 250 for help on retrieving the process
command name.
Linux onlyOr
name="command_name"nameregex="regularexpression on the command
name"nameregex is a regular expression applied on thecommand name for selecting the process to monitor.
name is name of the resource associated with themonitored process.
.
As regular expressions are defined inside theXML file userconfig.xml, special charactersinterpreted by XML like '<' or '>' cannot be
used in regular expressions.
Example: set nameregex="oracle _. *"name="oracle" for monitoring oracle process that matchthe regular expression
The associated resource is proc.oracle
The nameregex attribute is optional

Or
name="service_name"service="yes"name is the name of the service to monitor. It is also thename of the resource associated with the monitored
service.
At max 63 characters.
Example:set name="W32Time" service="yes" for monitoring theWindows Time service

39 A2 38MC 03

247

SafeKit User's Guide

set name="ntpd" service="yes" for monitoring theLinux Time service (systemd ntpd.service)
The service attribute is optional, and the default value
is noclass=The process belongs to a class.

"prim"|The monitoring of a class starts only when the commandsafekit errd enable "classname" -m AM is executed.

"both"|"pre"|"second"|"sec"|"othername"

Activation/deactivation of prim, both, pre, second, andsec classes are automatically done by SafeKit in the<user/> component with start_prim/stop_prim,
start_both/stop_both, start_second/stop_second,
start_sec/stop_sec. For scripts details, see 14 page
267.
With another class name, you must explicitly
activate/deactivate process monitoring after/before thestart/stop of the process.

[argregex="regularexpression on process
arguments"]

Regular expression matching the list of arguments of theprocess to monitor, including the executable name.
Optional parameter.
The regex engine is POSIX Extended regex (see POSIX
documentation):✓in Windows, case insensitive mode✓in Linux, case sensitive mode
As regular expressions are defined inside theXML file userconfig.xml, special charactersinterpreted by XML like '<' or '>' cannot be
used in regular expressions.

See 13.9.4 page 250 for help on retrieving the list ofarguments of a process.
Linux examples with vi editor on myfile<proc name="vi" argregex=".*myfile.*" …
<proc name="vi" argregex="/myrep/myfile.*" …
<proc name="vi" argregex="/myrep/myfile" …
Windows examples with notepad editor on myfile<proc name="notepad.exe" argregex=".*myfile.*" …
<proc name="notepad.exe"
argregex="c:\\myrep\\myfile.*" …
<proc name="notepad.exe"
argregex="c:\\myrep\\myfile" …

248

39 A2 38MC 03

Userconfig.xml for a module configuration

atleast="1"

Minimum number of processes that must be running.
If this minimum is not reached, then SafeKit triggers anactionExample: name="oracle" argregex=".*db1.*"
atleast="1" means that an action will be triggered if
less than one oracle instance is running on db1.
When set to -1, this criterion is meaningless.
Default value: 1

action="restart"|"stopstart"|"stop"|"noaction"|"executable_name"

Action (or handler) to execute on the application module.
noaction means logging a message, restart triggers a
local restart and stopstart triggers a failover.
To avoid a loop on reproducible fault, a maxloop counteris incremented at each restart/stopstart command. For
the maxloop definition, see section 13.2 page 210.
To define a special handler, either set an absolute pathor a path relative to the "bin" directory of the module:
SAFE/modules/AM/bin/. We recommend a relative pathand a handler defined inside the module.
When defining a special handler, a new class name must
be associated with the monitored process.
For a special handler on Linux, on success, end with exit0
For a special handler on Windows, on success, end with
%SAFEBIN%\exitcode 0With a different value, SafeKit performs a stopstartcommand.
When running special handlers, the maxloop counter is
not incremented. To increment it:
safekit incloop –m AM –i <handler name>
This command increments the counter and returns 1
when the limit has been reached.
Default value: stopstart

start_after=[nb polling
cycles]Without the start_after attribute the monitoring ofprocesses is immediately effective.
Otherwise, it is delayed for (n-1)*polltimer (inseconds) where:n is the value given in start_after parameter
polltimer is the value set on the errd flag (30seconds by default)
For example, if start_after="3", the server is delayedfor 60 seconds ((3-1)*30).

39 A2 38MC 03

249

SafeKit User's Guide

The start_after parameter is useful if the process takes acertain time to start.
Default value: 0
Advanced parametersatmax="-1"

Maximum number of processes that can run.
If this maximum is reached, then SafeKit triggers anaction.
atmax="-1" means that this criterion is meaningless.
With atmax="0", an action is triggered each time theprocess is started.
Default value: -1 this criterion is meaningless</errd>13.9.4<errd> commands
If the command is used inside a module script, then the SAFEMODULE
environment variable is set and the -m AM parameter is not necessary

safekit –r errdpoll_running This command prints into the file
SAFEVAR/errdpoll_reserrd (SAFEVAR=/var/safekit
on Linux and SAFEVAR=c:\safekit\var on Windows if
c: is the installation drive), one line for each running
process with following fields:<pid> <command name> <command full name andarguments list> (parent=<parent pid>)
In Windows, the command name is displayed in lower
case.
Useful to find the process name and its arguments foran <errd> configuration
safekit errd disable"classname" –m AM

Suspends the monitoring of the processes included in
the class classname (for the application module AM).
Must be explicitly done in stop_... scripts before
stopping the application, for processes in class different
from prim, both, second, sec.

safekit errd enable"classname" –m AM

Resumes the monitoring of the processes defined withthe class classname (for the application module AM).
Must be explicitly done in start_... scripts after starting
the application, for processes in class different fromprim, both, second, sec.

250

39 A2 38MC 03

Userconfig.xml for a module configuration

safekit errd suspend –m AM

Suspends the monitoring of all processes except
SafeKit processes (for the application module AM).
Useful when stopping manually the application without
triggering error detection.

safekit errd resume –m AM

Resumes the monitoring of processes suspended withsafekit errd suspend (for the application moduleAM).

safekit errd list –m AM

Lists all processes monitored by SafeKit (including
SafeKit processes) and defined in the application
module AM.
The list displayed may be truncated due to internal
limits. The full list can be found in the file
SAFEVAR/modules/AM/errdlist.
SAFEVAR=/var/safekit on Linux and
SAFEVAR=c:\safekit\var on Windows if c: is theinstallation drive.

safekit kill–name="process_name"[–argregex="…"]–level="kill_level"<errd> component must run.
level="test": only display the process list
level="terminate": kill processes
level="9": send SIGKILL signal to processes (Linux
only)
level="15": send SIGTERM signal to processes (Linux
only)
Windows examples ("class CatlRegExp" for more
information):safekit kill –name="notepad.exe"–argregex=".*myfile.*" –level="terminate"safekit kill –name="notepad.exe"–argregex="c:\\myrep\\myfile.*"–level="terminate"Linux examples ("man regex" for more information) :safekit kill –name="vi"–argregex=".*myfile.*" –level="9"safekit kill –name="vi"–argregex="/myrep/myfile.*"–level="9"13.10

Checkers (<check> tag)

SafeKit brings built-in checkers with failover rules (for default failover rules details, see
13.18.5 page 265). The checkers are:13.11 “TCP checker (<tcp> tags)” page 253

39 A2 38MC 03251

SafeKit User's Guide

13.12 “Ping checker (<ping> tags)” page 254
13.13 “Interface checker (<intf> tags)” page 25613.14 “IP checker (<ip> tags)” page 257
13.15 “Custom checker (<custom> tags)” page 258
13.16 “Module checker (<module> tags)” page 260
13.17 “Splitbrain checker (<splitbrain> tag)” page 262

13.10.1 <check> example
All built-in checkers are configured under a single <check> section:<check><!-- Insert below <tcp> <ping> <intf> <ip> <custom> <module> <splitbrain> tags--></check>

13.10.2 <check> syntax<check><tcp …><to …/></tcp>…
<ping …>
<to …/></ping>…
<intf …>
<to …/></intf>…
<ip …><to …/></ip>…
<custom …/>…
<module …>[<to …/>]</module>…
<splitbrain …/></check>
The <check> tag and full subtree can be changed with a dynamic
configuration.

252

39 A2 38MC 03

Userconfig.xml for a module configuration

13.11

TCP checker (<tcp> tags)
By default, a <tcp> checker makes a local restart of the application
when the checked tcp service is down.

13.11.1 <tcp> example<check><tcp ident="R1test" when="prim" ><to addr="R1" port="80"/></tcp></check>
Insert the <tcp> tag into the <check> section if this one is already defined.

See also example in 15.8 page 284.

13.11.2 <tcp> syntax<tcp
ident="tcp_checker_name"when="prim|second|both|pre"><toaddr="IP_address" or "name_to_check"
port="TCP_port_to_check"[interval="10"][timeout="5"]/></tcp>

13.11.3 <tcp> attributes<tcp

Set as many <tcp> sections as there are TCP checkers.

ident="tcp_checker_name"

TCP checker name.

when="prim|second|both"

Use this value for a TCP checker related to theapplication.
The when value sets the checker start and stop schedule
respectively after and before the application’s eponym
start and stop scripts (start_prim/stop_prim,
start_second/stop_second, start_both/stop_both).
Action in case of failure: safekit restart of theapplication module. For default failover rules detail, see
13.18.5 page 265.
At each restart, the maxloop counter is incremented. For
its definition, see 13.2.3 page 211.

39 A2 38MC 03

253

SafeKit User's Guide

when="pre"Use this value for a TCP checker not related to theapplication.
The checker is started/stopped after/before module
scripts prestart/poststop.
You must add a special failover rule for this "tcp"checker. Typically:
external_tcp_service: if (tcp.tcp_checker_name== down) then wait();This rule executes a stopwait and puts the application
module in the WAIT state while the external TCP service
is not responding. See 13.18 page 263 for more
information.
At each stopwait, the maxloop counter is incremented(see 13.2.3 page 211 for its definition).

<toaddr="IP_@" or "name"

IP address or name to check (ex.: 127.0.0.1 for a local
service).
IPv4 or IPv6 address.

port="value"

TCP port to check.

[interval="10"]Interval in seconds between two connections trials.
Default value: 10 seconds[timeout="5"]

Connection establishment timeout in seconds.
Default value: 5 seconds</tcp>

13.12

Ping checker (<ping> tags)

By default, a <ping> checker stops the module and waits for the ping
to be up.

13.12.1 <ping> example<check><ping ident="testR2" ><to addr="R2"/></ping></check>

254

39 A2 38MC 03

Userconfig.xml for a module configuration

Insert the <ping> tag into the <check> section if this one is already defined.

See also the example in 15.9 page 284.

13.12.2 <ping> syntax<ping
ident="ping_checker_name"[when="pre"]><toaddr="IP_address" or "name_to_check"[interval="10"][timeout="5"]/></ping>13.12.3 <ping> attributes<ping

Set as many ping sections as there are ping checkers.

ident="ping_checker_name"

Ping checker name as displayed in the commandsafekit state -v –m AM. Name of checkers must be
unique.

[when="pre"]Default if not set.
Started/stopped after/before module scripts
prestart/poststop.
Executes a stopwait and puts the application module in
the WAIT state if there is no reply to the ICMP pingrequests (see default failover rules definition in 13.18.5
page 265).
At each stopwait, the maxloop counter is incremented(see 13.2.3 page 211 for its definition).

<toaddr="IP_@ or name"

External IP address or name to check.
IPv4 or IPv6 address.

[interval="10"]Interval in seconds between two ping requests.
Default value: 10 seconds[timeout="5"]

Reply timeout in seconds to the ping.
Default value: 5 seconds</ping>

39 A2 38MC 03

255

SafeKit User's Guide

13.13

Interface checker (<intf> tags)
By default, a <intf> checker stops the module and waits for the
network interface to come back up.

13.13.1 <intf> example<check><intf ident="test_eth0"><to local_addr="192.168.1.10"/></intf></check>
Insert the <intf> tag into the <check> section if this one is already defined.

See also the example in 15.10 page 285.

13.13.2 <intf> syntax<intf
ident="intf_checker_name"[when="pre"]><tolocal_addr="interface_physical_IP_address"/></intf>13.13.3 <intf> attributes<intf<intf> sections are automatically generated
on network interface when <interface
check="on"> is set (see the virtual IP
definition in 13.5 page 217).

ident="intf_checker_name"Interface checker name[when="pre"]Default.
Started/stopped after/before module scripts
prestart/poststop.
Execute a stopwait and put the application module in
the WAIT state if intf is "down" (see the default
failover rules in 13.18.5 page 265).
At each stopwait, the maxloop counter is incremented(see 13.2.3 page 211 for its definition).

<to local_addr="IP_@ />Physical IP address configured on the network interface
to check.
IPv4 or IPv6 address.

256

39 A2 38MC 03

Userconfig.xml for a module configuration</intf>

13.14

IP checker (<ip> tags)

In LINUX and Windows, this checker checks that the IP address is locally defined; inWindows it also detects IP conflicts.
By default, a <ip> checker makes a local stopstart of the module
when the checked ip address is down.

13.14.1 <ip> example<check><ip ident="ip_check" ><to addr="192.168.1.10" /></ip></check>
Insert the <ip> tag into the <check> section if this one is already defined.

See also the example in 15.11 page 286.

13.14.2 <ip> syntax<ip
ident="ip_checker_name"[when="prim"]><toaddr="IP_address" or "name_to_check"[interval="10"]
/>
</ip>

13.14.3 <ip> attributes<ip

Set as many ip sections as there are ip checkers.

ident="ip_checker_name"ip checker name as displayed in the safekit state -v–m AM command. Name of checkers must be unique.

[when="prim"]

Default if not set.
The checker is started/stopped after/before the module
scripts start_prim/stop_prim.
Action in case of failure: safekit stopstart of theapplication module (see the default failover rules in
13.18.5 page 265).
At each stopstart, the maxloop counter is incremented(see 13.2.3 page 211 for its definition).

39 A2 38MC 03

257

SafeKit User's Guide<toaddr="IP_@ or name"Local IP address or name to check.
IPv4 or IPv6 address.

[interval="10"]Interval in seconds between two checks.
Default value: 10 seconds

</ip>

13.15

Custom checker (<custom> tags)

A custom checker is a program (script or other) that you develop for your module. It is aloop performing a test at an appropriate periodicity. According to the result of the test,
the program sets the state of a resource ("up" or "down"). Then a special failover ruledecides which action must be taken when the resource is down.

13.15.1 <custom> example<check><custom ident="AppChecker" when="prim" exec="mychecker" action="restart"/></check>
Insert the <custom> tag into the <check> section if this one is alreadydefined.

See the example in 15.12 page 286.

13.15.2 <custom> syntax<custom
ident="custom_checker_name"when="pre|prim|second|both"
exec="executable_path"arg="executable_arguments"
action="wait|stop|stopstart|restart"/>

13.15.3 <custom> attributes<custom

Set as many custom sections as there arecustom checkers.

ident="custom_checker_name"

Custom checker name (network IP address).
A custom checker must set its associatedresource state itself, using the commandsafekit set –r
custom.custom_checker_name –v up|down.

258

39 A2 38MC 03

Userconfig.xml for a module configuration

when="pre"The checker is started/stopped after/before
module scripts prestart/poststop.
You must also set the action attribute toaction="wait".
This executes a stopwait and puts theapplication module in the WAIT state while theresource is down.
Note that SafeKit automatically initializes thestate of the associated resource to init, andthe failover machine stays in the WAIT state if
the state of the custom checker is notevaluated to up. For more information on the
failover machine, see 13.18 page 263.
At each stopwait, the maxloop counter isincremented (see 13.2.3 page 211 for its
definition).

In SafeKit < 8, the action was
configured by adding a failover rule
into <failover> tag. For instance:
wait_custom_checker: if(custom.custom_checker_name ==down) then wait();when="prim"|"second"|"both"

The checker is started/stopped after/before
module scripts start_prim/stop_prim,
start_second/stop_second,
start_both/stop_both.
You must also set the action attribute toaction="stop|stopstart|restart".
At each error detection, the maxloop counter isincremented (see 13.2.3 page 211 for its
definition).
In SafeKit < 8, the action was
configured by adding a failover rule
into <failover> tag. For instance:
restart_custom_checker: if(custom.custom_checker_name ==down) then restart();39 A2 38MC 03259

SafeKit User's Guide

exec="executable_path"

Defines the executable path of the custom
checker.
Can be a binary executable or a script file.
When the path of executable_path is relative,
it is relative to SAFEUSERBIN. In this case, put
your executable file in SAFE/modules/AM/bin/of your application module and use a relative
path. See 10.1 page 155 for more information
on path values.
We recommend a relative path and an
executable inside the module.
In Windows, the executable can be a binary ora ps1, vbs or cmd script
In Linux, the executable can be a binary or ashell script

arg="executable_arguments"

Defines the executable arguments when thecustom checker is started.

action="wait|stop|stopstart|restart"Generates the failover rule associated with theresource that will perform the specified action
if the resource is down. The failover rule isnamed : c_<ident value>.

13.16

Module checker (<module> tags)

The module checker checks the availability of another module. It is started/stopped in
the prestart /poststop phase before the start of the application. When the module
checker detects that the external module is down, SafeKit executes a stopwait and puts
the server in the WAIT state until the external module is detected as up by the module
checker. The module checker also triggers a stopstart when it detects that the external
module is stopping or has been restarted (either by a SafeKit stopstart, restart or
failover). See 13.18.5 page 265 for the default failover rules.
At each stopwait or stopstart, the maxloop counter is incremented (see 13.2.3 page211 for its definition).
The module checker connects to the SafeKit web service on the node running the module
to get the module state (see 10.6 page 166 for details on the web service).

13.16.1 <module> example
Example for the default configuration of the SafeKit web service (protocol: HTTP, port:
9010):<check><module name="mirror"><to addr="M1host" port="9010"/></module></check>

260

39 A2 38MC 03

Userconfig.xml for a module configuration

Example for the secured configuration of the SafeKit web service (protocol: HTTPS, port:
9453):<check><module name="mirror"><to addr="M1host" port="9453" secure="on"/></module></check>
Insert the <module> tag into the <check> section if this one is alreadydefined.
For examples, see 15.3 page 277 and 15.13 page 288.

13.16.2 <module> syntax<module[ident="module_checker_name"]
name="external_module_name">[<toaddr=" IP_@ or name the Safekit server running the external module"
port="port of the SafeKit httpd server"[interval="10"][timeout="5"][secure="on"|"off"]/>]</module>

13.16.3 <module> attributes<modulename="external_module_name"][ident="module_checker_name"][<to

Set as many <module> sections as there are module
checkers.
Name of the module checker.
Name of the external SafeKit module to check.
Default: external_module_name_<IP_@ or nameof the server>
Definition of the server(s) running the external
module to check.
Default is the local server.

addr="IP_@ or name of theserver"IP address or name of the external module.

port="port of the SafeKit
web service"

Port of the SafeKit web service.

[interval="10"]

IPv4 or IPv6 address.

9010 for HTTP ; 9453 for HTTPSInterval in seconds between two checks.
Default value: 10 seconds.

39 A2 38MC 03

261

SafeKit User's Guide[timeout="5"]

Check reply timeout in seconds.
Default value: 5 seconds[secure="on"|"off"]

Use HTTP protocol (secure="off") or HTTPS(secure="on")Default value: off/>]</module>

13.17

Splitbrain checker (<splitbrain> tag)

SafeKit provides a splitbrain checker that is suits mirror architectures. Split brain is asituation where, due to temporary failure of all network links between SafeKit nodes, andpossibly due to software or human error, both nodes switched to the primary role while
isolated. This is a potentially harmful state, as it implies that the application is running on
both nodes. Moreover, when file replication is enabled, modifications to the data aremade on the two nodes.
The split-brain checker detects the loss of all connectivity between nodes and selects only
one node to become the primary. The other node is not up-to-date anymore and goes
into the WAIT state until:
the heartbeat becomes available againorthe administrator runs safekit commands to force the start as primary (safekit
stop then safekit prim).
The primary node election is based on the ping of an IP address, called the witness. The
network topology must be designed so that only one node can ping the witness in case of
split brain. If this is not the case, both nodes will go primary.
•

Ping between nodes and witness must be enabled•

Since SafeKit 8.2.1, multiple witnesses can be defined. This makes it
possible to tolerate the failure of one witness, at least one of whichmust be accessible

13.17.1 <splitbrain> example<check><splitbrain ident="SBtest" exec="ping" arg="192.168.1.100 192.168.2.120"/></check>
Insert the <splitbrain> tag into the <check> section if this one is alreadydefined.

262

39 A2 38MC 03

Userconfig.xml for a module configuration

13.17.2 <splitbrain> syntax<splitbrain
ident="witness"
exec="ping"arg="witness1_IP_name/>witness2_IP_name"13.17.3 <splitbrain> attributes<splitbrain

Set only one splitbrain checker.

ident="witness"

Name displayed in the safekit state -v –m AM command for thewitness state. It represents the state of the witness(es).
The resource is assigned to :- up, if at least one witness responds- down, if not all witnesses respond[when="pre"]Fixed value.
Started/stopped after/before module scripts prestart/poststop.
The witness state is stored in splitbrain.witness. It can bedisplayed using the safekit state -v –m AM command.
On splitbrain detection, the server with splitbrain.witness=”up”
goes primary; the other one with splitbrain.witness=”down” sets
the resource splitbrain.uptodate to down and goes into the WAIT
state (for default failover rules, see 13.18.5 page 265).
At each stopwait, the maxloop counter is incremented (see 13.2.3
page 211 for its definition).

exec="ping"

Fixed value.
Use a pinger to ping the witness and set splitbrain.witness state.

arg="witness1_IP_namewitness2_IP_name"

List of IP addresses or witness names to ping.
IPv4 or IPv6 address.
Multiple witness definition supported since SafeKit 8.2.1.

</splitbrain>

13.18

Failover machine (<failover> tag)

SafeKit comes with checkers (network interface, ping, TCP, custom, module checkers)
which regularly (by default every 10 seconds) check resources and set the state to up or
down (see 13.10 page 251 for checkers definition). The failover machine regularly (bydefault every 5 seconds) evaluates the global state of all resources and triggers a failover
according to failover rules programmed in a simple language.
In farm architecture, the failover machine can work only on the states of local resources
whereas in mirror architecture, the failover machine can work on the states of local and
39 A2 38MC 03

263

SafeKit User's Guide

remote resources. As the states of resources are exchanged on heartbeat channels, it isbetter to have several heartbeat channels (see 13.3 page 213 for heartbeats definition).

13.18.1 <failover> example<failover><![CDATA[
ping_failure: if (ping.testR2 == down) then stopstart();]]></failover>

13.18.2 <failover> syntax<failover [extends="yes"] [period="5000"] [handle_time="15000"]><![CDATA[
label: if (expression) then action;…
]]></failover>
The <failover> tag and subtree cannot be changed with a dynamic
configuration.

13.18.3 <failover> attributes<failover[extends="yes"|"no"]If set to yes, the new failover rules extend the default failover
rules (see 13.18.5 page 265 for its definition).
If set to "no", the new failover rules overwrite the default one(avoid this configuration).
Default value: yes.

[period="5000"]

Period in milliseconds between two evaluations of failover
rules.
Default value: 5000 milliseconds (5 seconds)[handle_time="15000"] A failover action must be stable (the same) at least during thetime handle_time (in milliseconds) before being applied by
the failover machine.
Default value: 15000 milliseconds (15 seconds).
handle_time must be a multiple of the period value.

13.18.4 <failover> commands
safekit set [–m AM] -r
This command sets the state of one resource:
resource_class.resource_id
Examples:-v resource_statesafekit set -r custom.myresource -v up
safekit set -r custom.myresource -v down

264

39 A2 38MC 03

Userconfig.xml for a module configuration[-n] [-l]Each assignment of the main resources is stored in a log
to keep track of their status. Use -n to disable thislogging or -l to force it.

safekit stopwait -i"identity"

Equivalent to wait() command of the failover machine(see 13.18 page 263).
With stopwait, (1) poststop and prestart scripts are
not executed and (2) checkers when="pre" are notstopped.

The other commands (restart(), stopstart(), stop(), swap()) of the failover
machine are equivalent to control commands (with the -i identity parameter)
described in 9.4 page 146.
maxloop / loop_interval / automatic_reboot are applied if -i
identity is passed to commands (for these attributes details, see 13.2 page210). This is the case when called from the failover machine.

13.18.5 Failover rules
The default failover rules for the SafeKit checkers are:<failover><![CDATA[/* rule for module checkers */module_failure: if (module.? == down) then wait();/* rule for interface checkers */interface_failure: if (intf.? == down) then wait();/* rule for ping checkers */ping_failure: if (ping.? == down) then wait();/* rule for tcp checkers */tcp_failure: if (tcp.? == down) then restart();/* rule for ip checkers */ip_failure: if (ip.? == down) then stopstart();/* rules for splitbrain */splitbrain_failure: if (splitbrain.uptodate == down) then wait();]]></failover>
They are defined into SAFE/private/conf/include/failover.xml.
There are also failover rules dedicated to file replication management.
The WAKEUP command is automatically generated when no wait() rule applies.

Since SafeKit 7.5, default failover rules are using a new syntax, and rules forthe rfs component are set into the file SAFE/private/conf/include/rfs.xml.

In addition to the default rules, the user can define his own rules (for a custom checker
for example) using the following syntax:
label: if ( expression ) then action;
39 A2 38MC 03

265

SafeKit User's Guidewith:label ::= string
action ::= stop() | stopstart() | wait() | restart() | swap()
expression ::= ( expression )| ! expression| expression && expression| expression || expression| expression == expression| expression != expression| resource ::= [local. | remote.] 0/1resource_class.resource_id| resource_state
The syntax to design the resources is as follows:
resource ::= [local. | remote.] 0/1resource_class.resource_id(default: local)
resource_class ::= ping | intf | tcp | custom | module | heartbeat | rfs
resource_id ::= * | ? | nameresource_state ::= init | down | up | unknown
init

Special initialization state of a resource when the checker is notstarted.
If a resource in the init state is used in a failover rule, SafeKit does
evaluate the rule.

up

Resource OK.

down

Resource KO.

unknown

Special state of a remote resource; the remote state is unknown at
the test time (ex.: when the remote module is stopped).

266

39 A2 38MC 03

14.Scripts for a module configuration
14.1 “List of scripts” page 267
14.2 “Script execution automaton” page 269
14.3 “Variables and arguments passed to scripts” page 270
14.4 “SafeKit special commands for scripts” page 270
To enable scripts call, <user> tag must be defined in userconfig.xml as described in
13.7 page 243. This tag could be added or removed dynamically.
Scripts must executables:✓in Windows, an executable with the extension and type: .cmd, .vbs, .ps1,.bator .exe✓in Linux, any type of executable

Each time you update scripts, you must apply the module configuration onto the servers(with the SafeKit console or command).
Examples of scripts are given in 15.1 page 274 for a mirror module, and in 15.2 page275 for a farm module.
During the configuration phase, scripts are copied from SAFE/modules/AM/binin the execution environment directory SAFE/private/modules/AM/bin(=SAFEUSERBIN, do not touch scripts at this place) where AM is the module
name.

14.1

List of scripts

Below the list of scripts that can be defined by the user. The essential scripts start/stop
are those that start and stop the application within the module.

14.1.1

Start/stop scripts

start_prim
stop_prim

Scripts for a mirror module.

start_bothstop_both

Scripts for a farm module.

To start & stop application on the ALONE or PRIM server

To start & stop application on all UP servers in a farm cluster
In the special case they are defined in a mirror module, they are also
executed on both servers (PRIM, SECOND or ALONE)

start_second
stop_second

Special scripts for a mirror module
To start & stop application on the "SECOND" server
When the secondary server becomes the primary one,stop_second followed by start_prim is executed

39 A2 38MC 03

267

SafeKit User's Guide

start_sec
stop_sec

Special scripts for a mirror module

stop_[both,prim,
second,sec] force

Scripts for all modules

prestart
poststop

Scripts for all modules

The stop scripts are called twice: once for a graceful shutdown of theapplication (without force as first argument), a second time with aforce parameter for a rapid shutdown (with force as first argument).

Executed at the very beginning of the module start and at its end.
By default, prestart contains stop_sec, stop_second, stop_prim,
stop_both to stop application before starting the module under thecontrol of SafeKit.

transition

Script for all modules
This script is executed on state transitions described in 14.2 page269

14.1.2Other scripts

configconfig is called when executing the safekit config –m AM commandon the application module. You can make a special application
configuration in this script.

deconfig

deconfig is called when executing the safekit deconfig –m AM
command, which is itself called at the application module uninstallation.
You can remove a special application configuration made previously inthe config script.

confcheckconfcheck is called when executing the safekit confcheck –m AM
command on the application module. You can add in this script sometests for checking changes on the application configuration files.

state

state is called when executing the safekit state –v –m AM commandon the application module. You can display a special state of theapplication.

level

level is called when executing the safekit level –m AM command on
the application module. You can display the application version.

268

39 A2 38MC 03

Scripts for a module configuration

14.2

Script execution automaton
Example: first transition from STOP to WAIT calls the script transition STOPWAIT start is called.
Most of the time, stop scripts are called twice (without the force parameterand then with the force parameter). In that case the script name is written initalic.

39 A2 38MC 03

269

SafeKit User's Guide

14.3

Variables and arguments passed to scripts

All scripts are called with 3 parameters:✓

the current state (STOP,WAIT,ALONE,PRIM,SECOND,UP),✓

the next state (STOP,WAIT,ALONE,PRIM,SECOND,UP)✓

the action (start, stop, stopstart or stopwait).

The stop scripts are called twice:✓a first time for a graceful shutdown of the application✓a second time with a force parameter for a forced shutdown (with force as firstargument)

The environment variables that can be used inside scripts are:✓

SAFE, SAFEMODULE, SAFEBIN, SAFEUSERBIN, SAFEVAR, SAFEUSERVAR (fordetails, see 10.1 page 155)✓

all variables defined in <user> tag of userconfig.xml (see 13.7 page 243).

14.4

SafeKit special commands for scripts

Special commands are installed under SAFE/private/bin. Special commands can becalled directly in module scripts with %SAFEBIN%\specialcommand or$SAFEBIN/specialcommand. Outside module scripts, use safekit -r command.

safekit -r<special command>[<args>]<special command> <args> executed within the SafeKit
environment. When the command name is not an absolute path,the command is searched in SAFEBIN=SAFE/private/bin
directory.
If you use special commands outside SafeKit scripts,
prefix them with safekit -r specialcommand

14.4.1

Commands for Windows

14.4.1.1 sleep, exitcode, sync commands
On Windows, you can use the following basic commands:%SAFEBIN%\sleep.exe <timeout value in seconds>
To be used inside stop scripts because net stop service is not synchronous%SAFEBIN%\exitcode.exe <exit value>
To return an error value when the script exits%SAFEBIN%\sync.exe \\.\<drive letter:>
To sync file system cache of a disk

270

39 A2 38MC 03

Scripts for a module configuration

14.4.1.2 namealias command%SAFEBIN%/namealias [-n | -s ] <alias name>–n to add a new NetBIOS name (set into start_prim) or -s to suppress the NetBIOS
name (set into stop_prim)
You can also use the SafeKit command netnames (or the windows command nbtstat) tolist NetBIOS information.

14.4.2

Commands for Linux

14.4.2.1 Managing the crontab$SAFEBIN/gencron[del | add]

del to disable the entries in stop_prim (by inserting
comments)
or
add to enable the entries in start_prim (by removing
comments).

<user name>

User name in the crontab.

[all |<command name>]

all: to apply on all entries
orto apply on the name of the command-c "<comment>"

Header of the comment that will be inserted.

For example, to disable/enable the entry from the admin's crontab,
5 0 * * * $HOME/bin/daily.job >> $HOME/tmp/out 2>&1
Insert into stop_prim:$SAFEBIN/gencron del admin daily.job -c "SafeKit configuration for$SAFEMODULE"
And insert into start_prim:$SAFEBIN/gencron add admin daily.job -c "SafeKit configuration for$SAFEMODULE"
14.4.2.2 Bounding command$SAFEBIN/boundcmd <timeout value> <command path> [<args>]
Bound a command with a timeoutboundcmd returns the exit code of the command when the command terminates before
the timeout; otherwise, it exits with the value 2.
For example, to flush data on disk with a timeout of 30 seconds, run:
$SAFEBIN/boundcmd 30 /bin/sync 1>/dev/null 2>&1

39 A2 38MC 03

271

SafeKit User's Guide

14.4.2.3 Commands for Windows and Linux
safekit –r
List running processes as a tree (except for all) and optional killprocesstree list |safekit –r processtree list all
kill …
List all running processes.
safekit –r processtree list <process command name>
List all running processes with the specified command name.
safekit –r processtree kill <process command name>
List and kill all running processes with the specified command
name.
safekit –r processtree list | kill <process command
name>| all <regular expression on the full command –path and arguments>
List (and kill) all running process with the specified command
name and arguments.
Windows examples ("class CatlRegExp" for more information):safekit –r processtree kill notepad.exe ".*myfile.*"safekit –r processtree list all “mirror”
Linux examples ("man regex" for more information) :safekit –r processtree kill vi ".*myfile.*"safekit –r processtree list all “mirror”
safekit incloop-m AM –i <handler
name>

SafeKit provides a maxloop counter, the number of restart andstopstart of the module on error detection. The module isstopped when this counter reaches the maxloop value over the
loop_interval period.
When running special handlers, the maxloop counter is notincremented. To increment it, use the command:safekit incloop –m AM –i <handler name>
It increments the maxloop counter for the module AM and returns
1 when the limit has been reached.

safekit resetloop

Reset the maxloop counter to the value 0-m AM [–i <handler
name>]
safekit checkloop-m AM

For checking the maxloop counter for the module AM, use thecommand: safekit checkloop –m AM
It returns 0 when the maxloop counter is not reached or
the last increment occurred outside loop_interval
It returns 1 when the maxloop counter is reached andthe last increment occurred during loop_interval

272

39 A2 38MC 03

15.Examples of userconfig.xml and module scripts
15.1 “Generic mirror module example with mirror.safe” page 274
15.2 “Generic farm module example with farm.safe” page 275
15.3 “A Farm module depending on a mirror module example” page 277
15.4 “Dedicated replication network example” page 278
15.5 “Network load balancing examples in a farm module” page 278
15.6 “Virtual hostname example with vhost.safe” page 280
15.7 “Software error detection example with softerrd.safe” page 282
15.8 “TCP checker example” page 284
15.9 “Ping checker example” page 284
15.10 “Interface checker example” page 285
15.11 “IP checker example” page 286
15.12 “Custom checker example with customchecker.safe” page 286
15.13 “Module checker example with leader.safe and follower.safe” page 288
15.14 “Mail notification example with notification.safe” page 289

Some examples are taken from the modules delivered with the SafeKit package, under
SAFE/Application_Modules. You can install them with the web console (see 3.3.1 page44) to examine the configuration file and module scripts in detail.
Other examples of integration are described underhttps://www.evidian.com/products/high-availability-software-for-applicationclustering/cluster-configuration/.
The .safe are platform dependent and therefore different in Windows andLinux.

In the following, the examples use this global cluster configuration:<cluster><lans><lan name="net3"><node name="node1" addr="10.1.0.2"/><node name="node2" addr="10.1.0.3"/><node name="node3" addr="10.1.0.3"/></lan><lan name="default"><node name="node1" addr="192.168.1.1"/><node name="node2" addr="192.168.1.2"/></lan><lan name="repli"><node name="node1" addr="10.0.0.2"/><node name="node2" addr="10.0.0.3"/></lan></lans>39 A2 38MC 03

273

SafeKit User's Guide</cluster>

15.1

Generic mirror module example with mirror.safe

Below is the configuration file and module scripts of the generic mirror module,
mirror.safe, in Windows. For Linux, please refer to the mirror.safe delivered with theLinux package.
conf/serconfig.xml - see 13 page 209<!-- Mirror Architecture with Real Time File Replication and Failover --><!DOCTYPE safe><safe><service mode="mirror" defaultprim="alone" maxloop="3" loop_interval="24"
failover="on"><heart pulse="700" timeout="30000"><heartbeat name=”default” ident=”flow”/></heart><rfs async="second" acl="off" locktimeout="100" nbrei="3" iotimeout="300"><replicated dir="c:\test1replicated" mode="read_only"/><replicated dir="c:\test2replicated" mode="read_only"/></rfs><vip><interface_list><interface check="on" arpreroute="on"><real_interface><virtual_addr addr="192.168.4.10" where="one_side_alias"/></real_interface></interface></interface_list></vip><user nicestoptimeout="300" forcestoptimeout="300" logging="userlog"/></service></safe>bin/start_prim.cmd - see 14 page 267@echo off
rem Script called on the primary server for starting application services
rem For logging into SafeKit log use:rem "%SAFE%\safekit" printi | printe "message"
rem stdout goes into Scripts log
echo "Running start_prim %*"set res=0rem Fill with your services start call
rem net start "myservice" /Y
set res=%errorlevel%
if %res% == 0 goto end:stop"%SAFE%\safekit" printe "start_prim failed"
rem uncomment to stop SafeKit when criticalrem "%SAFE%\safekit" stop -i "start_prim"

274

39 A2 38MC 03

Examples of userconfig.xml and module scripts:endbin/stop_prim.cmd - see 14 page 267@echo off
rem Script called on the primary server for stopping application services
rem For logging into SafeKit log use:rem "%SAFE%\safekit" printi | printe "message"
rem ---------------------------------------------------------rem
rem 2 stop modes:
remrem - graceful stopremcall standard application stop with net stoprem
rem - force stop (%1=force)remkill application's processesrem
rem ---------------------------------------------------------rem stdout goes into Scripts log
echo "Running stop_prim %*"set res=0
rem default: no action on forcestop
if "%1" == "force" goto end
rem Fill with your service(s) stop call
rem net stop "myservice" /Yrem If necessary, uncomment to wait for the stop of the services
rem "%SAFEBIN%\sleep" 10
if %res% == 0 goto end"%SAFE%\safekit" printe "stop_prim failed"
:end

15.2

Generic farm module example with farm.safe

Below is the configuration file and module scripts for the generic farm module,
farm.safe, in Windows. For Linux, please refer to the farm.safe delivered with theLinux package.
conf/userconfig.xml - see 13 page 209<!-- Farm Architecture with Load-Balancing and Failover --><!DOCTYPE safe><safe><service mode="farm" maxloop="3" loop_interval="24"><!-- Cluster Configuration --><!-- Set nodes on your network --><farm><lan name=”default” /><lan name =”net3” /></farm>39 A2 38MC 03275

SafeKit User's Guide<vip><interface_list><interface check="on" arpreroute=”on”><virtual_interface type="vmac_directed"><virtual_addr addr="192.168.4.20" where="alias"/></virtual_interface></interface></interface_list><loadbalancing_list><group name="FarmProto"><!-- Set load-balancing rule --><rule port="9010" proto="tcp" filter="on_port"/></group></loadbalancing_list></vip><user nicestoptimeout="300" forcestoptimeout="300" logging="userlog"/></service></safe>bin/start_both.cmd - see 14 page 267@echo off
rem Script called on all servers for starting applications
rem For logging into SafeKit log use:rem "%SAFE%\safekit" printi | printe "message"
rem stdout goes into Scripts log
echo "Running start_both %*"set res=0rem Fill with your services start call
rem net start "myservice" /Y
set res=%errorlevel%
if %res% == 0 goto end:stop
set res=%errorlevel%"%SAFE%\safekit" printe "start_both failed"
rem uncomment to stop SafeKit when criticalrem "%SAFE%\safekit" stop -i "start_both":end
bin/stop_both.cmd - see 14 page 267@echo off
rem Script called on all servers for stopping application
rem For logging into SafeKit log use:rem "%SAFE%\safekit" printi | printe "message"

276

39 A2 38MC 03

Examples of userconfig.xml and module scripts
rem ---------------------------------------------------------rem
rem 2 stop modes:
remrem - graceful stopremcall standard application stop with net stoprem
rem - force stop (%1=force)remkill application's processesrem
rem ---------------------------------------------------------rem stdout goes into Scripts log
echo "Running stop_both %*"set res=0
rem default: no action on forcestop
if "%1" == "force" goto end
rem Fill with your services stop call
rem net stop "myservice" /Yrem If necessary, uncomment to wait for the stop of the services
rem "%SAFEBIN%\sleep" 10
if %res% == 0 goto end"%SAFE%\safekit" printe "stop_both failed"
:end

15.3

A Farm module depending on a mirror module example

In the example below, the farm module can only start if the mirror module is started.
This architecture can be used to link an IIS farm module to a Microsoft SQL server mirror
module. It is based on the configuration of a module checker in the farm module. For
details, see 13.16 page 260.
farm/conf/userconfig.xml - see 13 page 209…
<!-- Checker Configuration: module dependency to mirror + local TCP checker --><check><module name="mirror"><to addr="192.168.1.31" port="9010"/></module></check>…
Note that the module dependency can be used when you deploy farm andmirror modules on the same SafeKit cluster or when you deploy farm andmirror modules on two different clusters.

39 A2 38MC 03

277

SafeKit User's Guide

15.4

Dedicated replication network example

The attribute ident="flow" on the heartbeat, allows to identify the replication flow. For
details, see 13.6 page 225.
conf/userconfig.xml - see 13 page 209…
<heart><heartbeat name=”default” /><!— 2nd heartbeat special for dedicated replicated network --><heartbeat name=”repli” ident="flow" /></heart>…

15.5

Network load balancing examples in a farm module

15.5.1

TCP load balancing example

With the following userconfig.xml configuration file, you are defining a farm of 3
servers with network load balancing and failover on TCP services 9010 (SafeKit web
service), 23 (Telnet), 80 (HTTP), 443 (HTTPS), 8080 (HTTP proxy) and 389 (LDAP).
With HTTP and HTTPS, network load balancing is set on the client IP address("on_addr") and not on the client TCP port ("on_port"), to ensure that thesame client is always on the same server over several TCP connections(stateful versus stateless servers: see 1.4 page 18)
conf/userconfig.xml - see 13 page 209<!DOCTYPE safe><safe><service mode="farm"><farm><lan name=”net3” /></farm><vip><interface_list><interface check="on" arpreroute=”on><virtual_interface type="vmac_directed"><virtual_addr addr="192.168.1.50" where="alias" /></virtual_interface></interface></interface_list><loadbalancing_list><group name="tcpservices" ><cluster><host name="node1" power="1" /><host name="node2" power="1" /><host name="node3" power="1" /></cluster><rule port="9010" proto="tcp" filter="on_port" /><rule port="23"
proto="tcp" filter="on_port" /><rule port="80"
proto="tcp" filter="on_addr" /><rule port="443" proto="tcp" filter="on_addr" /><rule port="8080" proto="tcp" filter="on_addr" />278

39 A2 38MC 03

Examples of userconfig.xml and module scripts<rule port="389"</group></loadbalancing_list></vip></service></safe>15.5.2

proto="tcp" filter="on_port" />UDP load balancing example

With the following userconfig.xml configuration file, you are defining a farm of 3
servers with network load balancing and failover on UDP services 53 (DNS), 1645
(RADIUS).
conf/userconfig.xml - see 13 page 209<!DOCTYPE safe><safe><service mode="farm"><farm><lan name=”net3” /></farm><vip><interface_list><interface check="on"><virtual_interface type="vmac_invisible"><virtual_addr addr="192.168.1.50" where="alias" /></virtual_interface></interface></interface_list><loadbalancing_list><group name="udpservices" ><cluster><host name="node1" power="1" /><host name="node2" power="1" /><host name="node3" power="1" /></cluster><rule port="53"
proto="udp" filter="on_ipid" /><rule port="1645" proto="udp" filter="on_ipid" /></group></loadbalancing_list></vip></service></safe>With "on_ipid", the load balancing is made on the IP identifier filed in thepacket IP header. The load balancing works even if the client always presents
the same client IP address and client port at input.

15.5.3

Multi-group load balancing example

With the following userconfig.xml configuration file, you are defining a farm of 3
servers with a priority for HTTP traffic on the 1st server, HTTPS on the 2nd server andproxy HTTP on the 3rd server.
conf/userconfig.xml - see 13 page 209<!DOCTYPE safe><safe><service mode="farm">

39 A2 38MC 03

279

SafeKit User's Guide<farm><lan name=”net3” /></farm><vip><interface_list><interface check="on" arpreroute=”on”><virtual_interface type="vmac_directed"><virtual_addr addr="192.168.1.50" where="alias" /></virtual_interface></interface></interface_list><loadbalancing_list><group name="http_service" ><cluster><host name="node1" power="3" /><host name="node2" power="1" /><host name="node3" power="1" /></cluster><rule port="80"
proto="tcp" filter="on_addr" /></group><group name="https_service" ><cluster><host name="node1" power="1" /><host name="node2" power="3" /><host name="node3" power="1" /></cluster><rule port="443"
proto="tcp" filter="on_addr" /></group><group name="httpproxy_service" ><cluster><host name="node1" power="1" /><host name="node2" power="1" /><host name="node3" power="3" /></cluster><rule port="8080"
proto="tcp" filter="on_addr" /></group></loadbalancing_list></vip></service></safe>15.6

Virtual hostname example with vhost.safe

The demonstration module vhost.safe shows how to set a virtual hostname (for details,
see 13.8 page 244)
conf/userconfig.xml - see 13 page 209…
<vhost><virtualhostname name="virtualname" envfile="vhostenv.cmd" /></vhost>…

280

39 A2 38MC 03

Examples of userconfig.xml and module scripts
In addition to this configuration, special commands must be executed in the module
scripts. Below is an example of Windows scripts. For Linux, please refer to thevhost.safe delivered with the Linux package.
bin/start_prim.cmd - see 14 page 267@echo off
rem Script called on the primary server for starting application services
rem For logging into SafeKit log use:rem "%SAFE%\safekit" printi | printe "message"
rem stdout goes into Scripts log
echo "Running start_prim %*"rem Set virtual hostname
CALL "%SAFEUSERBIN%\vhostenv.cmd"
rem Next commands use the virtual hostname
FOR /F %%x IN ('hostname') DO SET servername=%%xecho "hostname is "%servername%rem WARNING: previous virtual hostname setting is insufficient to change thehostname for services
rem If one service needs the virtual hostname, you need also to uncomment the rem
followingrem "%SAFE%\private\bin\vhostservice" SERVICE_TO_BE_DEFINED
set res=0rem Fill with your services start callset res=%errorlevel%
if %res% == 0 goto end:stop"%SAFE%\safekit" printe "start_prim failed"
rem uncomment to stop SafeKit when criticalrem "%SAFE%\safekit" stop -i "start_prim":end
bin/stop_prim.cmd - see 14 page 267@echo off
rem Script called on the primary server for stopping application services
rem For logging into SafeKit log use:rem "%SAFE%\safekit" printi | printe "message"
rem ---------------------------------------------------------rem
rem 2 stop modes:
remrem - graceful stopremcall standard application stop with net stoprem
rem - force stop (%1=force)remkill application's processesrem
rem ----------------------------------------------------------

39 A2 38MC 03

281

SafeKit User's Guide

rem stdout goes into Scripts log
echo "Running stop_prim %*"set res=0
rem Reset virtual hostname
CALL "%SAFEUSERBIN%\vhostenv.cmd"
rem Next commands use the real hostname
FOR /F %%x IN ('hostname') DO SET servername=%%xecho "hostname is "%servername%rem default: no action on forcestop
if "%1" == "force" goto end
rem Fill with your services stop callrem If necessary, uncomment to wait for the stop of the services
rem "%SAFEBIN%\sleep" 10
if %res% == 0 goto end"%SAFE%\safekit" printi "stop_prim failed":end
rem WARNING: if the virtual hostname was set for services in start_prim.cmd,
rem uncomment the following to restore the real hostname in last stop phase :rem "%SAFE%\private\bin\vhostservice" SERVICE_TO_BE_DEFINED

15.7

Software error detection example with softerrd.safe

The softerrd.safe module is a demonstration of the software error detection for mirror
architecture (for configuration details , see 13.9 page 245).
The module monitors the presence of:mybin and myappli started/stopped on the primary node with start_prim/stop_prim
myotherbin started/stopped on the secondary node with start_second/stop_second
Detecting the shutdown of:mybin causes the module to restart
myappli causes the execution of a special handler restart_myappli.cmd. This script
increments the maxloop counter and restarts the myappli process
myotherbin causes a stop of the module

The tests consist in killing the mybin, myotherbin or myappli processes with the safekit
kill command.
Below is an extract of softerrd.safe for Windows. For Linux, look at the one delivered
with the Linux package.
conf/userconfig.xml - see 13 page 209…
282

39 A2 38MC 03

Examples of userconfig.xml and module scripts<errd><proc name="mybin.exe" atleast="1" action="restart" class="prim"/><proc name="myotherbin.exe" atleast="1" action="stop" class="second"/><proc name="myappli.exe" atleast="1" action="restart_myappli"class="myappli"/></errd>…

bin/start_prim.cmd - see 14 page 267
Note the call to %SAFE%\safekit errd enable myappli for starting the monitoring of
the processes with class="myappli"@echo off
%SAFE%\safekit printi "start mybin"
start %SAFEUSERBIN%\mybin.exe 10000000%SAFE%\safekit printi "start myappli"
start %SAFEUSERBIN%\myappli.exe 10000000%SAFE%\safekit errd enable myappli:endbin/stop_prim.cmd - see 14 page 267
Note the call to %SAFE%\safekit errd disable myappli for stopping the monitoring of
the processes with class="myappli"@echo onrem default: no action on forcestop
if "%1" == "force" goto end%SAFE%\safekit printi "stop mybin"%SAFE%\safekit kill -level="terminate" -name="mybin.exe"%SAFE%\safekit printi "stop myappli"%SAFE%\safekit errd disable myappli%SAFE%\safekit kill -level="terminate" -name="restart_myappli.cmd"%SAFE%\safekit kill -level="terminate" -name="myappli.exe":endbin/restart_myappli.cmd
Note the increment of the loop counter and the stop of the module when maxloop isreached@echo off
rem Template for script called by errd on error detection instead of standard
restart%SAFE%\safekit printi "restart_myappli"
rem first disable monitoring of the application%SAFE%\safekit errd disable myappli

39 A2 38MC 03

283

SafeKit User's Guide

rem increment loop counter%SAFE%\safekit incloop -i "restart_myappli"
if %errorlevel% == 0 goto next
rem max loop reached%SAFE%\safekit stop -i "restart_myappli"%SAFEBIN%\exitcode 0:next
rem max loop not reached : go on restarting the application%SAFE%\safekit printi "Restart myappli"%SAFE%\safekit kill -level="terminate" -name="myappli.exe"
start %SAFEUSERBIN%\myappli.exe 10000000
rem finally, enable monitoring of the application%SAFE%\safekit errd enable myappli

15.8

TCP checker example

Below is an example of tcp checker definition that tests the Apache web service (forconfiguration details, see 13.11 page 253).
The default action when the tcp service is down is to restart locally the module (see13.18.5 page 265 for the default failover rules description).
conf/userconfig.xml - see 13 page 209…
<check><tcp
ident="Apache_80"
when="both"><toaddr="172.21.10.5"
port="80"interval="120"timeout="5"/></tcp></check>…

15.9

Ping checker example

The next example is the configuration of a ping checker that tests a router at
192.168.1.1 IP address (for configuration details, see 13.12 page 254). The default
action when the router is down is to stop locally the module and to wait for the ping to beup (see 13.18.5 page 265 for the default failover rules description).
conf/userconfig.xml - see 13 page 209…
<check ><ping ident="router"><to addr="192.168.1.1"/></ping>284

39 A2 38MC 03

Examples of userconfig.xml and module scripts</check>…

15.10

Interface checker example

Below is the example of an interface checker configuration automatically generated when<interface check="on"> is set (for configuration details, see 13.5 page 217). In theuserconfig.xml, the virtual IP address is defined as follows:
conf/userconfig.xml - see 13 page 209<vip><interface_list><interface check="on"><real_interface><virtual_addr addr="192.168.1.32" where="one_side_alias"/></real_interface></interface></interface_list></vip>The default action when the interface checker is down is to stop locally the module and to
wait for the interface to be up (see 13.18.5 page 265 for the default failover rules).
To generate the configuration of the interface checker, SafeKit computes the hardware
network interface, network and first IP address corresponding to the virtual IP address.
configuration generated in Windows<check><intf when="pre" ident="192.168.1.0"
intf="{8358A0EE-2F3F-4FEE-A33B-EDC406C0C858}"><to local_addr="192.168.1.228"/></intf></check>
Where {8358A0EE-2F3F-4FEE-A33B-EDC406C0C858} is the identity of the network
interface for the network 192.168.1.0 and with the IP address 192.168.1.228 as first
IP address (safekit –r vip_if_ctrl –L).
configuration generated in LinuxFor instance, a configuration generated on Linux is:<check><intf when="pre" ident="192.168.1.0" intf="eth2"><to local_addr="192.168.1.20"/></intf></check>
where eth2 is the identity of the network interface for the network 192.168.1.0 with
the IP address 192.168.1.20 as first IP address (all this information is get from theifconfig –a ipconfig or ip addr show command).
For configuration details, see 13.13 page 256.

39 A2 38MC 03

285

SafeKit User's Guide

15.11

IP checker example

Below is the example of an ip checker configuration automatically generated when<virtual_addr check="on" …> is set (for configuration details, see 13.5 page 217). In
the userconfig.xml, the virtual IP address is defined as follows:
conf/userconfig.xml - see 13 page 209…
<vip><interface_list><interface check="on" arpreroute="on"><real_interface><virtual_addr addr="192.168.1.99" where="one_side_alias" check="on"/></real_interface></interface></interface_list></vip>…
The default action when the ip checker is down is to stopstart locally the module (see13.18.5 page 265 for the default failover rules).
configuration generated in Windows and LinuxThe ip checker configuration generated is (for more information, see 13.14 page257):
<check><ip ident="192.168.1.99" when="prim"><to addr="192.168.1.99"/></ip></check>

15.12

Custom checker example with customchecker.safe

The customchecker.safe module is a demonstration mirror module with a custom
checker (see 13.15 page 258).
This custom checker tests the presence of a file on the primary server(when="prim"). The associated resource is called custom.checkfile(ident="checkfile"). It is set to up (file present) or down (file missing)
The associated failover rule (configured in <failover>), is named c_checkfile
and causes the module to restart if the resource is down (see 13.18.5 page 265
for failover rules). Since SafeKit 8, this failover rule is automatically generated
according to action attribute value.
This example can be used as a basis for writing your own checker.
conf/userconfig.xml for SafeKit >= 8 - see 13 page 209…
<check><custom ident="checkfile" exec="checker.ps1"
arg="c:\safekit\checkfile" when="prim" action="restart"/>286

39 A2 38MC 03

Examples of userconfig.xml and module scripts</check><user></user>…
conf/userconfig.xml for SafeKit < 8 - see 13 page 209…
<check><custom ident="checkfile" exec="checker.ps1"
arg="c:\safekit\checkfile" when="prim"/></check><user></user><failover><![CDATA[
c_checkfile:if( custom.checkfile == down ) then restart();]]></failover>…
bin/checker.ps1
Note the call to safekit set -r custom.checkfile -m AM to set the resource status(up or down)
param([Parameter(Mandatory = $true, ValueFromPipeLine = $true,
position=1)][String]$ModName,[Parameter(Mandatory = $true, ValueFromPipeLine = $true,
position=2)][String]$RName,[Parameter(Mandatory = $true, ValueFromPipeLine = $true,
position=3)][String]$Arg1Value,[Parameter(Mandatory = $false, ValueFromPipeLine = $false,
position=4)][String]$Grace="2",[Parameter(Mandatory = $false, ValueFromPipeLine = $false,
position=5)][String] $Period="5")# return up on success | down on failure
Function test([String]$Arg1Value){$res="down"# Replace the following by your test
if (Test-Path "$Arg1Value"){$res="up"}return $res}
$customchecker=$MyInvocation.MyCommand.Name$safekit="$env:SAFE/safekit.exe"$safebin="$env:SAFEBIN"$gracecount=0$prevrstate="unknown"# wait a little
Start-Sleep $Period
while ($true){
Start-Sleep $Period$rstate = test($Arg1Value)

39 A2 38MC 03

287

SafeKit User's Guideif($rstate -eq "down"){$gracecount+=1}else{$gracecount = 0
if($prevrstate -ne $rstate){& $safekit set -r "$RName" -v $rstate -i$customchecker -m $ModName$prevrstate = $rstate}}if($gracecount -ge $Grace){
if($prevrstate -ne $rstate){& $safekit set -r "$RName" -v $rstate -i$customchecker -m $ModName$prevrstate = $rstate}$gracecount = 0}}
The executable associated with the checker is automatically called with at least 2
arguments:
The 1st argument is the module name
The 2nd is the name of the resource to be assigned
If the <custom> configuration contains the arg attribute, its value is passed as the next
arguments.
The checker script is written with the following precautions:
The resource is only assigned if its value has changed
When the resource is down, the checker consolidates this state (grace times) before
assigning it. This can help to avoid false error detections.
Each time you modify the custom checker script in SAFE/modules/AM/bin/,you must apply the new configuration.

15.13 Module checker example with leader.safe andfollower.safe
This example describes the two application modules leader.safe and follower.safe
delivered with SafeKit:
The leader module defines shared SafeKit resources between followers like virtual IP
addresses and replicated directories
The follower modules contain individual start and stop of several applications that arethen isolated in different modules. Each follower module can be started and stopped
independently without stopping the other modules.

288

39 A2 38MC 03

Examples of userconfig.xml and module scripts
The leader module is configured for a mirror architecture. It also includes the start andstop of the follower modules.
Each follower module is configured for a light architecture with module scripts and error
detectors. The follower modules depend on the leader failover with the following module
checker:follower/conf/userconfig.xml - see 13 page 209<check><module name="leader"/></check>This is a shortcut for:<module name="leader"><to addr="127.0.0.1" port="9010"/></module>
If you change the listening port for the SafeKit web service (asdescribed in 10.6 page 166), replace the short configuration with the fullone and change the port value.

15.14

Mail notification example with notification.safe

The notification.safe module is a mirror demonstration module for sending
notification on main module state changes. The following example is for sending an email but you can replace it by any other notification mechanism. In Windows, it uses theSend-MailMessage from the Microsoft Powershell Utility. In Linux, it uses the mailcommand.
Each time you modify a script in SAFE/modules/AM/bin/, you must apply the
new configuration.

15.14.1 Notification on the start and the stop of the module
The following lines, inserted into at the end of the prestart script of a module (named
AM), send an e-mail with the name of the module and server on which the module isstarted:In Windows: c:\safekit\modules\AM\bin\prestart.ps1$sub= (Get-Item env:SAFEUSERBIN).Value$safebin= (Get-Item env:SAFEBIN).Value$module= (Get-Item env:SAFEMODULE).Value$action = $args[2]$retval = 0$hostname=(Get-Item env:computername).Value
if ( $action -eq "start" ) {echo "*** Start of module $module on $hostname"# insert here your notification: the module is starting# Send-MailMessage -From 'SafeKit' -To 'admin@mydomain.com' -Subject 'Start ofmodule $module on $hostname' -Body 'Running prestart'}
39 A2 38MC 03289

SafeKit User's Guide

In Linux: /opt/safekit/modules/AM/bin/prestart
if [ "$3" = "start" ]; thenecho "*** Start of module $SAFEMODULE on " `hostname`# insert here your notification: the module is starting#echo "Running prestart" | mail -s " Start of module $SAFEMODULE on`hostname`" admin@mydomain.com
fi
When the module is stopping, it can be notified using the poststop script. This one is notdelivered by default and can be created as follow (for the module named AM):In Windows: c:\safekit\modules\AM\bin\poststop.ps1# Script called on module stop
# after resetting SafeKit resourcesecho "Running poststop $args"
try{$module= (Get-Item env:SAFEMODULE).Value$hostname=(Get-Item env:computername).Value$action = $args[2]$retval = 0
if ( $action -eq "stop" ) {echo "*** Stop of module $module on $hostname"# insert here your notification: the module is stopping# Send-MailMessage -From 'SafeKit' -To 'admin@mydomain.com' -Subject'Stop of module $module on $hostname' -Body 'Running poststop'}}catch{$retval=-1}finally{echo "poststop exit ($retval)"
exit $retval}
In Linux: /opt/safekit/modules/AM/bin/poststop#!/bin/sh# Script called on module stop
# after resetting SafeKit resources# For logging into SaKit log use:
# $SAFE/safekit printi | printe "message"
echo "Running poststop $*"if [ "$3" = "stop" ]; thenecho "*** Stop of module $SAFEMODULE on " `hostname`# insert here your notification: the module is stopping#echo "Running poststop" | mail -s " Stop of module $SAFEMODULE on `hostname`"admin@mydomain.com
fi

15.14.2 Notification on module state changes
The module script transition can be used to send an e-mail on main local statetransitions of the module. For instance, it may be useful to know when the mirror module290

39 A2 38MC 03

Examples of userconfig.xml and module scripts
is going ALONE (on failover for instance). The script transition is not delivered bydefault and can be created as follow.
For a farm module, change the state values.
In Windows: c:\safekit\modules\AM\bin\transition.ps1# Script called on module state changeecho "Running transition $args"
try{$module= (Get-Item env:SAFEMODULE).Value$hostname=(Get-Item env:computername).Value$from = $args[0]$to = $args[1]$retval = 0
if ( $from -eq "WAIT" -and $to -eq "ALONE" ) {echo "*** Start ALONE of $module on $hostname"# insert here your notification: the module is starting as ALONE# Send-MailMessage -From 'SafeKit' -To 'admin@mydomain.com' Subject 'Start ALONE of module $module on $hostname' -Body 'Running
prestart'}if ( $from -eq "WAIT" -and $to -eq "PRIM" ) {echo "*** Start PRIM of $module on $hostname"# insert here your notification: the module is starting as PRIM
# Send-MailMessage -From 'SafeKit' -To 'admin@mydomain.com' Subject 'Start PRIM of module $module on $hostname' -Body 'Running
prestart'}if ( $from -eq "WAIT" -and $to -eq "SECOND" ) {echo "*** Start SECOND of $module on $hostname"# insert here your notification: the module is starting as SECOND# Send-MailMessage -From 'SafeKit' -To 'admin@mydomain.com' Subject 'Start SECOND of module $module on $hostname' -Body 'Running
prestart'}if ( $from -ne "WAIT" -and $to -eq "ALONE" ) {echo "*** Go ALONE of module $module on $hostname"# insert here your notification: the module is going ALONE# Send-MailMessage -From 'SafeKit' -To 'admin@mydomain.com' Subject 'Go ALONE of module $module on $hostname' -Body 'Running prestart'}
if ( $from -ne "WAIT" -and $to -eq "PRIM" ) {echo "*** Go PRIM of module $module on $hostname"# insert here your notification: the module is going PRIM
# Send-MailMessage -From 'SafeKit' -To 'admin@mydomain.com' Subject 'Go PRIM of module $module on $hostname' -Body 'Running prestart'}
if ( $from -ne "WAIT" -and $to -eq "SECOND" ) {echo "*** Go SECOND of module $module on $hostname"# insert here your notification: the module is going SECOND# Send-MailMessage -From 'SafeKit' -To 'admin@mydomain.com' Subject 'Go SECOND of module $module on $hostname' -Body 'Running prestart'}}catch{$retval=-1}finally{echo "transition exit ($retval)"

39 A2 38MC 03

291

SafeKit User's Guide

exit $retval}
In Linux: /opt/safekit/modules/AM/bin/transition#!/bin/sh# Script called on module state change# For logging into SaKit log use:
# $SAFE/safekit printi | printe "message"
echo "Running transition $*"hostname=`hostname`if [ "$1" = "WAIT" -a "$2" = "ALONE" ] ; thenecho "*** Start ALONE of module $SAFEMODULE on $hostname"# insert here your notification: the module is starting as ALONE#echo "Running poststop" | mail -s " Start ALONE of module $SAFEMODULE on$hostname" admin@mydomain.com
fi
if [ "$1" = "WAIT" -a "$2" = "PRIM" ] ; thenecho "*** Start PRIM of module $SAFEMODULE on $hostname"# insert here your notification: the module is starting as PRIM
#echo "Running poststop" | mail -s " Start PRIM of module $SAFEMODULE on$hostname" admin@mydomain.com
fi
if [ "$1" = "WAIT" -a "$2" = "SECOND" ] ; thenecho "*** Start SECOND of module $SAFEMODULE on $hostname"# insert here your notification: the module is starting as SECOND#echo "Running poststop" | mail -s " Start SECOND of module $SAFEMODULE on$hostname" admin@mydomain.com
fi
if [ "$1" != "WAIT" -a "$2" = "ALONE" ] ; thenecho "*** Go ALONE of module $SAFEMODULE on $hostname"# insert here your notification: the module is going ALONE#echo "Running poststop" | mail -s " Go ALONE of module $SAFEMODULE on$hostname" admin@mydomain.com
fi
if [ "$1" != "WAIT" -a "$2" = "PRIM" ] ; thenecho "*** Go PRIM of module $SAFEMODULE on $hostname"# insert here your notification: the module is going PRIM
#echo "Running poststop" | mail -s " Go PRIM of module $SAFEMODULE on$hostname" admin@mydomain.com
fi
if [ "$1" != "WAIT" -a "$2" = "SECOND" ] ; thenecho "*** Go SECOND of module $SAFEMODULE on $hostname"# insert here your notification: the module is going SECOND#echo "Running poststop" | mail -s " Go SECOND of module $SAFEMODULE on$hostname" admin@mydomain.com
fi

292

39 A2 38MC 03

16.SafeKit cluster in the cloud16.1 “SafeKit cluster in Amazon AWS” page 293
16.2 “SafeKit cluster in Microsoft Azure” page 297
16.3 “SafeKit cluster in Google GCP” page 300
You can install, configure, and administer SafeKit modules that run on virtual servers in
the cloud instead of on-premises physical servers. This requires a minimum of cloud
and/or server settings, especially to implement the virtual IP address.

16.1

SafeKit cluster in Amazon AWS

In the following, we suppose that you are familiar with:
Amazon Elastic Compute Cloud (Amazon EC2) that offers computing capacity in the
Amazon Web Services (AWS) cloud. For more information about the features of
Amazon EC2, see the Amazon EC2 product pageAWS CloudFormation that helps deploying instances and applications on Amazon EC2.
It permits to save a lot of time and effort so that you can spend less time managing
EC2 resources and more time focusing on your applications that run in AWS.
Before implementing a SafeKit module, the administrator must :1. Create instances (2 for a mirror module)
2. Make settings for AWS, instances, and SafeKit.
3. Then, apply specific settings for implementing your SafeKit module.
AWS settings
You must set AWS to:associate public addresses to each instance if you want to administer them with the
SafeKit web console from the internet
configure the security groups associated with network(s) to enable thecommunications of the SafeKit framework and the SafeKit web console. The ports toopen are described in 10.3.3.2 page 161
use a high-bandwidth, low-latency network if real-time replication is used in a mirror
moduleInstances settings
In each instance, you must also:
install the SafeKit package
apply the HTTPS configuration to secure the SafeKit web console (described in 11
page 175)

39 A2 38MC 03

293

SafeKit User's Guide

SafeKit settings
Finally, you must enter the SafeKit cluster configuration and apply it to all nodes (fordetails on cluster configuration, see 12 page 203). For example, the SafeKit cluster
configuration file would be:<cluster><lans><lan name="default"><node name="Server1" addr="10.0.11.10"/><node name="Server2" addr="10.0.12.10"/></lan></lans></cluster>
The default lan is used for SafeKit framework communications between cluster nodes.

16.1.1

Mirror cluster in AWS

Mirror module features are operational in the AWS cloud (real-time file replication,
failover, process death detection, checkers, …), except the virtual IP address failover.
Anyway, you can set up a SafeKit mirror module on the cluster and use the Elastic loadbalancing provided by AWS (see Elastic load balancing products in AWS) in such way that
all the traffic is routed only to the primary node. An IP address and/or DNS name isassociated with the load balancer that plays the role of the virtual IP.

You must configure yourself the AWS load balancer and the security group.
For the load balancer, you must:specify the rules for your application
set the SafeKit cluster nodes in the target groupconfigure the health check. It tests whether the instance is in a healthy state or an
unhealthy state.
The load-balancer routes the traffic only to healthy instances. It resumes routing
requests to the instance when this one has been restored to a healthy state.

294

39 A2 38MC 03

SafeKit cluster in the cloud
SafeKit provides a health checker for SafeKit modules. For this, configure it in the loadbalancer with:HTTP protocol
port 9010, the SafeKit web service port
URL /var/modules/AM/ready.txt, where AM is the module nameIn a mirror module, the health checker:
returns OK, that means that the instance is healthy, when the module state is
PRIM (Ready) or
ALONE (Ready)returns NOT FOUND, that means that the instance is out of service, in all other states

The AWS network security group must be at least configured to enable communications
for the following protocols and ports:
UDP - 4800 for the safeadmin service (between SafeKit cluster nodes)
UDP - 8888 for the module heartbeat (between SafeKit cluster nodes)
TCP – 5600 for the module real time file replication (between SafeKit nodes)
TCP – 9010 for the load-balancer health check and the SafeKit web console in HTTP
TCP – 9453 for the SafeKit web console in HTTPSTCP – 9001 for configuring the SafeKit web console for HTTPSThe module’s port value depends on the module id (for details, see 10.3.3.2
page 161).The previous values are the one for the first module installed on
the node.

16.1.2

Farm cluster in AWS

Most farm module features are operational in the AWS cloud (process death detection,
checkers), except the virtual IP address with load-balancing. Anyway, you can set up a
SafeKit farm module on the cluster and use the Elastic load balancing provided by AWS(see Elastic load balancing products in AWS). An IP address and/or DNS name isassociated with the load balancer that plays the role of the virtual IP.

39 A2 38MC 03

295

SafeKit User's Guide

You must configure yourself the AWS load balancer and the security group.
For the load balancer, you must:specify the rules for your application
set the SafeKit cluster nodes in the target groupconfigure the health check. These tests whether the instance is in a healthy stateor an unhealthy state.
The load-balancer routes the traffic only to healthy instances. It resumes routing
requests to the instance when this one has been restored to a healthy state.
SafeKit provides a health check for SafeKit modules. For this, configure it in the loadbalancer with:HTTP protocol
port 9010, the SafeKit web service port
URL /var/modules/AM/ready.txt, where AM is the module nameIn a farm module, the health check:returns OK, that means that the instance is healthy, when the module state(Ready)UP

returns NOT FOUND, that means that the instance is out of service, in all other states
The AWS network security group must be at least configured to enable communications
for the following protocols and ports:
UDP - 4800 for the safeadmin service (between SafeKit cluster nodes)
TCP – 9010 for the load-balancer health check and the SafeKit web console in HTTP
TCP – 9453 for the SafeKit web console in HTTPSTCP – 9001 for configuring the SafeKit web console for HTTPS

296

39 A2 38MC 03

SafeKit cluster in the cloud

16.2

SafeKit cluster in Microsoft Azure

In the following, we suppose that you are familiar with Microsoft Azure that is a cloudcomputing service created by Microsoft for building, testing, deploying, and managing
applications and services through a global network of Microsoft-managed data centers.
For more information about the features and use of Azure, see the Microsoft Azure portal.
Before implementing a SafeKit module, the administrator must :1. Create virtual machines (2 for a mirror module)
2. Make settings for Azure, virtual machines, and SafeKit.
3. Then, apply specific settings for implementing your SafeKit module.
Azure settings
You must set Azure to:associate public IP addresses and DNS name to virtual machines if you want toadminister them with the SafeKit web console from the internet
configure the network security group to enable the communications of the SafeKit
framework and the SafeKit web console. The ports to open are described in 10.3.3.2
page 161
use a high-bandwidth, low-latency network if real-time replication is used in a mirror
moduleVirtual machines settings
On each virtual machine, you must also:
install the SafeKit package
apply the HTTPS configuration to secure the SafeKit web console (described in 11
page 175)
SafeKit settings
Finally, you must enter the SafeKit cluster configuration and apply it to all nodes (fordetails on cluster configuration, see 12 page 203). For example, the SafeKit cluster
configuration file would be:<cluster><lans><lan name="default"><node name="Server1" addr="10.0.0.10"/><node name="Server2" addr="10.0.0.11"/></lan></lans></cluster>
The default lan is used for SafeKit framework communications between cluster nodes.

16.2.1

Mirror cluster in Azure

Mirror module features are operational in the Azure cloud (real-time file replication,
failover, process death detection, checkers, …) except the virtual IP address failover.

39 A2 38MC 03

297

SafeKit User's Guide

Anyway, you can set up a SafeKit mirror module on the cluster and use the loadbalancing provided by Azure (see Load Balancer in Azure) and route request only to theprimary node. An IP is associated with the load balancer that plays the role of the virtualIP.

You must configure yourself the Azure load balancer and the network security group.
For the load balancer, you must:specify the rules for your application
set the SafeKit cluster nodes into the backend poolconfigure the probe. It tests whether the instance is in a healthy state or an
unhealthy state.
The load balancer routes traffic only to healthy instances. It resumes routing requests tothe instance when the instance has been restored to a healthy state.
SafeKit provides a probe for SafeKit modules. For this, configure the probe in the loadbalancer with:HTTP protocol
port 9010, the SafeKit web service port
URL /var/modules/AM/ready.txt, where AM is the module nameIn a mirror module, the probe:298

39 A2 38MC 03

SafeKit cluster in the cloud
returns OK, that means that the instance is healthy, when the module state is
PRIM (Ready) or
ALONE (Ready)returns NOT FOUND, that means that the instance is out of service, in all other states
The Azure network security group must be at least configured to enable communications
for the following protocols and ports:
UDP - 4800 for the safeadmin service (between SafeKit cluster nodes)
UDP - 8888 for the module heartbeat (between SafeKit cluster nodes)
TCP – 5600 for the module real time file replication (between SafeKit nodes)
TCP – 9010 for the load-balancer health check and the SafeKit web console in HTTP
TCP – 9453 for the SafeKit web console in HTTPSTCP – 9001 for configuring the SafeKit web console for HTTPSThe module’s port value depends on the module id (see 10.3.3.2 page161).The previous values are the one for the first module installed on thenode.

16.2.2

Farm cluster in Azure

Most farm module features are operational in the Azure cloud (process death detection,
checkers), except the virtual IP address with load-balancing. Anyway, you can set up a
SafeKit farm module on the cluster and use the load balancing provided by Azure (seeLoad Balancer in Azure). An IP is associated with the load balancer that plays the role ofthe virtual IP.

You must configure yourself the Azure load balancer and the network security group.

39 A2 38MC 03299

SafeKit User's Guide

For the load balancer, you must:specify the rules for your application
set the SafeKit cluster nodes as backend
configure the probe. It tests whether the instance is in a healthy state or an
unhealthy state.
The load balancer routes traffic only to healthy instances. It resumes routing requests tothe instance when the instance has been restored to a healthy state.
SafeKit provides a probe for SafeKit modules. For this, configure the probe in the loadbalancer with:HTTP protocol
port 9010, the SafeKit web service port
URL /var/modules/AM/ready.txt, where AM is the module nameIn a farm module, the probe:
returns OK, that means that the instance is healthy, when the farm module state is
UP (Ready)returns NOT FOUND, that means that the instance is out of service, in all other states

The Azure network security group must be at least configured to enable communications
for the following protocols and ports:
UDP - 4800 for the safeadmin service (between SafeKit cluster nodes)
TCP – 9010 for the load-balancer health check and the SafeKit web console in HTTP
TCP – 9453 for the SafeKit web console in HTTPSTCP – 9001 for configuring the SafeKit web console for HTTPS

16.3

SafeKit cluster in Google GCP

In the following, we suppose that you are familiar with Google Cloud Platform (GCP) thatdelivers virtual machines running in Google's innovative data centers and worldwide fiber
network. For more information about the features and use of Google Cloud Platform, see
the Google Cloud Computing documentation.
Before implementing a SafeKit module, the administrator must :1. Create virtual machines (2 for a mirror module)
2. Make settings for Google Compute Engine (GCP), virtual machines, and SafeKit.
3. Then, apply specific settings for implementing your SafeKit module.

GCP settings
300

39 A2 38MC 03

SafeKit cluster in the cloud
You must set GCP to:
associate an external IP address (and optionally DNS name) to each virtual machine
instance if you want to administer them with the SafeKit web console from theinternet
configure the firewall rules for the Virtual Private Cloud (VPC) network to enable thecommunications of the SafeKit framework and the SafeKit web console. The ports toopen are described in 10.3.3.2 page 161
use a high-bandwidth, low-latency network if real-time replication is used in a mirror
moduleVirtual machines settings
On each virtual machine, you must also:
install the SafeKit package
apply the HTTPS configuration to secure the SafeKit web console (described in 11 page
175)
SafeKit settings
Finally, you must enter the SafeKit cluster configuration and apply it to all nodes (fordetails on cluster configuration, see 12 page 203. For example, the SafeKit cluster
configuration file would be:<cluster><lans><lan name="default"><node name=" Inst1" addr="10.132.0.4"/><node name=" Inst2" addr="10.32.0.6"/></lan></lans></cluster>
The default lan is used for SafeKit framework communications between cluster nodes.

16.3.1

Mirror cluster in GCP

Mirror module features are operational in the Google Cloud Platform (real-time file
replication, failover, process death detection, checkers, …) except the virtual IP address
failover. Anyway, you can set up a SafeKit mirror module on the cluster and use the loadbalancing provided by GCP (see Load Balancer in GCP) and route request only to theprimary node. An IP is associated with the load balancer that plays the role of the virtualIP.

39 A2 38MC 03

301

SafeKit User's Guide

You must configure yourself the Google load balancer and the network firewall.
For the load balancer, you must:specify the rules for your application
set the SafeKit cluster nodes as backend
configure the health check. It tests whether the instance is in a healthy state or an
unhealthy state.
The load balancer routes traffic only to healthy instances. It resumes routing requests tothe instance when the instance has been restored to a healthy state.
SafeKit provides a health check for SafeKit modules. For this, configure the health checkin the load balancer with:HTTP protocol
port 9010, the SafeKit web service port
URL /var/modules/AM/ready.txt, where AM is the module nameIn a mirror module, the health check:returns OK, that means that the instance is healthy, when the module state
PRIM (Ready) or
ALONE (Ready)returns NOT FOUND, that means that the instance is unhealthy, in all other states302

39 A2 38MC 03

SafeKit cluster in the cloud
The network firewall must be at least configured to enable communications for thefollowing protocols and ports:
UDP - 4800 for the safeadmin service (between SafeKit cluster nodes)
UDP - 8888 for the module heartbeat (between SafeKit cluster nodes)
TCP – 5600 for the module real time file replication (between SafeKit nodes)
TCP – 9010 for the load-balancer health check and the SafeKit web console in HTTP
TCP – 9453 for the SafeKit web console in HTTPSTCP – 9001 for configuring the SafeKit web console for HTTPSThe module’s port value depends on the module id (see 10.3.3.2 page161).The previous values are the one for the first module installed on thenode.

16.3.2

Farm cluster in GCP

Most farm module features are operational in the Google Cloud Platform (process death
detection, checkers), except the virtual IP address with load-balancing. Anyway, you canset up a SafeKit farm module on the cluster and use the load balancing provided by GCP(see Load Balancer in GCP). An IP is associated with the load balancer that plays the roleof the virtual IP.

You must configure yourself the Google load balancer and the network firewall.
For the load balancer, you must:specify the rules for your application

39 A2 38MC 03

303

SafeKit User's Guide

set the SafeKit cluster nodes as backend
configure the health check. It tests whether the instance is in a healthy state or an
unhealthy state.
The load balancer routes traffic only to healthy instances. It resumes routing requests tothe instance when the instance has been restored to a healthy state.
SafeKit provides a health check for SafeKit modules. For this, configure the health checkin the load balancer with:HTTP protocol
port 9010, the SafeKit web service port
URL /var/modules/AM/ready.txt, where AM is the module nameIn a farm module, the health check:returns OK, that means that the instance is healthy, when the farm module state is
UP (Ready)returns NOT FOUND, that means that the instance is out of service, in all other states

The network firewall must be at least configured to enable communications for thefollowing protocols and ports:
UDP - 4800 for the safeadmin service (between SafeKit cluster nodes)
TCP – 9010 for the load-balancer health check and the SafeKit web console in HTTP
TCP – 9453 for the SafeKit web console in HTTPSTCP – 9001 for configuring the SafeKit web console for HTTPS

304

39 A2 38MC 03

17.Third-Party Software
SafeKit comes with the third-party software listed below.
For licenses details, refer to the links or the license files into the SAFE/licenses directory(SAFE=/opt/safekit in Linux and SAFE=C:\safekit in Windows if%SYSTEMDRIVE%=C:).

libnetPacket Construction and Injection
Libnet license - license
Used for arpreroute and pingswagger-ui

https://github.com/swagger-api/swagger-ui
Apache2 License - https://github.com/swagger-api/swaggerui/blob/master/LICENSE
Swagger UI is a collection of HTML, JavaScript, and CSS assets thatdynamically generate beautiful documentation from a Swagger-compliant
API
Used for to visualize the SafeKit API

Sqlite3https://www.sqlite.org/about.html
Public Domain License - https://www.sqlite.org/copyright.html
SQLite is an in-process library that implements a self-contained,
serverless, zero-configuration, transactional SQL database engine
Used by SafeKit framework

And on Windows OS only :libxmlhttp://xmlsoft.org
MIT license - http://www.xmlsoft.org/FAQ.html#License
Used by the SafeKit frameworklibxslt

http://xmlsoft.org/XSLT/MIT license https://gitlab.gnome.org/GNOME/libxslt/blob/master/Copyright
Used by the SafeKit framework

Net-SNMP

http://net-snmp.sourceforge.net
BSD like and BSD license - http://www.net-snmp.org/about/license.html
Used by SafeKit SNMP agent in Windows

HTTP server

https://httpd.apache.org/
Apache license - https://www.apache.org/licenses/LICENSE-2.0
Used by the SafeKit web service for the web console, the distributedcommands, and the module checker

APR

https://apr.apache.org/

39 A2 38MC 03

305

SafeKit User's Guide

Apache license - https://www.apache.org/licenses/LICENSE-2.0
Used by the Apache HTTP server
PCRE

http://www.pcre.org/
BSD license - https://www.pcre.org/licence.txt
Used by the Apache HTTP server

libexpathttps://github.com/libexpat/libexpat
BSD license https://github.com/libexpat/libexpat/blob/master/expat/COPYING
Used by the Apache HTTP server

mod_auth_openidchttps://github.com/OpenIDC/mod_auth_openidc
Apache2 License https://github.com/OpenIDC/mod_auth_openidc/blob/master/LICENSE.
txt
mod_auth_openidc is an OpenID Certified™ authentication andauthorization module for the Apache 2.x HTTP server that implements
the OpenID Connect Relying Party
Used by the Apache HTTP server

cURL

http://curl.haxx.se
Curl license - https://github.com/curl/curl/blob/master/docs/LICENSEMIXING.md
Used by the distributed commands and the module checker

OpenSSLhttp://www.openssl.org
dual OpenSSL and SSLeay license https://www.openssl.org/source/license.html
Used when securing the web console, the distributed commands, and themodule checker

Lua

http://www.lua.org
MIT license - https://www.lua.org/license.html
Used by SafeKit framework and the web service

Info-ZIP

http://info-zip.org
BSD like license - http://infozip.sourceforge.net/license.html
Used to pack/unpack a .safe module

306

39 A2 38MC 03

Third-Party Software

SafeKit uses the following third-party packages for the SafeKit web console:
Angularhttps://angular.io
MIT License - https://github.com/angular/angular-cli/blob/main/LICENSE
Angular is an application-design framework and development platform forcreating efficient and sophisticated single-page apps.
@angular/animations, @angular/cdk, @angular/common, @angular/core,@angular/forms, @angular/material, @angular/material-moment-adapter,@angular/platform-browser, @angular/routerjszip

https://stuk.github.io/jszip/
MIT OR GPL-3.0-or-later license https://github.com/Stuk/jszip/blob/main/LICENSE.markdown
A library for creating, reading, and editing .zip files with JavaScript, with alovely and simple API.

materialicons

https://github.com/marella/material-icons

momenthttps://github.com/urish/angular-moment#readme

Apache-2.0 license - https://github.com/marella/materialicons/blob/main/LICENSE
MIT license - https://github.com/urish/angular-moment?tab=MIT-1-ovfile

ngx-logger

https://github.com/dbfannin/ngx-logger#readme
MIT license - https://github.com/dbfannin/ngx-logger?tab=MIT-1-ov-file
NGX Logger is a simple logging module for angular

rxjs

https://github.com/ReactiveX/rxjs
Apache2 License –https://github.com/ReactiveX/rxjs/blob/master/LICENSE.txt
Reactive Extensions For JavaScript

tslib

https://www.typescriptlang.org/
0BSD Copyright (c) Microsoft Corporation
Runtime library for typescript

vlq

https://github.com/Rich-Harris/vlq/blob/master/README.md
MIT license - https://github.com/Rich-Harris/vlq/blob/master/LICENSE
Convert integers to a Base64-encoded VLQ string, and vice versa

zone.js

https://github.com/angular/zone.js
MIT license - https://angular.io/license
Implements Zones for JavaScript

This list is available in file : safekit/web/htdcos/console//en/3rdpartylicenses.txt .

39 A2 38MC 03307

SafeKit User's Guide

308

39 A2 38MC 03

Log Messages Index“Action …” messages"Action forcestop called by web@<IP>/SYSTEM/root", 116, 146"Action prim called by web@<IP>/SYSTEM/root",98, 146"Action primforce called by SYSTEM/root", 106"Action restart called by web@<IP>/SYSTEM/root",73, 79, 116, 146"Action restart|stopstart called by customscript", 92, 120, 146"Action restart|stopstart called by errd", 86, 120, 146"Action restart|stopstart from failover rule tcp_failure", 87, 120, 146"Action second called by web@<IP>/SYSTEM/root", 98, 146"Action shutdown called by SYSTEM", 76, 85, 143"Action start called at boot time", 76, 77, 85, 143"Action start called automatically", 86, 87, 92"Action start called by web@<IP>/SYSTEM/root", 72, 79, 116, 146"Action stop called by web@<IP>/SYSTEM/root", 72, 79, 116, 146"Action stopstart called by failover-off", 103, 146"Action stopstart called by modulecheck", 91, 146"Action stopstart called by web@<IP>/SYSTEM/root", 116, 146"Action stopstart from failover rule customid_failure", 92, 120, 146"Action swap called by web@<IP>/SYSTEM/root", 73, 116, 146"Action wait from failover rule customid_failure", 92, 119"Action wait from failover rule tcpid_failure", 88, 119"Action wait from failover rule degraded_server", 101"Action wait from failover rule interface_failure", 89, 119"Action wait from failover rule module_failure", 91, 119"Action wait from failover rule notuptodate_server", 100, 119"Action wait from failover rule ping_failure", 90, 119"Action wait from failover rule splitbrain_failure", 119File replication and reintegration messages"Copied <reintegration statistics>", 75"Data may be inconsistent for replicated directories (stopped during reintegration)", 106"Data may not be uptodate for replicated directories (wait for the start of the remote
server)", 98, 100, 119"If you are sure that this server has valid data, run safekit prim to force start asprimary", 98, 100, 119

39 A2 38MC 03309

SafeKit User's Guide"If you are sure that this server has valid data, run safekit primforce to force start asprimary", 106"Reintegration ended (synchronize)", 75"Updating directory tree from /replicated", 75

Load-balancing messages"farm load: 128/256 (group FarmProto)" , 109, 82, 83"farm membership: node1 (group FarmProto)", 82, 83"farm membership: node1 node2 (group FarmProto)" , 109, 82, 83"farm membership: node2 (group FarmProto)", 83“Local state …” messages"Local state ALONE Ready", 97, 72, 78"Local state PRIM Ready", 97,72"Local state SECOND Ready",97, 72"Local state UP Ready",108 ,109
"Local state WAIT NotReady", 119, 103“Remote state …” messages"Remote state ALONE Ready", 97,78"Remote state PRIM Ready", 97, 72"Remote state SECOND Ready",97, 72"Remote state UNKNOWN Unknown", 77, 78“Resource …” messages"Resource custom.id set to down by customscript", 92, 119, 120"Resource custom.id set to up by customscript", 92"Resource heartbeat.0 set to down by heart", 77, 78"Resource heartbeat.flow set to down by heart", 77, 78"Resource intf.ip.0 set to down by intfcheck", 89, 119"Resource intf.ip.0 set to up by intfcheck", 89"Resource module.othermodule_ip set to down by modulecheck", 91, 119"Resource module.othermodule_ip set to up by modulecheck", 91"Resource ping.id set to down by pingcheck", 90, 119"Resource ping.id set to up by pingcheck", 90"Resource rfs.degraded set to up by nfsadmin", 101

310

39 A2 38MC 03

"Resource tcp.id set to down by tcpcheck", 87, 88, 119, 120"Resource tcp.id set to up by tcpcheck", 88“Script …” messages"Script start_prim", 267, 72, 73, 76, 77"Script stop_prim", 267, 72, 76, 78"Script start_both", 267, 79, 85"Script stop_both", 267, 79“Transition …” messages"Transition RESTART|STOPSTART from failover rule customid_failure", 92"Transition STOPSTART from failover-off", 103"Transition SWAP from defaultprim", 105"Transition SWAP from SYSTEM", 73"Transition WAIT_TR from failover rule customid_failure", 92"Transition WAIT_TR from failover rule interface_failure", 89"Transition WAKEUP from failover rule Implicit_WAKEUP", 88, 89, 90, 91, 92

Other messages"Begin of Swap", 73, 105"End of stop", 72, 79, 76, 85"Process appli.exe not running", "Service mySQL not running", 86, 120"Failover-off configured", 103"Previous halt unexpected", 77, 85"Reason of failover: no heartbeat", 77"Reason of failover: remote stop", 72, 76"Requested prim start aborted ", 106"Split brain recovery: exiting alone", 78"Split brain recovery: staying alone", 78"Stopping loop", 121, 86, 87, 88, 89, 90, 91, 92, 92, 120"Virtual IP <ip 1.10 of mirror> set", 74"Virtual IP <ip1.20 of farm> set", 80

39 A2 38MC 03

311

SafeKit User's Guide

312

39 A2 38MC 03

IndexArchitectures
mirror, farm… - 15
cloud - 293
Installation
install, upgrade… - 25
Consoleconfiguration, monitoring- 37
securing (https, …) - 175
Advanced Configuration
cluster.xml - 203
userconfig.xml - 209
module scripts - 267
examples - 273
Administration
mirror - 95
farm - 107
advanced - 155
command line – 141
Supporttests - 69
troubleshooting - 111
call desk - 133
log messages - 309
Othertable of contents - 5
third-party software – 305

39 A2 38MC 03

313

SafeKit User's Guide

314

39 A2 38MC 03