Onebm: the ELETTRA Framework for Programmable MachineOperations
D. Bulfone, C. Scafuri
Sincrotrone Trieste, ss 14 Km. 163.5, 34012 Basovizza, Trieste, Italy
Abstract
The Onebm (one button machine) project has beendeveloped to automate the routine operations at ELETTRA.
Its framework consists of a programmable task manager
with a Motif user interface, a set of operation description
files and a collection of software modules.
The programmable task manager can spawn and track
the status of any valid UNIX executable, either
binary object or shell script. The spawn mechanism isbased on the standard "fork()" and "exec()" system calls.
The task manager also intercepts the standard output
stream of each task by redirecting it to a UNIX pipe. Theintercepted output stream is displayed on demand
to the operator. The task manager checks also the output
stream for the presence of messages identified bypredefined tokens. These messages are used for error
notification and are displyed on a special window. An
important point of the task manager is its capability
to handle many tasks concurrently, leading to substantial
savings in the execution times. These characteristics,
associated with the minimization of operator errors,
produce an increase of the available beam time.
The description files are simple ASCII files following apredefined syntax containing the logic flow of the tasks
and the associated parameters. The rules to start a task areboolean expressions whose factors are the exit
status of other tasks. This formalism has been foundcapable of expressing any operation so far analyzed.
Each software module is a program designed to handle awell defined task on the machine, usually involving asingle machine subsystem. A C++ library has been written
to facilitate the development of new modules. The
classes of this library encapsulate all the ELETTRA field
access routines, provide utilities to generate the special
error messages recognized by the task manager andterminate the module with the correct exit status.
A detailed explanation of the design issues,
implementation choices and techniques is given.
1 Machine automation
The operations needed by a user oriented facility like a
synchrotron radiation source are usually very repetitiveand firmly established. An "operation" is defined as theset of steps necessary to bring the machinefrom one operating condition to another, like theswitching on of machine equipment after a shutdown or
the daily re-filling for user shifts.
The goal of the Onebm project is to automize as muchas possible such operations, which should be ideallycarried out by "just pressing one button" on the Control
Room console. The associated advantages are theminimization of the time needed to perform an operationand the almost complete elimination of operator errors.
The project result is the Onebm framework, which
includes the Onebm main program, the specification of thesyntax to write an operation description file, thedefinition of rules for accepting external programs as
Onebm modules and a support library for writing new
Onebm modules.
1.1 Overview and definitions
The strategy followed to handle the problem ofautomating an operation is the age old idea of partitioning
the problem into sub problems and solve them separately.
Therefore the "operation" is subdivided into a series of"tasks". A task usually handles a well defined subsystem of
the plant. It is started at a certain moment and mustterminate with a clear indication of its failure or success. A
task is carried out by a dedicated "module" which
consists of a UNIX executable. Modules are started withan arbitrary number of input parameters to add flexibility.
The execution of the tasks must be co-ordinated so that agiven task can be run only after the successful
termination of a well defined set of tasks. The rules that
regulate the task execution together with the associatedparameters constitute the so called high level description of
the operation [1]. We have devised a very simple syntaxto express these rules in a human and machine readableformat. An "operation description file" is written using astandard text editor and can be interpreted by the Onebm
main program.
Onebm starts the operation modules following theprecedence rules written in the operation description file. It
concurrently executes all the modules that are not blocked
by the successful termination of other modules. Thisfeature, based on the multi-tasking capability of UNIX,
minimizes the time needed to perform a given operation.
1.2 Planning an operation
If we change the word “operation” with “project”, we
see that we can take advantage of the PERT technique ofManagement Science [2] to analyze a givenmachine operation and the schedule of its associated tasks.
The PERT algorithm is widely used for business andmanagement planning, thanks also to some very powerfuland easy to use PC based implementations. This
mathematical tool identifies the critical time path of agiven operation pointing out bottlenecks.
A limitation of PERT is that it has no concept of task
failure and does not handle alternative routes to start a task.
More formally, the condition for the execution of a task isexpressed as the logical product (boolean AND) of thesuccessful termination of a set of tasks (a "milestone" inthe PERT terminology). In Onebm we can express thestarting condition of a task as a more general boolean
expression of sums (boolean OR) and products of both thesuccess or failure of a set of tasks. This richer set of rules
is necessary to handle alternative paths for task execution
and to plan some failure recovery actions (rules containing
negated terms).
Apart from this limitation we have found that a PERT
chart (figure 1) is a very handy tool for the definition of anew operation. It shows both the main execution path andthe task partitioning. Later we can exploit thecapabilities of the Onebm rules to handle the alternative or
fault recovery paths of execution.

simple rules that the candidate module must comply with.
• It must be based on UNIX conventions, system calls,libraries and system resources.
• It must take its execution parameters from the commandline.
• It must print any information or error message on thestandard output.
• It must terminate clearly at a given moment returning awell defined exit code to the calling environment,
indicating the success or failure of the execution.
• It must run as a regular process without turning itself
into a background process.
These rules are followed by any well behaved UNIXexecutable. Many of the already existing control room
programs could be used by Onebm with little or
no modification at all.
2.1 Special strings and exit conventions

Fig.1: PERT chart of the injection operation.
1.3 Dealing with faults and errors
Errors and failures can occur during theexecution of operation tasks. Two schemes are foreseen inthis case.
Onebm can trigger the execution of a dedicated error
recovery task on the faulty termination of a set of tasks byusing negated terms in the starting expression.
Alternatively, Onebm detects the fault condition andallows the operator to intervene manually. The operator
can restart the failed task, which in many cases solves thesituation, or he can try to solve the problem by other means,e.g. by resetting a faulty equipment. The operator thenmarks the task as "done", informing Onebm that it has
been completed, and automatically unlocks theremaining operation tasks.
It should be noted that Onebm is part of the ELETTRA
Control System [3] so that the operator has accessto many other tools which support his decisions in case ofproblems: the list of machine alarms and their history logfile [4], machine physics programs and individual control
panels with extensive diagnostics for each machine
equipment.
If the problem is not immediately solvable, the operator
marks the task as "ended with an unrecoverable error".
This usually inhibits the operation termination, which must
be repeated from the beginning after the problem has been
fixed.
2 Onebm modules
Almost any UNIX executable, binary or shell script, canbe used as a Onebm module. The following are the few

Onebm recognizes some special strings written by amodule to its standard output. These special strings are
displayed on a separate section of the module’s message
area on the Onebm user interface. These special strings areuseful to help the operator to quickly identify any
problem or malfunction of the running module.
The Onebm framework defines another set of
conventions for the exit codes of a module. These exit
codes extend the UNIX conventions for indicating thefaulty termination of a process, so that it is possible
to notify also the type of failure and force the correct exit
status of the task state-machine. The use of theseextensions, although highly recommended for newly
designed programs, is anyway not mandatory.
All the necessary definitions for formatting the special
strings and the extended exit codes are available
to programmers in a C-language header file.
3 The task scheduler
Onebm represents each task as a finite state-machine.
Figure 2 is a schematic of the implemented transition
diagram. Shaded arrows represent transitions which can beforced “manually" by the operator through the userinterface buttons.

Fig.2: task state transition diagram
The different states are defined as follows:
INITIAL: is the starting state for each task; the task has
never executed before.
FROZEN: the task execution is blocked manually bythe operator.

RUNNING: the task is executing for the first time.
ERROR: the task execution failed with an unrecoverable
error.
WARNING: the task execution failed with a recoverableerrorDONE: the task execution terminated successfully.
RECOVER: the task is being re-run after a recoverableerror.
3.1 The task spawner
When one of the tasks changes its state Onebm checks
the logic conditions for all the other tasks. All the tasks
which are found to be ready to run are spawned. The
spawning is performed by means of the standard fork() andexec() UNIX system calls. Onebm forks twice so thatthe original copy of Onebm (the father) can continue thenormal processing where the child copy of it forks again.
The newly forked child starts the desired module with anexec() call, while the intermediate child performs a wait()
call and suspends its execution until the spawned moduleterminates. Upon termination of the spawned module theintermediate child reports the module exit code
to the original Onebm.
The standard output of the spawned modules isredirected to a UNIX pipe. The other end of the pipe isread by the father process. All the messages written by amodule are thus intercepted and stored by Onebm. The
module exit code is also sent to the main process bymeans of this pipe.
4 The graphical user interface
Onebm has a graphical user interface based on the Motifstandard. The interface is designed to present the control
room operator only with the information strictly necessary
to follow the operation progress.
Each task has a dedicated area on the interface. During
the task execution its label changes colour and flashes
according to the state of the task. In case of problems
the operator can ask for more details about a task
by opening a window where all the module messages arelogged. The special Onebm strings are emphasized in red
and grouped in a reserved area. Another auxiliary window
available on request shows some essential help messages
about the task. These messages are configured by means of
the operation description file.
The auxiliary area of a task contains also the buttons
used to manually force the state transitions in case ofmalfunctions.
5 Support for writing new modules
An extensive analysis of various possible scenarios
convinced us that it is better to have a collection of small
and well focused programs, each of them dedicated to asingle task of the operation. This is a well known idea
which is, among the other things, one of the foundations ofthe UNIX environment.
The analysis of the various tasks showed us that themajority of them has a very repetitive pattern ofactions: open the Remote Procedure Call (RPC)
connection to the desired device, check for a pre-condition,
write a set of values, wait and check until thedesired operating point is reached, close the RPC
connection; in case of errors write a diagnostic message
to the standard output exploiting the special strings
recognised by Onebm. All these functions have been
encapsulated in a dedicated C++ class [5]. This class hides
the programmer of all the details of RPC handling and,exploiting C++ polymorphism, provides a uniform set ofcalls for all the control system points and data types. This
class supplies also the methods to automatically wait for amachine parameter to reach a desired value. The use of thisclass proved to be very effective. Of the twelve different
modules used for the daily injection operation atELETTRA, seven are based on the special C++ class, one
is a shell script starting some standard control panels,
two are existing machine physics and only two modulesare dedicated Onebm programs which are not exclusively
based on the special C++ class. This allowed us to save a
lot of developing time and assured a consistent and reliablebehaviour of the modules.
6 The injection operation
The daily injection operation at ELETTRA [6] is now
fully under Onebm control. The latest improvements are aseries of modules to assist the operator during the manual
filling of the ring, prepare the energy ramping of the ring
and automatically perform the ramp to the final operating
condition. These improvements lead to a further
reduction of the time needed to perform the injection
which can be as short as 20 minutes.
7 Conclusions
The Onebm project met all its goals. The framework is aportable, flexible and easy to use tool.
Thanks to its modularity and to its specialized support
library, modifications or extensions of existing operations
are implemented in a short time and with very little effort.
New operations can also be designed and installed with areduced investment in time and manpower.
References[1] F. Potepan “Automize Machine Operation at Control
Room Level: One Button Machine Program”, ST/MTN-96/3, Sincrotrone Trieste, June 1996.
[2] R. E. Markland “Topics in Management Science”,
John Wiley & Sons, 1983, pp. 405-416.
[3] D. Bulfone: “Status and Prospects of the ELETTRA
Control System”, Nucl. Instr. and Meth. A352, p.
63, 1994.
[4] P. Michelini, C. Scafuri: “A New Alarm System
Processor for ELETTRA”, Proc. EPAC96, Sitges,Spain, 1996.
[5] F. Potepan: “ONEBM modules”, ST/M-TN-96/4,
Sincrotrone Trieste, June 1996
[6] D.
Bulfone,C.
Scafuri:
Automating
ELETTRA operation with "one button machine". Proc.
Particle Accelerator Conference, Vancouver 1997.