Package: papa
Summarypapais a process kernel. It contains both a client library and a
server component for creating sockets and launching processes from a
stable parent process.DependenciesPapa has no external dependencies, and it never will! It has been tested
under the following Python versions:2.62.73.23.33.4Installation$> pip install papaPurposeSometimes you want to be able to start a process and have it survive on
its own, but you still want to be able to capture the output. You could
daemonize it and pipe the output to files, but that is a pain and lacks
flexibility when it comes to handling the output.Process managers such as circus and supervisor are very good for
starting and stopping processes, and for ensuring that they are
automatically restarted when they die. However, if you need to restart
the process manager, all of their managed processes must be brought down
as well. In this day of zero downtime, that is no longer okay.Papa is a process kernel. It has extremely limited functionality and it
has zero external dependencies. If I’ve done my job right, you should
never need to upgrade the papa package. There will probably be a few bug
fixes before it is really “done”, but the design goal was to create
something that did NOT do everything, but only did the bare minimum
required. The big process managers can add the remaining features.Papa has 3 types of things it manages:SocketsValuesProcessesHere is what papa does:Create sockets and close socketsSet, get and clear named valuesStart processes and capture their stdout/stderrAllow you to retrieve the stdout/stderr of the processes started by
papaPass socket file descriptors and port numbers to processes as they
startHere is what it does NOT do:Stop processesSend signals to processesRestart processesCommunicate with processes in any way other than to capture their
outputSocketsBy managing sockets, papa can manage interprocess communication. Just
create a socket in papa and then pass the file descriptor to your
process to use it. See theCircus
docsfor a
very good description of why this is so useful.Papa can create Unix, INET and INET6 sockets. By default it will create
an INET TCP socket on an OS-assigned port.You can pass either the file descriptor (fileno) or the port of a socket
to a process by including a pattern like this in the process arguments:$(socket.my_awesome_socket_name.fileno)$(socket.my_awesome_socket_name.port)ValuesPapa has a very simple name/value pair storage. This works much like
environment variables. The values must be text, so if you want to store
a complex structure, you will need to encode and decode with something
like theJSON module.The primary purpose of this facility is to store state information for
your process that will survive between restarts. For instance, a process
manager can store the current state that all of its managed processes
are supposed to be in. Then if the process manager is restarted, it can
restore its internal state, then go about checking to see if anything on
the machine has changed. Are all processes that should be running
actually running?ProcessesProcesses can be started with or without output management. You can
specify a maximum size for output to be cached. Each started process has
a management thread in the Papa kernel watching its state and capturing
output if necessary.A Note on Naming (Namespacing)Sockets, values and processes all have unique names. A name can only
represent one item per class. So you could have an “aack” socket, an
“aack” value and an “aack” process, but you cannot have two “aack”
processes.All of the monitoring commands support a final asterix as a wildcard. So
you can get a list of sockets whose names match “uwsgi*” and you would
get any socket that starts with “uwsgi”.One good naming scheme is to prefix all names with the name of your own
application. So, for instance, the Circus process manager can prefix all
names with “circus.” and the Supervisor process manager can prefix all
names with “supervisor.”. If you write your own simple process manager,
just prefix it with “tweeter.” or “facebooklet.” or whatever your
project is called.If you need to have multiple copies of something, put a number after a
dot for each of those as well. For instance, if you are starting 3
waitress instances in circus, call themcircus.waitress.1,circus.waitress.2, andcircus.waitress.3. That way you can query
for all processes namedcircus.*to see all processes managed by
circus, or query forcircus.waitress.*to see all waitress processes
managed by circus.Starting the kernelThere are two ways to start the kernel. You can run it as a process, or
you can just try to access it from the client library and allow it to
autostart. The client library uses a lock to ensure that multiple
threads do not start the server at the same time but there is currently
no protection against multiple processes doing so.By default, the papa kernel process will communicate over port 20202.
You can change this by specifying a different port number or a path. By
specifying a path, a Unix socket will be used instead.If you are going to be creating papa client instances in many places in
your code, you may want to just callpapa.set_default_portorpapa.set_default_pathonce when your application is starting and
then just instantiate the Papa object with no parameters.Telnet interfacePapa has been designed so that you can communicate with the process
kernel entirely without code. Just start the Papa server, then do this:telnet localhost 20202You should get a welcome message and a prompt. Type “help” to get help.
Type “help process” to get help on the process command.The most useful commands from a monitoring standpoint are:list socketslist processeslist valuesAll of these can by used with no arguments, or can be followed by a list
of names, including wildcards. For instance, to see all of the values in
the circus and supervisor namespaces, do this:list values circus.* supervisor.*You can abbreviate every command as short as you like. So “l p” means
“list processes” and “h l p” means “help list processes”Creating a ConnectionYou can create either long-lived or short-lived connections to the Papa
kernel. If you want to have a long-lived connection, just create a Papa
object to connect and close it when done, like this:class MyObject(object):
    def __init__(self):
        self.papa = Papa()

    def start_stuff(self):
        self.papa.make_socket('uwsgi')
        self.papa.make_process('uwsgi', 'env/bin/uwsgi', args=('--ini', 'uwsgi.ini', '--socket', 'fd://$(socket.uwsgi.fileno)'), working_dir='/Users/aackbar/awesome', env=os.environ)
        self.papa.make_process('http_receiver', sys.executable, args=('http.py', '$(socket.uwsgi.port)'), working_dir='/Users/aackbar/awesome', env=os.environ)

    def close(self):
        self.papa.close()If you want to just fire off a few commands and leave, it is better to
use thewithmechanism like this:from papa import Papa

with Papa() as p:
    print(p.list_sockets())
    print(p.make_socket('uwsgi', port=8080))
    print(p.list_sockets())
    print(p.make_process('uwsgi', 'env/bin/uwsgi', args=('--ini', 'uwsgi.ini', '--socket', 'fd://$(socket.uwsgi.fileno)'), working_dir='/Users/aackbar/awesome', env=os.environ))
    print(p.make_process('http_receiver', sys.executable, args=('http.py', '$(socket.uwsgi.port)'), working_dir='/Users/aackbar/awesome', env=os.environ))
    print(p.list_processes())This will make a new connection, do a bunch of work, then close the
connection.Socket CommandsThere are 3 socket commands.p.list_sockets(*args)Thesocketscommand takes a list of socket names to get info about.
All of these are valid:p.list_sockets()p.list_sockets('circus.*')p.list_sockets('circus.uwsgi','circus.nginx.*','circus.logger')Adictis returned with socket names as keys and socket details as
values.p.make_socket(name, host=None, port=None, family=None, socket_type=None, backlog=None, path=None, umask=None, interface=None, reuseport=None)All parameters are optional except for the name. To create a standard
TCP socket on port 8080, you can do this:p.make_socket('circus.uwsgi', port=8080)To make a Unix socket, do this:p.make_socket('circus.uwsgi', path='/tmp/uwsgi.sock')A path for a Unix socket must be an absolute path ormake_socketwill raise apapa.Errorexception.You can also leave out the path and port to create a standard TCP socket
with an OS-assigned port. This is really handy when you do not care what
port is used.If you callmake_socketwith the name of a socket that already
exists, papa will return the original socket if all parameters match, or
raise apapa.Errorexception if some parameters differ.See themake_socketsmethod of the Papa object for other parameters.p.remove_sockets(*args)Theremove_socketscommand also takes a list of socket names. All of
these are valid:p.remove_sockets('circus.*')p.remove_sockets('circus.uwsgi','circus.nginx.*','circus.logger')Removing a socket will prevent any future processes from using it, but
any processes that were already started using the file descriptor of the
socket will continue to use the copy they inherited.Value CommandsThere are 4 value commands.p.list_values(*args)Thelist_valuescommand takes a list of values to retrieve. All of
these are valid:p.list_values()p.list_values('circus.*')p.list_list_values('circus.uwsgi','circus.nginx.*','circus.logger')Adictwill be returned with all matching names and values.p.set(name, value=None)To set a value, do this:p.set('circus.uswgi', value)You can clear a single value by setting it toNone.p.get(name)To retrieve a value, do this:value = p.get('circus.uwsgi')If no value is stored by that name,Nonewill be returned.p.remove_values(*args)To remove a value or values, do something like this:p.remove_values('circus.*')p.remove_values('circus.uwsgi','circus.nginx.*','circus.logger')You cannot remove all variables so passing no names or passing*will raise apapa.Errorexception.Process CommandsThere are 4 process commands:p.list_processes(*args)Thelist_processescommand takes a list of process names to get info
about. All of these are valid:p.list_processes()p.list_processes('circus.*')p.list_processes('circus.uwsgi','circus.nginx.*','circus.logger')Adictis returned with process names as keys and process details as
values.p.make_process(name, executable, args=None, env=None, working_dir=None, uid=None, gid=None, rlimits=None, stdout=None, stderr=None, bufsize=None, watch_immediately=None)Every process must have a uniquenameand anexecutable. All
other parameters are optional. Themake_processmethod returns adictthat contains the pid of the process.Theargsparameter should be a tuple of command-line arguments. If
you have only one argument, papa conveniently supports passing that as a
string.You will probably want to passworking_dir. If you do not, the
working directory will be that of the papa kernel process.By default, stdout and stderr are captured so that you can retrieve them
with thewatchcommand. By default, thebufsizefor the output
is 1MB.Valid values forstdoutandstderrarepapa.DEVNULLandpapa.PIPE(the default). You can also passpapa.STDOUTtostderrto merge the streams.If you passbufsize=0, not output will be recorded. Otherwise,
bufsize can be the number of bytes, or a number followed by ‘k’, ‘m’ or
‘g’. If you want a 2 MB buffer, you can passbufsize='2m', for
instance. If you do not retrieve the output quicky enough and the buffer
overflows, older data is removed to make room.If you specifyuid, it can be either the numeric id of the user or
the username string. Likewise,gidcan be either the numeric group
id or the group name string.If you want to specifyrlimits, pass adictwith rlimit names
and numeric values. Valid rlimit names can be found in theresourcesmodule. Leave off theRLIMIT_prefix. On my system, valid names areas,core,cpu,data,fsize,memlock,nofile,nproc,rss, andstack.rlimit={'cpu': 2, 'nofile': 1024}Theenvparameter also takes adictwith names and values. A
useful trick is to doenv=os.environto copy your environment to the
new process.If you want to run a Python application and you wish to use the same
Python executable as your client application, a useful trick is to passsys.executableas theexecutableand the path to the Python
script as the first element of yourargstuple. If you have no other
args, just pass the path as a string toargs.p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)The final argument that needs mention iswatch_immediately. If you
passTruefor this, papa will make the process and return aWatcher. This is effectively the same as doingp.make_process(name,...)followed immediately byp.watch(name),
but it has one fewer round-trip communication with the kernel. If all
you want to do is launch an application and monitor its output, this is
a good way to go.p.remove_processes(*args)If you do not care about retrieving the output or the exit code for a
process, you can useremove_processesto tell the papa kernel to
close the output buffers and automatically remove the process from the
process list when it exits.p.remove_processes('circus.logger')p.remove_processes('circus.uwsgi','circus.nginx.*','circus.logger')p.watch_processes(*args)Thewatch_processescommand returns aWatcherobject for the
specified process or processes. That object uses a separate socket to
retrieve the output of the processes it is watching.Optimization Note:Actually, it hijacks the socket of yourPapaobject. If you issue any other commands to thePapaobject that
require a connection to the kernel, thePapaobject will silently
create a new socket and connect up for the additional commands. If you
close theWatcherand thePapaobject has not already created a
new connection, the socket will be returned to thePapaobject. So
if you launch an application, usewatchto grab all of its output
until it closes, then use thesetcommand to update your saved
status, all of that can occur with a single connection.The Watcher objectWhen you usewatch_processesor when you domake_processwithwatch_immediately=True, you get back aWatcherobject.You can use watchers manually or with a context manager. Here is an
example without a context manager:class MyLogger(object):
    def __init__(self, watcher):
        self.watcher = watcher

    def save_stuff(self):
        if self.watcher and self.watcher.ready:
            out, err, closed = self.watcher.read()
            ... save it ...
            self.watcher.acknowledge()  # remove it from the buffer

    def close(self):
        self.watcher.close()If you are running your logger in a separate thread anyway, you might
want to just use a context manager, like this:with p.watch_processes('aack') as watcher:
    while watcher:
        out, err, closed = watcher.read()  # block until something arrives
        ... save it ...
        watcher.acknowledge()  # remove it from the bufferTheWatcherobject has afilenomethod, so it can be used withselect.select, like this:watchers = []

watchers.append(p.watch_processes('circus.uwsgi'))
watchers.append(p.watch_processes('circus.nginx'))
watchers.append(p.watch_processes('circus.mongos.*'))

while watchers:
    ready_watchers = select.select(watchers, [], [])[0]  # wait for one of these
    for watcher in ready_watchers:  # iterate through all that are ready
        out, err, closed = watcher.read()
        ... save it ...
        watcher.acknowledge()
        if not watcher:  # if it is done, remove this watcher from the list
            watcher.close()
            del watchers[watcher]Of course, in the above example it would have been even more efficient
to just use a single watcher, like this:with p.watch_processes('circus.uwsgi', 'circus.nginx', 'circus.mongos.*') as watcher:
    while watcher:
        out, err, closed = watcher.read()
        ... save it ...
        # watcher.acknowledge() - no need since watcher.read will do it for usw.readyThis property isTrueif theWatcherhas data available to read
on the socket.w.read()Read will grab all waiting output from theWatcherand return atupleof(out, err, closed). Each of these is an array ofpapa.ProcessOutputobjects. An output object is actually anamedtuplewith 3 values:name,timestamp, anddata.Thenameelement is thenameof the process. Thetimestampis afloatof when the data was captured by the papa kernel. Thedatais a binary string if found in either theoutorerrarray. It is the exit code if found in theclosedarray. Using all
of these elements, you can write proper timestamps into your logs, even
if data was captured by the papa kernel minutes, hours or days earlier.Thereadmethod will block if no data is ready to read. If you do
not want to block, use either thereadyproperty or a mechanism such
asselect.selectbefore callingread.w.acknowledge()Just because your have read output from a process, the papa kernel
cannot know that you successfully logged it. Maybe you crashed or were
shutdown before you had the chance. So the papa kernel will hold onto
the data until you acknowledge receipt. This can be done either by
callingacknowledge, or by doing a subsequentreador aclose.w.close()When you are done with aWatcher, be sure to close it. That will
release the socket and potentially even return the socket back to the
originalPapaobject. It will also send off a finalacknowledgeif necessary.If you use a context manager, theclosehappens automatically.if watcher:A boolean check on theWatcherobject will returnTrueif it is
still active andFalseif it has received and acknowledged a close
message from all processes it is monitoring.WARNING: There should be only oneYou will get very screwy results if you have multiple watchers for the
same process. Each will get the available data, then acknowledge receipt
at some point, removing that data from the queue. Based on timing, both
will get overlapping results, but neither is likely to get everything.Shutting DownPapa is meant to be a long-lived process and it is meant to be usable by
multiple client apps. If you would like to shut Papa down, you can tryp.exit_if_idle(). This call will only exit Papa if there are no
processes, sockets or values. So if your app cleaned everything up and
no other app is using Papa,exit_if_idlewill allow Papa to die. It
will returnTrueif Papa has indicated that it will exit when the
connection closes.If you want to do a complete cleanup, kill all of your processes however
you like, then do:p.remove_processes('myapp.*')
p.remove_sockets('myapp.*')
p.remove_values('myapp.*')
if p.exit_if_idle():
    print('Papa says it will shutdown!')WARNING: If another process connects to Papa before the connection is
closed, Papa will remain open. Theexit_if_idlecommand will drop
the connection if it returns True so this is a very narrow window of
opportunity for failure.