Package: marcel
What's NewThis release contains a major new feature: workspaces. Aworkspacecontains a startup script, a command history, and environment
variables, separated from those of other workspaces.Startup script:The startup script has always been a convenient
place to import useful modules, define databases and remote access,
and define useful functions and variables. By providing a startup
script for each workspace, you don't have to put everything, for all
projects, into one startup script.Command history:Marcel has always had a persistent command
history, so the only novelty is the separation by workspace. For
example, you might have a workspace per project that you're working
on, and each project would then have its own command history.Environment variables:Environment variables in workspace
persist. So any variables defined from the command line are saved, and
when you next use marcel with the same workspace, your variables are
still there.Example:Marcel starts in the default workspace, which has identical behavior
to that of marcel in previous versions. All workspace operations can
be accomplished using thewsoperator. With no argumentswsreturns the current workspace.M0.20.0jao@loon~$ws
Workspace()Workspace()denotes the default workspace.If I import a module while in the workspace, it stays imported, even if I exit marcel,
restart it, and join the workspace again:# import the math module into the a6 workspaceM0.20.0(a6)jao@loon~$importmath
M0.20.0(a6)jao@loon~$(math.pi)3.141592653589793# Exit and then restart marcel, and go back to the a6 workspaceM0.20.0(a6)jao@loon~$exitjao@loon:~$marcel
M0.20.0jao@loon~$wsa6
Workspace(a6)# math is still importedM0.20.0(a6)jao@loon~$(math.pi)3.141592653589793All symbols, not just imported ones are scoped to the workspace, and are persistent.
For example, a persistent pipeline:# Define a pipeline to find all files under a directory, and compute the# sum of file sizes with a given extension.M0.20.0(a6)jao@loon~$extspace=(|dir,ext:\M+$ls-fr(dir)\M+$|select(f:f.suffix==ext)\M+$|map(f:f.size)\M+$|red+\M+$|)# Try it outM0.20.0(a6)jao@loon~$extspacegit/marcel.py43637344# Exit marcel, restart it, and go back to the workspaceM0.20.0(a6)jao@loon~$exitjao@loon:~$marcel
M0.20.0jao@loon~$wsa6
Workspace(a6)# Try the pipeline previously defined.M0.20.0(a6)jao@loon~$extspacegit/marcel.py43637344MarcelMarcel is a shell.
The main idea is to rely on piping as the primary
means of composition, as with any Unix or Linux
shell. However, instead of passing strings from one command to the
next, marcel passes Python values: builtin types such as lists,
tuples, strings, and numbers; but also objects representing files and
processes.Linux has extremely powerful commands such asawkandfind.  Most
people know how to do a few simple operations using these commands.
But it is not easy to exploit the full power of these commands
due to their reliance on extensive "sublanguages" which do:Filtering: What data is of interest?Processing: What should be done with the data?Formatting: How should results be presented?By contrast, marcel has no sublanguages.  You use marcel operators
combined with Python code to filter data, process it, and control
command output.The commands and syntax supported by a shell constitute a language
which can be used to create scripts. Of course, in creating a script,
you rely on language features that you typically do not use
interactively: control structures, data types, and abstraction
mechanisms (e.g. functions), for example.
Viewed as a programming language, shell scripting languages
are notoriously bad. I didn't think it was wise to bring another one
into the world. So marcel takes a different
approach, using Python as a scripting language, (see below for more
on scripting).PipelinesMarcel provides commands, calledoperators, which do the basic work of a shell.
An operator takes astreamof data as input, and generates another stream as output.
Operators can be combined by pipes, causing one operator's output to be the next operator's input.
For example, this command uses thelsandmapoperators to list the
names and sizes of files in the/home/jaodirectory:ls/home/jao|map(lambdaf:(f,f.size))Thelsoperator produces a stream ofFileobjects, representing the contents
of the/home/jaodirectory.|is the symbol denoting a pipe, as in any Linux shell.The pipe connects the output stream fromlsto the input stream of the next
operator,map.Themapoperator applies a given function to each element of the input stream,
and writes the output from the function to the output stream. The function is enclosed
in parentheses. It is an ordinary Python function, except that the keywordlambdais optional.
In this case, an incomingFileis mapped to a tuple containing the file and the file's size.Apipelineis a sequence of operators connected by pipes. They can be used directly
on the command line, as above. They also have various other uses in marcel. For example,
a pipeline can be assigned to a variable, essentially defining a new operator.
For example, here is a pipeline, assigned to the variablerecent, which selectsFiles modified within the past day:recent=(|select(file:now()-file.mtime<days(1))|)The pipeline being defined is bracketed by(|...|). (Without the brackets, marcel would
attempt to evaluate the pipeline immediately, and then complain because the parameterfileis not bound.)The pipeline contains a single operator,select, which uses a function to define
the items of interest. In this case,selectoperates on aFile, bound to the
parameterfile.now()is a function defined by marcel which gives the current time in seconds since
the epoch, (i.e., it is justtime.time()).Fileobjects have anmtimeproperty, providing the time since the last content modification.days()is another function defined by marcel, which simply maps days to seconds, i.e.,
it multiplies by 24 * 60 * 60.This pipeline can be used in conjunction with any pipeline yielding files. E.g., to locate
the recently changed files in~/git/myproject:ls~/git/myproject|recentFunctionsAs shown above, a number of operators, likemapandselect, take Python functions as
command-line arguments. Functions can also be invoked to obtain the value of an
environment variable.
For example, to list the contents of your home directory, you could write:ls/home/(USER)This concatenates the string/home/with the string resulting from the evaluation of
the expressionlambda: USER.USERis a marcel environment variable identifying the
current user, (so this command is equivalent tols ~).If you simply want to evaluate a Python expression, you could use themapoperator, e.g.map(5+6)which prints11. Marcel permits themapoperator to be inferred,
so this also works:(5+6)In general, you can elidemapfrom any pipeline.ExecutablesIn addition to using built-in operators, you can, of course, call any executable.
Pipelines may contain a mixture of marcel operators and host executables. Piping between
operators and executables is done via streams of strings.For example, this command combines operators and executables.
It scans/etc/passwdand lists the usernames of
users whose shell is/bin/bash.cat,xargs, andechoare Linux executables.mapandselectare marcel operators.
The output is condensed into one line through
the use ofxargsandecho.cat/etc/passwd\|map(line:line.split(':'))\|select(*line:line[-1]=='/bin/bash')\|map(user,*_:user)\|xargsechocat /etc/passwd: Obtain the contents of the file. Lines are piped to subsequent commands.map (line: line.split(':')): Split the lines at the:separators, yielding 7-tuples.select (*line: line[-1] == '/bin/bash'): select those lines in which the last field is/bin/bash.map (user, *_: user) |: Keep the username field of each input tuple.xargs echo: Combine the incoming usernames into a single line, which is printed tostdout.Shell FeaturesMarcel provides:Command history:Ahistoryoperator, rerunning and editing of previous commands,
reverse search, etc.Customizable prompts:Configured in Python, of course.Tab completion:For operators, flags, and filename arguments.Help:Extensive help facility, providing information on concepts, objects,
and operators.Customizable color highlighting:The colors used to render output for builtin types such
asFileandProcess, andhelpoutput can be customized too.Dynamic reconfiguration:Changes to configuration and startup scripts are picked up without restarting.ScriptingMarcel's syntax for constructing and running pipelines, and defining and using
variables and functions, was designed for interactive usage. Instead of extending
this syntax to a full-fledged scripting language, marcel provides a Python API,
allowing Python to be used as the scripting language.
While Python is
sometimes considered toalready bea scripting language, it isn't really.
Executing shell commands from Python code is cumbersome. You've got to useos.system, orsubprocess.Popen, and write some additional code to
do the integration.Marcel provides a Python module,marcel.api,
which brings shell commands into Python in a much cleaner way. For
example, to list file names and sizes in/home/jao:frommarcel.apiimport*forfile,sizeinls('/home/jao')|map(lambdaf:(f,f.size)):print(f'{file.name}:{size}')This code uses thelsandmapfunctions, provided bymarcel.api. These correspond to the
marcel operatorslsandmapthat you can use on the command
line. Output from thelsis a stream ofFiles, which are piped
tomap, which maps files to (file, file size) tuples.ls ... | map ...defines a pipeline (just as on the command line). The
Python class representing pipelines definesiter, so that
the pipeline's output can be iterated over using the standard
Pythonforloop.InstallationTo install marcel locally (i.e., available only to your username):python3-mpipinstallmarcelThis command installs marcel for the current user. To install for the entire system,
usesudo python3 -m pip install --prefix ...instead. (The value of the--prefixflag should
be something like/usr/local.)Marcel depends ondill. This package
will be installed automatically if needed, when marcel is installed
via pip.