Extending Unix Pipelines to Graphs
Channel: Diomidis Spinellis
Date: 22/08/2017

The Unix shell dgsh provides an expressive way to construct sophisticated and efficient non-linear pipelines. 

You can find the (open-access) full paper at http://dx.doi.org/10.1109/TC.2017.2695447, documentation at https://www.spinellis.gr/sw/dgsh/, and the software at https://github.com/dspinellis/dgsh.

In brief, dgsh pipelines can use standard Unix tools, as well as third-party and custom-built components. Dgsh allows the specification of pipelines that perform non-uniform non-linear processing. These form a directed acyclic process graph, which is typically executed by multiple processor cores, thus increasing the processing task’s throughput. A number of existing Unix tools have been adapted to take advantage of the new shell’s multiple pipe input/output capabilities. The shell supports visualization of the process graphs, which can also aid debugging. Dgsh was evaluated through a number of common data processing and domain-specific examples, and was found to offer an expressive way to specify processing topologies, while also generally increasing processing throughput.