A Domain-Specific Interpreter for
Parallelising a Large Mixed-Language
Visualisation Application
Karen Osmond, Olav Beckmann, Anthony J. Field and Paul H. J. Kelly
Department of Computing, Imperial College London,
180 Queen’s Gate, London SW7 2AZ, United Kingdom

http://www.doc.ic.ac.uk/˜ob3

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 1/20

MayaVi
Modular Visualisation Environment

Visualising Large Ocean Current Simulations

Graphical interface forcomposing analysis andrendering components
22,000 LOC, Python + VTKImperial CollegeLondon

Open source, active
development
Poor interactive performance
limits usefulness
A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 2/20

Python/VTK Visualisation Software Architecture
Visualisation typically involves a pipeline of feature-extraction operations
When working on extremely large datasets, response time for interactive
parameterisation of the visualisation pipeline is poor.
The challenge is to make visualisation of large datasets interactive byimproving use of memory hierarchy and parallelisation
Application or Script
written in Python, interpreted
Python VTK Bindings
VTKwritten in C++, compiled
OpenGL / XGL etc.

Multi-language: Python, C++, C
Component-based
Actively changing code base,maintained by people who haveno time for parallelisation
Mixed dynamic / static
Domain-specific semantics inDSL (VTK)

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 3/20

Object-Oriented Visualisation in VTKVTK Visualisation Pipeline

Graphics Model:
Object-oriented representation of 3D
computer graphics
Visualisation Model
Model of data flow.
Capable of representing
complex data-flow graphs:“visualisation pipelines”
Data-flow graphs can be
executed in a demand-driven ordata-driven manner.
Surprisingly similar to high-levelcompositional programmingmodels.

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 4/20

Domain-Specific Libraries: Typical Use
Program compiled withstandard compiler (gcc, icc, . . . )or interpreted with standard
interpreter (e.g. python).

vtkContourFilter...
vtkPolyDataMapper...
vtkActor...
...

DSL code mixed with othercode.

Renderuser-program
Domain-specific libararyNo domain-specificoptimisation.

Using such DSLs often dominates and constrains the way a softwaresystem is built just as much as a programming language.
Compiling a quasi domain-specific language without a domain-specificcompiler or optimiser.
Typically miss out on cross-component optimisation opportunities thatexploit the domain-specific semantics of the library.

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 5/20

Domain-Specific Interpreter Pattern
vtkContourFilter
vtkPolyDataMapper

vtkContourFilter

vtkActorRender

. . .
. . .
vtkActor

. . .
. . .
. . .
RenderCapturevtkPolyDataMapper

Optimisefor [all processors, per chunk]vtkContourFilter
vtkPloyDataMapper
vtkActor
Renderend

user-program
Domain-specific libarary

User program is unmodified and is compiled with or interpreted byunmodified language compiler or interpreter.
Capture all calls to methods from a DSL.
Apply domain-specific optimisation, then call the underlying library.

Imperial College
LondonA Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 6/20Domain-Specific Interpreter Pattern
vtkContourFilter
vtkPolyDataMapper

vtkContourFilter

vtkActor
Render. . .
. . .
vtkActor

. . .
. . .
. . .

CapturevtkPolyDataMapper

RenderOptimisefor [all processors, per chunk]vtkContourFilter
vtkPloyDataMapper
vtkActor
Renderend

user-program
Domain-specific libarary

Applicability (Requirements)

Profitability

Reliable capture
VTK/Python bindings

Domain-specific semantics
Piecewise evaluation valid

Reliable capture of data-flow
through DSL routines.
Opaque VTK data structures

Opportunities for optimisations
across method calls
Size of intermediate data

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 7/20

Domain-Specific Interpreter for VTK in Python
mv vtkpython.py vtkpython_real.py then vtkpython.py:
2

if ("vtkdsi" in os.environ): # Control DS Interpreter via Environment

3

import vtkpython_real # Original vtkpython.py re−named

4

from vtkdsi import proxyObject

5for className in dir(vtkpython_real): # For all classes in this module

678

exec "class " + className + "(proxyObject): pass" # class with no methods (yet)else:from vtkpython_real import ∗ # fall−through to original VTK Python

For all classes from vtkpython_real.py, create a class by the same
name, with no methods, derived from proxyObject.
Explicit hooks for capturing all field and method accesses (cf. AOP)
176
253
256

class proxyObject:
def __getattr__(self, callName): # Catch−all method
return lambda ∗callArgs: self.proxyCall(callName, callArgs) # lambda call

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 8/20

Visualisation Recipes
The scheme we showed on the previous slide works lazily for all calls
through VTK Python interface
We need to identify force points (i.e. Render()).
Lazy indirection causes Python’s reflection mechanism to break;therefore we actually use a more eager scheme.
The proxy stores all calls made to VTK in a visualisation recipe.
When a force point is reached, the recipes are evaluated.
1[’construct’, ’vtkConeSource’, ’vtkConeSource_913’]

2[’callMeth’, ’vtkConeSource_913’, ’return_926’, ’SetRadius’, ’0.2’]

3[’callMeth’, ’vtkConeSource_913’, ’return_927’, ’GetOutput’, ’’]4[’callMeth’, ’vtkTransformFilter_918’, ’return_928’, ’SetInput’, "self.ids[’return_927’]"]5[’callMeth’, ’vtkTransformFilter_918’, ’return_929’, ’GetTransform’, ’’]6[’callMeth’, ’return_929’, ’return_930’, ’Identity’, ’’]Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 9/20

Optimising VTK Visualisation Pipelines
Simulations generating the datasets we are visualising are run inparallel, resulting in a parallel tetrahedral VTK data set.
This means: XML file giving locations of partitions
Normally, VTK fuses the partitions into one whole dataset.
If a dataset has not been generated as a collection of partitions, wecan use METIS to create a partitioned version.
VTK does have parallel routines — data-parallel using MPI.
We are interested in a more dynamic scenario, steered from a client.
SPMD
Data-Parallel

Render

P1

P2

P3

RenderRenderRenderRenderClientP4

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 10/20

Coarse-Grained Tiling of VTK Visualisation Pipelines
RenderRender

RenderRender

ClientOur domain-specificvtkpython interpreter builds adata-structure representing thesequence of operations
performed.
When the user-application calls
Render(), we apply thispartition-by-partition on thedata-set.

Large intermediate data meansthat multi-stage visualisation
pipelines make poor use ofmemory hierarchy.

Imperial CollegeLondon

The only difference is anenvironment variable.
Domain-specific semantics
determine the validity of thistransformation.

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 11/20

Coarse-Grained Tiling of VTK Visualisation Pipelines

Calculating isosurfaces one partition at a time, showing outlines of partitions.

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 12/20

Shared-Memory Parallelisation
The first obstacle is that Pythoninterpreter is not thread-safe!

RenderPython Global Interpreter Lock

This can be overcome bymanually lifting the GIL(global interpreter lock) onthe C++ side.
Some VTK routines are also notthread-safe, or do not haveparallel semantics.
Rendering via OpenGL is notthread-safe.

P1

P2

P3

P4

Plan: execute the visualisation
pipelines for each tile in parallelon an SMP.

Imperial CollegeLondon

So we do not lift the GIL
when calling C++-side
rendering from Python.

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 13/20

Distributed Memory Parallelisation
Use a cluster of machines to perform the calculation in parallel and thenrender on one client machine.
Used Python library Pyro to provide RMI-like features for Python.

Pyro allows ’pickleable’ (serialisable) objects to be transferred overthe network.
Our recipes can be transferred to servers in the cluster in this way.
Unfortunately, VTK objects cannot be serialised using the ‘pickle’mechanism.
Therefore use a shared filesystem to transfer VTK objects.
This is a dynamic, client-server model of distributed memory
parallelisation, not data-parallel.

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 14/20

EvaluationBenchmark Scenario
Open a dataset
representing flow overheated sphere
Plot seven isosurfaces atdifferent values
Platforms
Athlon 1600+, dual SMP,
256 KB L2, 1GB RAM,
Linux 2.4
Cluster of 4 Pentium 4 2.8
GHz, 512 KB L2, 1GB
RAM, Linux 2.4
Size: Approx 325 MB

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 15/20

Results
for IsoSurface
Benchmark
Resultsfor IsoSurface
Benchmark
IsoSurface Benchmark: Athlon 2-way SMP 1600+
70

Time in Seconds

605040

BaseTilingSMP302010024816

Number of Partitions

Benchmark consists of loading a dataset representing flow over heatedsphere and calculating 7 isosurfaces at different values.

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 16/20

Results for IsoSurface Benchmark
IsoSurface Benchmark:Cluster of Pentium 4 2.0 GHz
504540Time in Seconds

3530
BaseTiling
DMP25

2015

105024816

Number of Partitions

Benchmark consists of loading a dataset representing flow over heatedsphere and calculating 7 isosurfaces at different values.

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 17/20

Related Work
Dynamic component assembly software architectures
SciRun / BioPSE / Uintah“Virtual data-grid” projects
Dynamic cross-component optimisation
Telescoping languages (Rice, LLNL)
Code generation approaches
Kitware tool: Paraview
This makes use of VTK’s data-parallel routines, relies on MPI
Grid workflow engines
Related in that we assemble a workflow at runtime, then execute
Our work illustrates an interesting pathway for facilitating “legacycode” to operate in such an environment.

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 18/20

What’s Next
Use of metadata to carry additional domain-specific semantics
Parallel semantics, thread-safe on C++ side, use-def equations
Reducing the size of the polygon set before rendering
Cache full sets on servers, return decimated sets to client

full80%85%

Level of detail (LOD) and region-of-interest (ROI) selection
Multiple time stepsSpeculatively applying the recipe to future timesteps
Aim to achieve smooth rendering of a series of timesteps

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 19/20

Conclusion
We have parallelised a large, open-source visualisation application
without changing a single line of code.
Entirely transparent to application program, controlled via anenvironment variable.
Works for any Python visualisation script using VTK.
Use Python to implement a domain-specific interpreter for adomain-specific library
Facilitated by reliable capture of DSL calls and known data-flow due
to opaque objects on the C++ side.
Optimisations
Coarse-grained tiling
SMP parallelisation
Distributed memory parallelisation
Dynamic, runtime parallelisation

Imperial CollegeLondon

A Domain-Specific Interpreter for Parallelising a Large
Mixed-Language Visualisation Application— 20/20