1
日本ソフトウェア科学会第 37 回大会 (2020 年度) 講演論文集

Sanajeh: A DSL for GPGPU programming withPython objects
Jizhe Chenxin Hidehiko Masuhara Matthias Springer Youyou CongGPGPU (general purpose computing on graphics processing units) is one of the economical methods ofparallel programming. However, in order to obtain high performance, the programmers must write code ina low-level programming language such as C and pay attention to memory allocation. We propose Sanajeh,
a Python DSL (domain-speciﬁc language) that compiles object-oriented programs into GPGPU code. It is alanguage which is based on the Single-Method Multiple-Objects (SMMO) model. Sanijeh compiles parallel
Python code into C++/CUDA code and utilizes DynaSOAr for eﬃcient GPU memory allocation.

1 Introduction
GPGPU is a method that uses GPUs (graphics processing units) to perform computations thatare usually performed by CPUs (central processing
units). Although a single core on a GPU operates
at a lower frequency than a CPU does, by utilizing the computation power of multiple GPU cores,
programmers can gain high performance in parallel
programs in a more economical way.
Object-oriented programming (OOP) is a widelyused programming paradigm but it is seen as
too ineﬃcient for high-performance computing(HPC). Recent work has shown that eﬃcient OOP
is feasible on GPUs when following the SingleMethod Multiple-Objects (SMMO) [11] programming model. In SMMO, programs are designed
to run one method on all objects of a class. It
ﬁts well with parallel Single-Instruction MultipleData (SIMD) architectures. SMMO model allows programmers to write eﬃcient parallel programs in OOP style. The performance of many
SMMO applications depends on eﬃcient memory
Sanajeh: Python オブジェクトを用いる GPGPU プロ
グラミングのためのドメイン固有言語
Jizhe Chenxin, Hidehiko Masuhara, Youyou Cong, 東京工業大学数理・計算科学コース, Dept. of Mathematical and Computing Science, Tokyo Institute ofTechnology.
Matthias Springer, グーグル合同会社, Google Japan
G.K..

(de)allocation.
In this paper, we propose a Domain Speciﬁc
Language (DSL) called Sanajeh. Sanajeh provides API (section 4) for programmers to write
SMMO applications with high performance in normal Python syntax. Programmers do not have tobother with GPU memory allocation, which is difﬁcult implement eﬃciently. Also, Sanajeh allows
users to use Python libraries in HPC programs. We
evaluated Sanajeh’s performance with an N-Body
simulation (section 7).

2 Background2. 1 DynaSOAr
DynaSOAr [12] is an eﬃcient GPU memory allocator for SMMO applications. It stores data in a
Structure of Arrays (SOA) memory layout to improve the eﬃciency of memory access. However,programmers must adhere to a certain coding styleand follow certain coding conventions when using
DynaSOAr.
• Pre-declarations for ﬁelds:
DynaSOAr requires user to pre-declare all theﬁelds of a class that resides on the device (wecall these classes “device classes”) through aspecial syntax (Listing 1). Such code is diﬀerent from ordinary C++ (Listing 2) and harder
to read. Also, all device classes have to be childclasses of “AllocatorT::Base”.
• Speciﬁcation for device data:日本ソフトウェア科学会第 37 回大会 (2020 年度) 講演論文集2

c l a s s Body : public A l l o c a t o r T : : Base {public :d e c l a r e f i e l d t y p e s ( Body , f l o a t ,float , float , float , float ,float , float )private :F i e l d <Body , 0> p o s x ;F i e l d <Body , 1> p o s y ;F i e l d <Body , 2> v e l x ;F i e l d <Body , 3> v e l y ;F i e l d <Body , 4> f o r c e x ;F i e l d <Body , 5> f o r c e y ;F i e l d <Body , 6> mass ;}device void Body : : a p p l y f o r c e ( Body∗
other ) {// Update ‘ o t h e r ‘ .
i f ( o t h e r != t h i s ) {f l o a t dx = p o s x − o t h e r −>p o s x ;f l o a t dy = p o s y − o t h e r −>p o s y ;f l o a t d i s t = s q r t ( dx∗dx + dy∗dy ) ;f l o a t F = k G r a v i t y C o n s t a n t ∗ mass ∗o t h e r −>mass/ ( dist ∗ dist +kDampeningFactor ) ;o t h e r −>f o r c e x += F∗dx / d i s t ;o t h e r −>f o r c e y += F∗dy / d i s t ;}}Listing 1

Field pre-declarations (DynaSOAr)
device void Body : : update ( ) {v e l x += f o r c e x ∗kDt / mass ;v e l y += f o r c e y ∗kDt / mass ;p o s x += v e l x ∗kDt ;p o s y += v e l y ∗kDt ;in an N-Body simulationc l a s s Body : public A l l o c a t o r T : : Base {private :float pos x ;float pos y ;float vel x ;float vel y ;float force x ;float force y ;f l o a t mass ;}Listing 2

Ordinary C++ ﬁeld declarations(equiv. to Fig. 1)

Since DynaSOAr is a CUDA framework,
programmers must annotate functions andcertain ﬁelds with “ device ” and/or “host ” modiﬁers, indicating whether code/data should run/reside on the GPU or on theCPU.
We use DynaSOAr as our device memory allocator.
2. 2 Host and device
In GPGPU programs, code is separated into twoparts: Host code and Device code. Host code
run on the host (CPU), includes code that invokes
the GPU kernel, while device code run on the device (GPU).
In CUDA programs, users have to write“ device ” keywords (Listing 3). These keywords
marked as red are required for data on the device.

i f ( p o s x < −1 | | p o s xv e l x = −v e l x ;}> 1) {i f ( p o s y < −1 | | p o s yv e l y = −v e l y ;}> 1) {}
Listing 3“ device ” keywords before

function declarations in an N-Body simulation.

2. 3 Foreign function interface
Foreign function interface is a mechanism used tocall functions written in another language. Sanajeh uses CFFI [9], which is a Python library, tocall C++ function from Python. We choose the“in-line”, “ABI mode” provided by the CFFI library. Listing 4 shows a example of using CFFI.
Declarations like function prototypes are written in“C − likedeclarations”, and the path of the compiled shared library (in Sanajeh, a “.so” ﬁle) is written in “libpath”. By this, users can call C++ function directly through “lib.f unction name”.
Sanajeh ﬁrst compiles Python device code into
C++/CUDA code, then the code is called throughCFFI. Details for using CFFI are hidden by Sanajeh
API (section 4) so users does not need to write suchcode.

日本ソフトウェア科学会第 37 回大会 (2020 年度) 講演論文集
import c f f i
f f i = c f f i . FFI ( )f f i . c d e f ( ”C−l i k e d e c l a r a t i o n s ” )l i b = f f i . dlopen ( ” l i b p a t h ” )Listing 4

Example of using CFFI, “in-line”.
“ABI mode”

3 Overview
In this section, we discuss two challenges in Sanajeh.
3. 1 Distinct host code and device code
In Sanajeh, code also includes host code anddevice code (2. 2). Host code in Sanajeh is executed directly by the Python interpreter, but device code needs to be compiled into C++/CUDA(3. 2). Sanajeh is designed to free users from explicitly specifying device data by writing code withextra syntax like the ” device ” keywords in Listing 3. The recognition of device code is an issue inSanajeh.
We use CallGraphAnalyzer to solve this issue.
CallGraphAnalyzer tracks all code related to device classes and marks them as device code. It
only requires the speciﬁcations of device classes.
CallGraphAnalyzer will be described in section 5.
3. 2 Compile device code
Python is a highly abstracted language. In orderto gain high performance in SMMO applications,
we have to use a low-level programming language.
In Sanajeh, device code is compiled into
C++/CUDA by Ahead-Of-Time compilation. The
compiled code includes DynaSOAr syntax for calling DynaSOAr API to do memory allocation. As
shown in Listing 1 and Listing 2, DynaSOAr’s syntax is diﬀerent from those in usual programmingstyle. Fields in device classes need to be predeclared through DynaSOAr syntax, which is deﬁned using C++ templates.
It is obvious that DynaSOAr’s syntax makes programs more complex. Sanajeh users can write device code in normal programming style in Python.
We uses a compiler called Py2Cpp to do the compilation from Python to C++/CUDA. It is described3

in section 6. We then compiles those C++/CUDA
code generated by Py2Cpp into a shard library using the Nvidia CUDA Compiler (NVCC), and calls
them through FFI (section 2. 3)

4 Sanajeh’s API
Sanajeh provides API separately for device codeand host code.
4. 1 Device API
Device code is not run by the Python interpreter, instead they are compiled into according
C++/CUDA code by Py2Cpp. Therefore API for
device code is only used for recognizing purpose.
The current API for device code is shown below:• device do(cls, func, *args): Run function
func on all objects of class cls sequentiallywith the arguments of func. Here func has tobe a function declared in the device classes.
• Random API: Random API is provided
for random number generation, for example
the rand init function and the rand uniformfunction. Corresponding to the cuRAND library.
• Math API: Math API is provided for Mathcomputation, for example the sqrt function.
Corresponding to the CUDA Math API.
4. 2 Host API
Host code is executed by the Python interpreter.
Sanajeh provides the following host API:• initialize(): Load the above shared library
to Python FFI module.
• device class(*clss): Specify the deviceclasses through variadic arguments *clss.
• parallel do(cls, func, *args): Run function func on all objects of class cls parallellywith the arguments of func. Here func has
to be a function declared in the device classes.
Notice that there is no parallel do API in device API (section 4. 1), since device code is executed in units of one thread and parallelism
cannot be applied further more.
• parallel new(cls, object num): Create objects of a class cls on device memory with anumber of object num.
• do all(cls, func): Run function func on
copies of all objects of class cls sequentially.

4

日本ソフトウェア科学会第 37 回大会 (2020 年度) 講演論文集

Here func is a function which receives an object as an argument. The object is a copy of an
object of class cls. Functions like print canbe used in func.

5 CallGraphAnalyzer
CallGraphAnalyzer is designed to free users of
Sanajeh from explicitly specifying all host code anddevice code. It uses the abstract syntax tree (AST)of Python source code as input, tracks calling relationships between functions and using of variables,
then creates a CallGraph data structure (section
5. 2). After user speciﬁes the device classes, byanalyzing the CallGraph, all device code will bemarked.
5. 1 Deﬁnition of Sanajeh’s device code
Here are the deﬁnitions of Sanajeh device code:
1. Declarations of the device classes is device
code. These are speciﬁed by users.
2. Declarations for the functions and variables
called/used in those classes are device code.
3. Declarations for the functions and variables
called/used in above functions are device code.
This is applied recursively.
5. 2 CallGraph data structure
CallGraph is a data structure which has threekinds of nodes: ClassNode, FunctionNode andVariableNode:• ClassNode: Represents a class. There are
four sets in this node, each stores functions declared in that class, functions called in thatclass, variables declared in that class and variable used in that class. Each element inthese sets is stored as either FunctionNodeor VariableNode. Sanajeh does not support
nested class.
• FunctionNode: Represents a function. Similar to ClassNode, FunctionNode has sets tostores variables declared and used in the function. Sanajeh does not support nested functions too.
• VariableNode: Represents a variable. Nameand type of the variable is stored in
VariableNode. Notice a global variable can not
be used both in device and host code.
Fig.1 showed an example of the CallGraph structure for the code in Listing 5. For illustrationg l o b a l 1 : int = 5
g l o b a l 2 : i n t = 10g l o b a l 3 : i n t = 15class HostClass :def H o s t F u n c t i o n ( s e l f ) −> None :Function3 ( )class DeviceClass :def D e v i c e F u n c t i o n ( s e l f ) −> None :s e l f . f i e l d 1 = global1 + 1
s e l f . f i e l d 2 = Function1 ( )def F u n c t i o n 1 ( ) −> i n t :Function2 ( )return 0
def F u n c t i o n 2 ( ) −> None :global g l o b a l 3
local1 = 0
g l o b a l 3 += l o c a l 1
def F u n c t i o n 3 ( ) −> None :local2 = 1
global g l o b a l 2
g l o b a l 2 += l o c a l 2
Listing 5

A Sanajeh sample program

purpose we do not mark all calling/using relationships between nodes but only those between device
nodes.
5. 3 Marking of device data
After the CallGraph structure is created, devices
code will be marked by tracking the declaration,
calling and using relationships between the nodes.
As shown in Fig.1, since B is a device class, bytracking the declarations in class B: B F, field1 andfield1 are marked. During the marking, functions
called and variables used in those nodes are also
marked. In this example, Function1 is called in
B F so it was marked. Recursively, Function2 ismarked. Then since local1 is declared and usedin Function2, global variable global3 is used in
Function2 , they are marked as well. global1 isalso marked because it was used in B F.

6 Py2Cpp
Py2Cpp is a compiler which compiles Python device code to C++/CUDA code. It uses the check
result of CallGraphAnalyzer (section 5) to trans-

日本ソフトウェア科学会第 37 回大会 (2020 年度) 講演論文集

HostClassglobal1

HostFunction

DeviceFuntion

DeviceClass

ﬁeld1

Function2

ﬁeld2
Class

local1

Function1

5

local2Function3

Function

Variable

Declaration
global2

global3

Calling/Using

Fig. 1

CallGraph of Listing 5

form the Python AST of device code to C++ AST
code (section 6. 1). Then C++ code will be buildedfrom C++ AST (section 6. 2)
6. 1 Transform Python AST to C++ AST
The traversing uses the Python ast library. The
library provides is designed following Visitor Pattern. There is a visitor function for every kind of
node in the AST. Behavior when visiting the node
is deﬁned in it. Py2Cpp checks the mark resultsof CallGraphAnalyzer when visiting class, function
and variable nodes. If the related CallGraph nodeof a Python AST node is marked as device, all childnode of that node will be transformed into C++
AST nodes. Finally a C++ AST of all device code
is constructed.
Data types is an important issue during thetransformation. Type annotations are supported
from Python 3.6. Although Python runtime does
not enforce function and variable type annotations,since type information is important for memory allocation, Sanajeh requires users to explicitly write
type hints for device data (as shown in Listing 5).
There is a type converter in Py2Cpp. For now it only
supports bool, int and float data types, but weare going to extend it to support other data types
in the future, for example arrays.

6. 2 Build C++ code
After C++ AST is generated, C++ code will be
generated from the C++ AST. We generate two
ﬁles from C++ AST, one is “.h” header ﬁle and another is “.cu” source ﬁle. The “ device ” keyword
and other syntax will be add during this process.
Other syntax includes:• Pre-declarations of ﬁelds:
Pre-declarations of ﬁelds are generated fromclass AST nodes in C++ AST. code in Listing 1 can be generated automatically.
• code for API callings:
API calling in DynaSOAr uses C++ templates
to pass argument like class and function. Although Python has mechanism to pass classand function as an argument, templates cannot be called through FFI (section 2. 3) except
for functions. We generate a function for eachclass. The function includes C++ template,which is used to call DynaSOAr API. For example, Listing 6 shows this the function for
calling parallel new on class Body.
• code for callback:
In Sanajeh’s API (section 4), there is a do allfunction. This function receives a Python function, say PF, while PF receives an Python object as a parameter. The constructor functionof the class, say CF, will be included together

日本ソフトウェア科学会第 37 回大会 (2020 年度) 講演論文集

6

extern ”C” i n t p a r a l l e l n e w B o d y ( i n t
object num ) {a l l o c a t o r h a n d l e −>p a r a l l e l n e w <Body>( o b j e c t n u m ) ;return 0 ;}Listing 6

Function for calling parallel new onclass Body

void Body : : d o ( void ( ∗ l ) ( f l o a t , f l o a t ,float , float , float , float , float ) ) {l ( this−>po s x , this−>p o s y , this−>v e l x , this−>v e l y , this−>f o r c e x , this−>f o r c e y , this−>mass ) ;}extern ”C” i n t B o d y d o a l l ( void ( ∗ l ) (float , float , float , float , float ,float , float ) ) {a l l o c a t o r h a n d l e −>templated e v i c e d o <Body>(&Body : : do ,l);return 0 ;}Listing 7

do and do all function of class Body

with PF in such an lambda expression:
lambda{∗args} : P F (CF ({∗args})){∗args} is the ﬁelds of the device class. This
lambda expression, say lexp, is passed to C++
as a callback function. In c++, there is ado all function for every class that receives
lexp, and passes it to another function do using DynaSOAr API. In do function, all ﬁeldsof the class is passed to lexp. As a result, lexp
is called multiple times in C++ territory until copies of all device objects are created by CF
and passed to F. Finally, the ﬁelds of those copy
objects are accessed in F. Example code for do
and do all is shown in Listing 7. Py2Cpp collects the information of ﬁelds of device classes
then generates such code automatically.

7 Performance Evaluation
To see how calling device code through FFI module aﬀects overall performance, we evaluated theperformance of Sanajeh compared to DynaSOAr by
an N-Body simulation program.
N-Body is a 2D particle simulation. It simulates
movements of a large number of bodies. Each bodyhas position, velocity and mass. Every iteration of
the simulation computes force between every twobodies according to Newton s theory of gravity,
by advancing the simulation by a small period oftime, the new velocity and position of the bodies
are updated accordingly.
We measured the overall execution time of NBody simulation with 100 iterations for diﬀerent
number of objects in Sanajeh and DynaSOAr on
NVIDIA TITAN Xp GPU (12 GB device memory).
We compiled the device code with NVCC (-O3)
from the CUDA Toolkit 10.1 on Ubuntu 18.04.4.
Host code was run with Python 3.7.4. Fig.2 shows
the result of the execution time. The vertical axis isexecution time in seconds, and the horizontal axis
is the number of bodies.
In conclusion, the execution time of each iteration in Sanajeh and DynaSOAr has no much diﬀerence. The increase percentage of execution timefrom Sanajeh’s to DynaSOAr’s is no more than
1.5%. It is conceivable that this extra time is theexecution time of FFI module. It does not inﬂuencemuch on overall performance.
We measured the compile time spent by Py2Cpp.
Py2Cpp spent an average time of 0.006s to compile
N-Body’s Python code into C++/CUDA ones. It
will take up less and less up proportion of overall
execution time when body objects increases.

8 Related work
There have been high-level DSLs for GPGPU
programming. Such as SPOC [2], which uses
stream processing with Ocaml, and Ikra [5], which
is a data-parallel extension to Ruby. In Python,
Klöckner et al. designed PyCUDA and PyOpenCL[4], two Python libraries for GPU computing basedon CUDA. They are designed for experienced
CUDA users and require users to write code in
CUDA syntax as a string. This is not friendly
to a Python programmer who doe not understand
C++/CUDA syntax. Another library for GPU
computing is called CUPY [6]. Its syntax is close to
NumPy [7], a widely-used Python library for data
computing. It provides the same functionality as日本ソフトウェア科学会第 37 回大会 (2020 年度) 講演論文集

Fig. 2

7

Execution time of N-Body simulation with 100 iterations

NumPy, but executes on a GPU.
Although the above libraries have good performance in most of the cases in GPGPU programming by Python, they are not designed and optimized for OOP. In Python, a list of objects isstored as an Array of Structures (AOS) [3] [8]. Previous works [1] [10] [13] have shown that switching toa Structure of Arrays (SOA) data layout can speed
up HPC applications by several factors compared toa traditional AOS layout. Therefore, Python’s implementation of a list of objects is not a good choicefor GPU programs. Our work uses DynaSOAr [12]as memory allocator, which allocates memory dynamically with SoA performance characteristics.

9 Conclusion
We proposed a DSL for GPGPU programmingwith Python objects called Sanajeh. Sanajeh provides API for users to write SMMO applications.
By compiling device code to C++/CUDA code
automatically, Sanajeh users can write GPGPU
in OOP style using a high abstracted language,
Python, in the meanwhile without losing performance.

References[ 1 ] Besl, P.: A case study comparing AoS (Arraysof Structures) and SoA (Structures of Arrays) datalayouts for a compute-intensive loop run on Intel
Xeon processors and Intel Xeon Phi product family
coprocessors, Intel Article, Vol. 392271(2015).
[ 2 ] Bourgoin, M., Chailloux, E., and Lamotte, J.L.: SPOC: GPGPU programming through stream
processing with OCaml, Parallel Processing Letters,Vol. 22, No. 02(2012), pp. 1240007.
[ 3 ] Goodrich, M. T., Tamassia, R., and Goldwasser,
M. H.: Data structures and algorithms in Python,Wiley Hoboken, 2013.
[ 4 ] Klöckner, A., Pinto, N., Lee, Y., Catanzaro,
B., Ivanov, P., and Fasih, A.: PyCUDA and PyOpenCL: A scripting-based approach to GPU runtime code generation, Parallel Computing, Vol. 38,
No. 3(2012), pp. 157–174.
[ 5 ] Masuhara, H. and Nishiguchi, Y.: A dataparallel extension to Ruby for GPGPU: toward aframework for implementing domain-speciﬁc optimizations, Proceedings of the 9th ECOOP Workshop on Reﬂection, AOP, and Meta-Data for Software Evolution, 2012, pp. 3–6.
[ 6 ] Nishino, R. and Loomis, S. H. C.: Cupy: A
numpy-compatible library for nvidia gpu calculations, 31st confernce on neural information processing systems, (2017), pp. 151.
[ 7 ] Oliphant, T. E.: A guide to NumPy, Vol. 1, Trelgol Publishing USA, 2006.
[ 8 ] Phillips, D.: Python 3 object-oriented program-8

日本ソフトウェア科学会第 37 回大会 (2020 年度) 講演論文集
ming, Packt Publishing Ltd, 2015.
(2019).
[ 9 ] Rigo, A. and Fijalkowski, M.: CFFI documen[12] Springer, M. and Masuhara, H.: DynaSOAr: A
tation, 2012.
Parallel Memory Allocator for Object-Oriented Pro[10] Springer, M. and Masuhara, H.: Ikra-Cpp: A
gramming on GPUs with Eﬃcient Memory Access,
C++/CUDA DSL for Object-Oriented ProgramarXiv preprint arXiv:1810.11765, (2018).
ming with Structure-of-Arrays Layout (WPMVP[13] Strzodka, R.: Abstraction for AoS and SoA lay18), ACM, Article, Vol. 6, No. 9(2018).
out in C++, GPU computing gems Jade edition,[11] Springer, M.: Memory-Eﬃcient Object-Oriented
Elsevier, 2012, pp. 429–441.
Programming on GPUs, arXiv preprint arXiv:1908.05845,