The Real Face of Python — An Introduction
By: Chukwunonso Nwafor
Tags: C, Python, Cpp, Django

updated for Python 3.8.3

by Lukas Blazek — Unsplash

For the sake of performance and reliability, solve interpreted problems in interpreted languages and compiled problems in compiled languages.

Python gives you the power of three languages: an interpreted language Python and two compiled languages C and C++.

When I talk about Python, I mean CPython.

Introduction

It is very difficult (if not impossible) for an interpreted language to beat the execution speed of a compiled language.

One of the selling points of Python is the speed (development speed) at which ideas can be bootstrapped using it. But as web apps are becoming very ambitious in providing solutions with god-like speed, interpreted languages are put on trial. Here enters the Python/C API.

CPython gives you the speed of compiled languages through the Python/C API. With this, performance (if needed, after profiling) can be added based on stepwise refinement of your original solution without altering your source code. This is explained below.

Consider a module first.py that defines a type A that is a critical component of a particular project as shown below:

# module first.py

# A might define additional attributes and methods

class A(dict):

pass

A python developer reasoning in python, uses type A to solve the problem it was intended for, it worked, everybody is happy.

But then your project starts suffering from performance problem, and after profiling you found out that A is the culprit. You can solve the performance problem by stepwise refinement of module first.py with a single import without altering A as shown below.

# new first.py # A might define additional attributes and methods

class A(dict):

pass try:

from _utils import _A as A

except ImportError:

pass

Where _A is defined in an extension module _utilsmodule.c by a python developer reasoning in the C programming language.

Without Further Ado

Everything is an object in python. The CPython implementation sees this objects as variables of the type PyObject*. Each of this objects have a type and a reference count.

Thus we can define _A in _utilsmodule.c as shown below.

#define PY_SSIZE_T_CLEAN

#include <Python.h> /* Our object */

typedef struct {

PyDictObject dict;

} AObject; /* Our Type */

static PyTypeObject AType = {

PyVarObject_HEAD_INIT(NULL, 0)

.tp_name = "_utils._A",

.tp_doc = "_A objects",

.tp_basicsize = sizeof(AObject),

.tp_itemsize = 0,

.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,

}; /* Our Module */

static PyModuleDef _amodule = {

PyModuleDef_HEAD_INIT,

.m_name = "_utils",

.m_doc = "_utils provides efficient data strutures and algorithms",

.m_size = -1,

}; /* Initialize our module */

PyMODINIT_FUNC

PyInit__utils(void)

{

PyObject *m; AType.tp_base = &PyDict_Type;

if (PyType_Ready(&AType) < 0)

return NULL; m = PyModule_Create(&_amodule);

if (m == NULL)

return NULL; Py_INCREF(&AType);

if (PyModule_AddObject(m, "_A", (PyObject *)&AType)) {

Py_DECREF(&AType);

Py_DECREF(m);

return NULL;

}

return m;

}

To build and install we first create a setup.py file as shown below.

# setup.py

from distutils.core import Extension, setup setup(name="UtilsPackage",

version="1.0.0",

author="Nwafor Chukwunonso",

ext_modules = [Extension(name="_utils", sources=["_utilsmodule.c"])]

)

We can build using the command

python setup.py build

and install using the command

python setup.py install

Showing Use Case

We can use type _A as you would the default dict type. as shown below.

>>> from _utils import _A as A

>>>

>>> A

<class '_utils._A'>

>>> A.__bases__

(<class 'dict'>,)

>>>

>>> type(A)

<class 'type'>

>>>

>>> type(A())

<class '_utils._A'>

>>>

>>> a = A({"obi": "chidi", "nwafor": "chukwunonso"})

>>> a

{'obi': 'chidi', 'nwafor': 'chukwunonso'}

>>>

>>> a["nwafor"]

'chukwunonso'

>>>

Use Case from first.py

We originally defined the extension module to be callable from first.py. So let us test it.

>>> from first import A

>>> A

<class '_utils._A'>

>>>

Well for starters it worked. Our type _A from module _utils was called. But we noticed something else, the class of A is not <class ‘first.A’> but <class ‘_utils._A’>, this can result in confusion depending on the use case of _utils.

We can change the tp_name of AType in _utilsmodule.c to “first.A” from “_utils._A” to reflect that we are calling _A from first.py. With that, If we build and install we now have:

>>> from first import A

>>> A

<class 'first.A'>

>>> A.__module__

'first'

>>> A()

{}

>>>

Exactly as we wanted. However this has radically decoupled _A from _utils based on the change of tp_name. If we want to use _A from _utils without going through first.py, we also have the same problem of it showing <class ‘first.A’> instead of <class ‘_utils._A’> as shown below.

>>> from _utils import _A

>>> _A

<class 'first.A'>

>>>

This two points should be considered when you create your extension modules.