Advanced Python: Metaprogramming
By: Farhad Malik
Tags: Python, Fintech, Python3, Technology, Programming

Advanced Python: Metaprogramming

Explaining what, why and how Metaprogramming works in Python

Metaprogramming is a complex yet one of the most interesting topics in Python programming language. Metaprogramming makes the Python programming language extremely powerful.

Metaclass is above classes

We have come across decorators. We often use decorators to enhance the Python code and add extra functionality to the existing functions/classes. These decorators are part of metaprogramming in Python.

Code generators are also part of metaprogramming. We can introspect the functions, classes, types of an object and modify them at runtime. The IDEs use the metaprogramming feature to provide code analysis.

This article will illustrate what Metaprogramming is, where it can be used, and how it works in practice.

Expert level Python developers, who intend to implement frameworks and their own libraries in Python often use the Metaprogramming features of Python.

This is an advanced level topic for Python developers and I recommend it to everyone who is/or intends in using the Python programming language.

Metaprogramming can be troublesome if it is not understood properly due to the fact that unexpected side effects can be encountered. This article will explain it in an easy to understand manner.

If you want to understand the Python programming language from the beginner to an advanced level then I highly recommend the article below:

Article Aim

This article will provide an overview of the following topics:

What is Metaprogramming In Python? How Does Metaprogramming work? Where can we use Metaprogramming?

Before I begin, I want to clarify that metaprogramming is a complicated programming topic and unless the requirement can only be solved via Metaprogramming, I would recommend the developers choose a different programming approach than Metaprogramming.

Therefore, only use the Metaprogramming concepts if you absolutely have to use it.

The expert level python developers generally have a thorough understanding of the concept.

1. What Is Metaprogramming In Python?

Before I begin, let’s remember that everything is an object in Python.

A function, constant, variable, literally everything is an object. To elaborate, even a class is an object. As a result, we can treat a class as any other object and pass the class as a parameter, store it, and modify it at runtime.

A class is an object that can be used to instantiate new objects. A class can be seen as a bucket that groups the objects. It defines the protocols/rules for the objects that it creates. A metaclass is above a class. It groups a set of classes together. We can have meta-information about classes within a metaclass.

As an instance, consider the code below:

def get_fin_tech_explained():

class FinTechExplained:

pass

return FinTechExplained



print(get_fin_tech_explained())

This method returns the class FinTechExplained:

<class '__main__.get_fin_tech_explained.<locals>.FinTechExplained'>

As the class can add protocol/rules to an object, a metaclass adds protocol/rules to a class.

Python uses a metaclass to create a class for you.

The first key note to remember is that everything is an object in Python including a class and each class is created by a metaclass.

A metaclass allows us to add special behavior to a class.

Before we move any further, let’s understand what type() is

The constructor of the type class is called to retrieve the type of an object.

As an instance, I can do:

def get_fin_tech_explained():

class FinTechExplained:

pass

return FinTechExplained



print(type(get_fin_tech_explained)) print(type(get_fin_tech_explained()))

The first print will print:

<class 'function'>

This is because get_fin_tech_explained belongs to the class function.

The second print will print:

<class 'type'>

This is because the return of get_fin_tech_explained() belongs to the class type.

The method returns the class. It means each class is essentially a type of a type.

The second keynote to remember is that each class is an instance of the type, type.

The class type is a metaclass. This class is used to create other classes in the Python programming language. It defines the rules/protocols of the classes it creates.

Now the constructor of the type can be used to create classes that can then create instances of the class.

Consider this class below:

class FinTechExplained:

def __init__(self, blog_name):

self.blog_name = blog_name



def get_blog_name(self):

return f'Blog name is : {self.blog_name}'



fin_tech = FinTechExplained('Metaprogramming')

print(fin_tech.get_blog_name())

The class has a constructor that takes in a blog_name and a method that returns the name of the blog.

As an instance, the print statement returns:

Blog name is : Metaprogramming

We can create a class using the type constructor. We are going to create the FinTechExplained class by using the type() constructor without declaring the class.

def FinTechExplained_init__(self, blog_name):

self.blog_name = blog_name



fin_tech = type("FinTechExplained",

(),

{"__init__": FinTechExplained_init__,

"get_blog_name": lambda self:f’Blog name is : {self.blog_name}’})

This is equivalent to us creating the FinTechExplained class ourselves.

The constructor of the type() takes in the following parameters:

type(cls, what, bases=None, dict=None)

the first parameter is the name of the class

the second parameter is the tuple of base classes

the third parameter is a dictionary of key (attributes) and values (implementation)

As you can see that the dict parameter is where we passed in a dictionary with two keys where the keys are the method names and the values are the implementation of the methods.

Once I run the following statements:

fin_tech = FinTechExplained('Metaprogramming')

print(fin_tech.get_blog_name())

I will get the following output:

Blog name is : Metaprogramming

This is also the reason why we have a __dict__ property for all of our objects.

A metaclass allows us to add special behavior to a class.

As an instance, a descriptor is a protocol in Python. It provides __get__ and __set__ methods. The descriptor protocol defines how attribute access is interpreted by the language. The methods of the descriptor class can be reused.

Internally, when we set and get a property, python calls the __dict__.__get__ and __set__ methods. If the methods do not exist then Python calls the _getattr_ and __set_attr__ methods.

Therefore, if your class attribute is an object and that object has a descriptor then it implies that we want Python to use the __get__ and __set__ methods and we want it to follow the descriptor protocol.

Descriptors, when combined with metaclasses, can achieve powerful behaviour. We can combine __set_name__ on the descriptor classes to take in the surrounding class and property names.

The key to remember is that the functions are descriptors in Python. They can be added to the class at run time. Remember the method binding is performed on the attribute lookup as part of the descriptor protocol.

If we want to decorate all of the methods of our class with a certain decorator then we can use a meta-class

2. How do we do Metaprogramming?

We can start by creating our own metaclass. The first step is to create a class that inherits from the type, type.

The type will add additional behavior to the new metaclass. A class can only have one metaclass.

class FinTechExplainedMeta(type):

The snippet of code above shows that we have created a new metaclass called FinTechExplainedMeta.

The key point to note is that a metaclass in defined by inheriting from type.

When we create a class, we can set the metaclass attribute to a metaclass that is inherited from type.



The metaclass will have a __new__ method that calls type.__new__ method.

class FinTechExplainedMeta(type):

def __new__(cls, what, bases=None, dict=None):

return type.__new__(cls, what, bases, new_dict)

The metaclass has essentially access to the class name, its parents, and all of the attributes.

We can now add validation rules in the Meta.__new__ method and validate all of the parameters.

We can use the magic methods of Python programming language to override the behaviour so that any class that makes our new class as the Metaclass will now need to conform to the rules.

__new__() is another example of metaprogramming. It creates new class instances and it is not bound to an instance of the class in nature. This method is called before __init__() is called. We can override the super class’s __new__().

The return of the __new__() method is the instance of the class. This is useful when we want to modify the creation of immutable data types such as tuple.

Now I am going to modify the __new__ method of the FinTechExplainedMeta class and check that the class has a method ‘get_blog’.

class FinTechExplainedMeta(type):

def __new__(cls, what, bases=None, dict=None):

print(dict)

if 'get_blog' in dict:

print('Great you have get_blog')

else:

raise Exception('get_blog missing')

return type.__new__(cls, what, bases, dict)

Now I will create a new class called FinTechExplained (as above) and have the FinTechExplainedMeta as its metaclass.

class FinTechExplained(metaclass=FinTechExplainedMeta):

pass

Notice, how setting the metaclass attribute at the class level is required.

As a result, when I run this code, even before instantiating the FinTechExplained object, it will break as soon as the module is loaded:

`C:\Users\farhadm\PycharmProjects\Learning\venv\Scripts\python.exe C:/Users/farhadm/PycharmProjects/Learning/PythonGeneral/meta_programming.py

Traceback (most recent call last):

File "C:/Users/farhadm/PycharmProjects/Learning/PythonGeneral/meta_programming.py", line 41, in <module>

class FinTechExplained(metaclass=FinTechExplainedMeta):

File "C:/Users/farhadm/PycharmProjects/Learning/PythonGeneral/meta_programming.py", line 38, in __new__

raise Exception('get_blog missing')

Exception: get_blog missing

Notice how the exception is thrown even before I created an instance of the class. This is how we can enforce every class to have certain methods/attributes that our framework/library requires.

If you want to learn more about the magic methods, I highly recommend reading this article:

I can now create a new class FinTechExplained and have its metaclass as FinTechExplainedMeta and ensure FinTechExplained contains the get_blog method:

class FinTechExplainedMeta(type):

def __new__(cls, what, bases=None, dict=None):

print(dict)

if 'get_blog' in dict:

print('Great you have get_blog')

else:

raise Exception('get_blog missing')

return type.__new__(cls, what, bases, dict)



class FinTechExplained(metaclass=FinTechExplainedMeta):

def get_blog(self):

return 'get_blog is there.'



Now as soon as I load the module, it will print out:

{'__module__': '__main__', '__qualname__': 'FinTechExplained', 'get_blog': <function FinTechExplained.get_blog at 0x00000024DE033730>}

Great you have get_blog

We can see the printed dict. This is because of the print(dict) statement in the FinTechExplainedMeta class.

We can use metaclass’s __new__ method. It is called whenever a class is instantiated and it is called before the __init__ method.

Metaclasses can be used to modify the class attributes.

As an instance, I am now going to modify all of the methods so that they are all prefixed with ‘Farhad’. The functionality of Farhadget_blog will remain as the functionality of get_blog above. This is how we can add new methods to a class via metaclass:

class FinTechExplainedMeta(type):

def __new__(cls, what, bases=None, dict=None):

print(dict)

if 'get_blog' in dict:

print('Great you have get_blog')

else:

raise Exception('get_blog missing')



new_dict= {}

for key, val in dict.items():

new_dict['Farhad'+key] = val



return type.__new__(cls, what, bases, new_dict)



class FinTechExplained(metaclass=FinTechExplainedMeta):

def get_blog(self):

return 'get_blog is there.'

pass



fintech = FinTechExplained()

print(fintech.Farhadget_blog())

Even though Farhadget_blog() doesn’t explicitly exist in the FinTechExplained class but the metaclass added it for us.

As a result, the following line is printed:

fintech.Farhadget_blog()

--> get_blog is there

The expert level python developers generally have a thorough understanding of the metaclasses concept.

Python offers a utility known as exec. We can use the exec* to execute any sequence of Python statements.

We can also use eval() and compile() functions to execute Python code.

We can also add meta hooks that are called before any other import is called. This is done by adding sys.meta_path and are called as part of sys.path processing.

Projects use these features to dynamically add the code via string.

3. Uses Of Metaprogramming

There are a number of uses of Metaprogramming, including:

If we want to check that a class was defined correctly, we can use the metaclass.

We can use metaclasses to raise errors during module imports.

If we want every module of our framework to have methods with a particular signature or if we want to have our classes to have certain naming conventions and methods/attributes then we can use metaprogramming to achieve it.

Code generators are also part of metaprogramming. We can introspect the functions, classes, types of an object and modify them at runtime. The IDEs use the metaprogramming feature to provide code analysis.

Each time a base class is subclassed, we can run a specific code using metaclass.

We can use a subclass to annotate or modify properties before the class is used.

We can use a metaclass to validate classes. It can also be used to set the attributes of your class by using the class.__dict__ attribute.

We can use a metaclass to validate classes. It can also be used to set the attributes of your class by using the class.__dict__ attribute. Metaclasses can be used to modify the class attribute.

We often use special methods and metaclasses to change the way Python objects work. We can even extend Python syntax by creating our own domain-specific language (DSL) using the abstract syntax tree (AST).

ORM (object-relational mapping) classes traditionally use metaclasses. Developers who concentrate on writing frameworks often use metaclasses.

FinTechExplained — Thank you for reading

4. Summary

This article provided an overview of the following topics:

What is Metaprogramming In Python How Does Metaprogramming work Where can we use Metaprogramming

A function, constant, variable, literally everything is an object. To elaborate, even a class is an object. As a result, we can treat a class as any other object and pass the class as a parameter, store it, and modify it at runtime.

To understand the concept of Metaprogramming, it’s important to know that metaclass is a type. The type itself is a class. It can define other classes. Therefore it’s a metaclass.

This is an advanced level topic for Python developers and I recommend it to everyone who is/or intends in using the Python programming language. Metaclass can be troublesome if they are not understood properly due to the fact that unexpected side effects can be encountered.

Metaprogramming is a complex yet one of the most interesting topics in Python programming language. Metaprogramming makes the Python programming language powerful.