Exploring Unicode in Python
By: Chris Webb
Tags: Unicode, Python, Python Programming

Photo by Brett Jordan on Unsplash

You might be surprised to hear that Unicode can represent up to 1,114,112 characters, 137,994 of which have currently been allocated. They include letters from a huge variety of alphabets, punctuation, numbers and general-purpose symbols.

Each character has a name and a category to help you track down the ones you need, and in this project I will write a Python module which returns details of those satisfying specified search criteria.

This project consists of the following two files which you can clone/download from the Github repository.

unicodefilter.py

unicodefilter_test.py

Let’s look first at unicodefilter.py.

Firstly we import unicodedata from the Python standard library. This provides a number of methods but I will just be using it to get the full name and category of each character.

get_characters

The get_characters function takes two arguments which are search strings for character names and category names respectively. Their defaults are empty strings which will return all characters. They are converted to lower case in the first two lines of the function to make the function case-insensitive.

Categories come to us from the unicodedata module as two-character strings so we need some way of obtaining the full category names. I have done this using a dictionary created by the _create_category_names function which I’ll describe further down.

Next we create an empty list before iterating from 0 to 137994. Within a try/except we use the chr function to obtain the character equivalent of n , and then use unicodedata.name to attempt to get the character’s name. Some characters are control characters so do not have a Unicode name; these will cause unicodedata.name to raise a ValueError exception which we can just ignore with pass .

If we make it to the next line we use unicodedata.category to get the character’s category as a 2-digit code, and then get the full category name from the category_names dictionary.

Next we check to see whether the two filter criteria are in the character name and category name respectively. Note that lower() is called on these, again to make the function case-insensitive. If there is a match we create a dictionary with six pieces of information about the current character and then add it to the list.

Finally the list of characters satisfying the search criteria is returned.

_create_category_names

As I mentioned above categories have a two character code, the first character (in upper case) standing for the major category and the second (lower case) character standing for the minor category.

This function simply creates a dictionary with the category codes as keys and the category names as values, and which was used in get_characters .

Now let’s write a bit of code to test the module.

The unicodedata module provides a unidata_version function which returns the version of Unicode it supports. I wrote this code using Python 3.7 which was released in June 2018 when the latest version of Unicode was 11, so that is what unidata_version returns, as you can see in the screenshots below.

The call to unicodefilter.get_characters includes arguments to get Coptic numbers, and is followed by a loop to print them in a table. The widths for character names and category names might look a bit high but some of the names are very long.

Now we can run the program.

python3.7 unicodefilter_test.py

This is our list of coptic numbers.

This is the output with a character_name_like argument of “chess”.

The elephant in the room with this terminal-based program is that it is no more than a demo of the underlying code, and certainly not suitable to be released as a user application.

However, the unicodefilter module could be used as the basis for a GUI allowing users to enter filter criteria and then display matching characters in the way my test program does but in a more user-friendly way. It could also be used as part of a web application (Django, Flask etc.) providing the same functionality.

Oh, and another thing: you could write a chess program…!