Auto-Docs for Python. Documenting code is boring, so why... | by James Briggs | Towards Data Science

Good documentation is, whether we like it or not, absolutely crucial to the success of a project. So why is good documentation so hard to find?
In reality, it is often neglected and treated as an inferior product to the code itself. It is simply too time consuming to produce good, user-friendly documentation.
This neglect is a mistake. The adoption of the tools we build largely rests on ease of use. If documentation is hard to navigate, comprehend, or simply not attractive to use, users will not like it.
So, with that in mind, I believe we should all strive for simplistic, but comprehensive documentation.
We will cover how to quickly build a auto-documentation tool for any Python project. With this, documentation can be generated for complex projects with huge amounts of code in nothing more then a few seconds.
Regular Expressions (regex) are our friend here, we will rely on them heavily throughout the article. My intention with this article is to really focus on this aspect of the code, which I hope you will learn a lot from (I certainly have).
If you use NumPy/SciPy docstring standards, you can even use the exact code I use, which is available on GitHub here!
Before starting, out-of-the-box documentation tools are readily available. There is nothing wrong with using them, in fact, Sphinx, the only one I have used comprehensively is actually very good.
However, as of yet I haven’t the other tools particularly quick at producing the style of documentation that I like.
Nonetheless, they are good and I advice you to consider all of your options. If you are happy with Sphinx (or another) tool, use it. If you want to build something in your own style (and learn regex), use this guide.
The autodocs documenting structure can be represented as a tree like so:
From this we can see there are several key structures we must be able to extract from our code, these are:
Docstrings: (containing descriptions, parameters, parameter dtype etc) these are the block quote sections we use to describe each script, class, and function.
Classes: the class definitions themselves, their descriptions (contained within the docstring), and methods.
Functions: the functions themselves, alongside their docstrings, which contain function description and parameters.
Parameters: contained within the function docstring. Every parameter contain a descriptions, datatype, and an ‘optional’ flag.
We will take each of these components and their respective sub-components and use this information to develop another tree-like structure of simplistic, comprehensible documentation.
As you can see, we can identify docstrings by the triple quotes surrounding each. In regex, we write (?s)\"{3}.*?\"{3}.
There are a few things happening here.
\ is used as an escaping character. We place this in front of characters that have other meanings in Python " or regex {[()]} to ‘escape’ their native meaning. Indicating that we actually just mean to look for that character.
We have two sets of \"{3}, this simply matches the two sets of triple quotes we have.
.* matches any character except newlines, ? makes this a ‘lazy’ quantifier — stopping the regex matching the very first instance of """ to the very last instance of """ , instead we match between the first and second, third and fourth, etc.
(?s) is a global pattern modifier, s tells the regex that we want .* to also match newlines.
The beginning of a class can easily be identified by the word class followed by a space, a few letters, and a colon.
To prevent us from matching everything after the beginning of a class, we identify the end of the class as the first place that a newline is directly followed by a text character.
In regex, we write (?sm)class [\w\d_]+:.*(^\w).
[\w\d_] matches text, numeric and underscore characters, adding + means it will match an unlimited number of these characters.
(?sm) is again our global pattern modifier, the m flag tells the regex that we want ^ and $ to signify the start and end of a line respectively.
(^.) is the start of a line ^ immediately followed by any character ..
Functions are a little bit different. Unlike classes, where we capture everything contained within the class (so that we can extract the class methods too), we only need the function name and docstring.
Fortunately, this makes our first function matching regex much easier, we write (?s)def [\w\d_]+\(.*?\):\s+\"{3}.*?\"{3}.
Each function also contains a name, description, and parameter list that we must extract. We do this in parts, parameter extraction is a little more complex and we will cover next.
The function name we extract using def [\w\d_]+\(, with which we remove def and (, leaving just the function name.
name = re.search(r"def [\w\d_+\(", func).group()name = name.replace("def ", "").replace("(", "").strip()
Description desc and the parameter list params are extracted by pulling out the docstring with (?s)\"{3}.*?\"{3} and splitting on Parameters\n. We then remove triple quotes and excessive white-space.
docstring = re.search(r"(?s)\"{3}.*\"{3}").group()desc, params = docstring.split("Parameters\n")desc = re.sub(r"\s+", " ", desc.replace('"""', "")).strip()
We have our parameter list contained in params , which looks like this:
To match each parameter, we write (?s)\w+ : .*?(?=\w+ :).
For the most part we have already covered everything in this, except for one part — (?=\w+ :).
In this, we know that \w+ : will match multiple letters followed by a space and colon. However, it is wrapped in (?=). This is called a positive lookahead assertion.
By adding this, we are telling the regex to assert (?=) that what we have matched is immediately followed by \w :.
If we put this into an online regex tester, we can see it works almost perfectly, but if unfortunately misses the final parameter (as it is not followed by \w :).
We solve this by extracting a single parameter at a time. In this loop, if we find that no parameters have been found, we try again with a modified parameter finding regex (?s)\w+ : .*, extracting the final parameter.
Our final, deepest layer of extraction requires us to extract the parameter name, datatype, ‘optional’ flag, and description.
For this, we simply split our parameter text new_param by newlines \n. Now, at index 0 we have the first line containing the parameter name, datatype, and possibly the optional flag. Let’s modify the code above to include this extraction.
The only additional logic here is the splitting of the first line of new_param by :, giving us name and dtype. Followed by a check for the word 'optional' in dtype, giving us optional.
Then we simply add these parameter details to the new params dictionary. Giving us something like:
At this point, we have finished with the code that will extract everything we need from our code. We have created the tree structure we touched upon earlier:
Anything contained within the scripts or classes row will be given a dedicated documentation page. The script page will contain a name, description, and link to any classes below it.
When building the documentation, we will loop through every script and class contained within a project. Iteratively building the documentation for each.
With this, we can move onto building the HTML code for our docs.
The docs are split into four key sections:
Script/class information. This appears right at the top of each page. In this is the information from the very first script or class level docstring.
Breadcrumb navigation bar. Allowing us to navigate up and down documentations levels.
Classes section. In here we will have a list of script classes, clicking on a class button will display the class description and documentation link.
Functions section. This contains a list of functions, detailing their respective parameters.
Before dynamically building these sections, we do need to add <head> information to our initial HTML page, we will keep it very simple for now.
We will be using Bootstrap for the CSS and JS in our page. Within the <head> we attach the Bootstrap CSS stylesheet using the <link> element. The JS scripts are added later.
In Python, we can either add this code into a triple quote block, or read it in from file. The variable html will initially just contain this <head> element.
This is our script/class information appearing at the top of our page.
As we loop through each script and class independently, we can treat them the same. The only difference between the two being that script pages will contain class sections, whereas classes will not (unless you have a class inside a class).
For docs.py, fullpath will just contain a single entry ['docs']. But for lower levels, such as DocsBuilder , this will contain the list of layers leading to the class ['docs', 'DocsBuilder']. Our main header is therefore the last index in the list fullpath[-1].
The top-level description is contained within desc.
Next is our navigation section, we initialize this with our highest level page, the readme.
Following this we loop through the items in fullpath to create a full ‘breadcrumb path’ to our current page.
As the filenames produced will all be camel case, and include any preceding classes/scripts. We apply the same formatting to the path variable. This makes DocsBuilder -> docs.docsbuilder.
Classes are all stored in a dictionary creatively named classes. This dictionary contains class name and description key-value pairs.
First we check that the current level contains classes, if not, we don’t add a class section.
If we do have classes, we add the beginning of the class and class buttons section. We then loop through classes, adding a button for each item, containing the class name only.
Next we add the end of the class buttons and the beginning of the class descriptions (and links). We loop through each item, adding the button contents. This allows us to show the class description and link when clicking on a button.
Finally, we end the class section.
Functions are stored in a dictionary called funcs, which is formatted as above.
There is a lot more going on here due to the additional layer of parameters inside each function.
All functions are contained within an unordered list <ul>, where each list item <li> consists of a single function.
We first add the function name, followed by the function example contained within the <kbd> element, built by joining all parameters with ",".
Following these is the parameter table. Here we initialize the beginning of the table with <table> and <tbody> tags. A loop builds the table row by row, consisting of the parameter name and description.
After this are several tag closing sections. As the functions section is our final dynamically built part of the page, we can add the Bootstrap JS scripts and closing tags.
Once we have put all of these code extraction and page building scripts together, we can build documentation with just a few lines of code.
Here of-course, everything that you have seen throughout the article has been built into the DocsBuilder class.
All of the extraction regex we worked through is contained in the extract method. The HTML building code within the build method.
I have run the same scripts on much larger projects containing many scripts and classes. Nonetheless, all documentation is built in an instant.
You can find all of the code in the autodocs GitHub repo. It is by no means a finished project, so please feel free to suggest improvements or point out unnecessary code!
If you have any other questions or suggestions, feel free to reach out via Twitter or in the comments below.
Thanks for reading!
If you enjoyed the article, you may be interested in another article I recently wrote on processing text data from the web. With a focus on data preparation for Machine Learning: