Package: illiterate
IlliterateUnobtrusive literate programming experience for Python pragmatistsilliterateis a Python module that helps you applysomeof the literate programming paradigm
without requiring a meta-programming language (likenoweb) or any preprocessing step to actually
get your code up and running.It works kind of opposite to how literate programming suggests, in the sense that you write code-first
and embed documentation into your code. I know, purist literate programmers will hate this but hey, it's a compromise.What is all this (il)literate programming about?If you've never heard about literate programming before, then I suggest you to read at least theWikipedia entryand then we can continue discussing.
It is a fascinating topic, and there aremany resourcesout there.Back already? Ok, so here is illiterate's take on this matter.Ideally, for literate programming to work, you would code in a meta-language mixing prose, code, and macros.
This is great if everyone that will ever write code in your project is willing to indulge in literate programming.
Sadly, I have found that more often that not, this is not the case.
Hence, even though it is awesome, literate programming has some major practical drawbacks that, at this moment,
make it impossible for many people to apply it widely, including:Poor support from editors and lack of tooling, which is not just a matter of syntax highlighting. The very feature that makes
literate programming extra powerful, i.e., macros, makes it almost impossible for any semantic analysis to work, so forget
about intellisense, smart completion, or interactive linting.A hard entry curve, since unfortunately people in the 21st century still learns to code the "old" way, that is,
code-first. Introducing someone into literate programming is hard because it takes some time to grok it and understand the benefits.It's hard to incrementally switch to it. If you already have a somewhat large program written in the "traditional" way,
it's very hard to port it to the literate programming paradigm incrementally.All these reasons make it, at least for me, almost impossible to apply pure literate programming in anything more
than toy projects. However, I do love the paradigm, and I do think it makes you a better programmer, and makes your code
easier to maintain and understand. I wanted a way to introduce as much of literate programming as possible into the
traditional programming paradigms, but still being able to use the same tools, introducing literate programming
idiosyncrasies incrementally into existing
codebases but "flying under the radar" as much as possible, so detractors don't complain.Hence,illiteratewas born. It is called that way in part because is kind of a twist on the literate programming
paradigm, and also because it is supposed to help us illiterates to write more literate code.So what does illiterate proposes?Glad you asked. The idea is to encourage a more literate codebase while introducing as few changes as possible.
Specifically, you should not need to use new tools, editor extensions, or preprocessors. Code written using the
illiterate style looks exactly like regular code, but hopefully, a bit better.Everything stems from these key principles:Documentation for a codebase should be written as prose, and it should be enjoyable to read it top to bottom.
It should not be simply a list of modules and methods with few-line descriptions; rather, it should be a cohesive
piece of literature that clearly explains the authors' intents for any small details and how everything fits into the bigger
picture.Documentation should be as close as possible to real code, ideally right next to it, instead of in external
markdown files that can easily get out of sync. Furthermore, there should be some automated way
to ensure that documentation is up to date, ideally with embedded unit tests that fail if documentation is wrong.Documentation should be written both for users of your code and future developers, and it should be
easy for anyone to dive as deep as they want. This means that users only interested in calling your high-level
API can easily understand how to use it, while collaborators or anyone wanting to understand how everything works
should also be able to follow all the details.To achieve these objectives, illiterate proposes really only one major paradigm shift:Your code is the documentation.That's it. You will simply write all the documentation for your code right inside your code, as comments and as Python docstrings,
according that what is more convenient for each case. But keep this in mind,all your code will be published as-is for documentation purposes.Now you are forced to think about your code in terms of: "well, this will be read by users at some point, so I better make it as publishable as possible".
This means that you can no longer simply put some throw-away code in some forsaken_tmp.pyfile. That file will appear in the documentation, so it better be publishable.How this works?The only thing that illiterate does is taking your Python repository and turn it into Markdown files.
It will parse all your code, and output nicely formatted Markdown versions of each.pyfile. It is up to you that what is writen in those.pyfiles is something worth publishing as documentation.To use it, you simply run:python -m illiterate build [src] [output]Where[src]is the folder that is the root of your project's code (i.e., the top-level folder with an__init__.pyinside), and[output]is where you want the markdown files. You can add--copy from:toto copy verbatim some files into the output folder.
I do this for copying theReadme.mdinto anindex.mdwhich becomes the homepage.For example, in this project, standing on the root folder (where this Readme is located), you would run the following (🤓 yeah, it is kind if Inception-ish):python -m illiterate build illiterate docs --copy Readme.md:index.mdThis will take all the code inilliterate, convert it to Markdown, and drop it inside thedocsfolder.
It will also copy theReadme.mdfile intodocs/index.md.A tweak using settings.The same procedure can be obtained using configuration files. The configuration files have the following structure:inline: true # true if we want to process the comments within code blocks
output: docs # The address of the output folder
sources: # List of files to be processed or copied
  sample_module.file: sample_module/file.py  # An input is made up of the address of the output file without suffix and using periods as a separator and the address of the input file.The configuration is usually in a file calledilliterate.ymlsomehow mimicking mkdocs.Although these configuration files can be done completely by hand, we recommend using the commandpython -m illiterate preset initto create it. This command uses the same parameters as thepython -m illiterate buildcommand but instead of parsing the code directly it creates the appropriate configuration file.
As long as we have a configuration file to start with, we can modify it and include other details to suit our purpose as our code progresses. Then we can make illiterate work according to that setting using the commandpython -m illiterate preset buildto build the Markdowns.What you do with those Markdowns is up to you. In this project, I usemkdocsfor documentation.
If you havemkdocs, then make sure to have yourmkdocs.ymlcorrectly configured so that it renders those freshly created markdowns.
You can also see themkdocs.ymlin this repository to get an idea of how that configuration looks, but beware, I'm using some custom
themes and other stuff you might or might not want.You can mix illiterate with regular markdown simply by hand-crafting all the documentation you want in pure Markdown and then conveniently designing yourmkdocs.yml.Next stepsThis project is quite small, but it is a self-fulfilling prophecy. The remainingdocumentationhas been written withilliterate. Just keep reading and you'll see for yourself what does this mean.