The Ultimate Guide to Linting
By: Kovid Rathee
Tags: Programming, Code, Software Development, Style Guides, Data Engineering

The Ultimate Guide to Linting

An essential guide to linting for making your code more readable and life easier

Background

Code quality has been a big topic of discussion in the last decade. More people are talking about it now as more people have to read more code on a daily basis. Reading other people’s code is hard. We all know that! A lot of tools, IDEs have made our lives easier by providing a standard framework to work within. These tools enforce some level of sanity but a lot still needs to be done by the developer.

Good, readable code saves more time than we think. The first step, I think, to writing good, readable code is to be stylistically consistent — be it indentation, the spaces between two functions, the way you declare variables, even the way you import libraries. All this has to be taken care of while you’re writing your code for the first time. After that, there are tools to help you out.

A linter is one such tool that helps identify potential issues with the code based just on style guides, naming standards, typos, and bugs too. Linter is a part of the larger picture of code quality. Most languages, based on their stylistic grammar and syntactical structure, have found standard ways of writing code in them. Yes, there are debates over tabs and spaces but most of the other debates are, more or less, settled. Based on set patterns, tools that enforce these patterns are available in the market.

A very short introduction to Linting by Ahsan Zahid.

As engineers have realized the importance of writing readable code. This trend has spiked the use of fancy IDE features which help you write code better. It could be as simple a feature as autocomplete (e.g., Intellisense) or a query beautifier. Software developers, data engineers, data scientists, infrastructure engineers, data analysts — all these people spend a large chunk of their time writing code — code that would be maintained, code that other people would read. An average data engineer uses at least 2 IDEs — one for writing Python or Scala scripts for ETL, orchestration etc., the other one for writing good old SQL.

Good, readable code saves more time than we think.

Linters for Python

As Python is, more or less, the de facto language for data engineers and data scientists, let’s talk about it. One of the first proposals to use a style guide for uniformity and consistency was from the original authors of Python themselves. It is PEP8. Since the PEP8, a couple of other major linters have evolved and taken over the market like — Pylint and Flake8.

Flake8 is essentially a wrapper around the following — PyFlakes, pycodestyle, and Ned Batchelder’s McCabe script. Every major IDE, code editor supports this. For instance, if you’re using VS Code, you can follow this link to understand how to set up or disable linting there. Google’s Python style guide is based on pylint.

If you write code in Java, Scala, or any other language, there are linters available for them too. Here’s a repository from Google which contains style guides for all the languages they use

Linters for SQL

Almost all major SQL IDEs have an option to enable syntax checkers and query beautifiers and linters. SQL is also, for some reason, one of the most abused languages when it comes to style, consistency, and readability. Try reading someone else’s queries. I have. Bothered by that issue, a while ago, I wrote about the benefits of using a style guide while writing SQL.

Similar to Python, there are a number of style guides for writing SQL too. But there are different challenges when it comes to styling SQL. Different databases, data warehouses support different features of SQL, some are specific to the platform being used. It is, therefore, hard to come up with a unique style guide that works for Snowflake, Redshift, SparkSQL, MySQL, PostgreSQL, MS SQL Server, and Oracle at the same time. Not only that, but the style guides also have to be version-specific too.

Try reading someone else’s queries.

The most comprehensive and sensible style guide I could find was written by Simon Holywell. There are a couple of other style guides like GitLab’s SQL Style Guide and Mozilla’s owner SQL Style Guide. Both of these are also well thought-through. All that’s good but how to check if your SQL code follows a given style guide. Where are the linters?

There are two well-maintained linters for SQL — sql-lint and sqlfluff. You can choose one of them or write your own and integrate it into your CICD pipeline. Let’s now talk about the last bit I said — the CICD pipeline.

Conclusion

Finally, linting should be a part of your CICD pipeline. Executing the linter manually on your code is, again, moving in the wrong direction. The idea of linting was to reduce the possibility of inducing human errors. Running the linter in the IDE, hence, shouldn’t be the only place where linting is run. Just like automation tests, linting has to become a part of the pipeline. Before your code is merged to the next level, it should be linted automatically — and if the lint fails, the pipeline should fail.

Linting is more than just checking for style, it can be about whatever you want it to be under the purview of code quality. You can write your custom checks inside the linter. A linter can contain any kind of logic that you want. The idea is that there’s a piece of code that essentially goes through your piece of code to check if it follows the rules you have defined.