Leading with commas — ugly or efficient? An investigation over 320 GB of SQL code
By: Felipe Hoffa
Tags: Bigquery, Data Science, Big Data, Github, Open Source

Leading with commas — ugly or efficient? An investigation over 320 GB of SQL code

Winning arguments with data: Let’s analyze 320 Gigabytes of open source SQL code to determine if we should use trailing or leading commas. Popularity is not enough — can we determine which style leads to success?

Switching from BigQuery’s custom SQL to standard SQL:2011 has been great — except when not: I’m not allowed to leave trailing commas anymore. They are really practical when adding, removing, and commenting lines (Go even enforces them). To keep my productivity up, I was forced to switch to a controversial style: Now I open my enumeration lines with commas.

Leading with commas is not a popular style — but I’ll argue that it’s a productive one. Can I prove this? Here I analyze 320 GB of SQL code from 80,000 GitHub repositories to support this choice.

Background

Many programming languages (C, Python, JavaScript, Go, …) will allow you to leave a trailing comma after the final element in a list. But others won’t, like SQL (or the JSON format). This has lead a vocal minority to advocate in favor of a dreadful practice: Start new lines with a leading comma, in the name of efficient code writing and change management.

Would you prefer SQL code formatted like this:

# trailing commas

SELECT name,

company,

salary,

state,

city

FROM `employees`

WHERE state='CA'

or like this:

# leading commas

SELECT name

, company

, salary

, state

, city

FROM `employees`

WHERE state='CA'

Let’s bring the data

Luckily we all have access to terabytes of open source code ready to be analyzed in BigQuery. There is a huge table with all of the available extracted code — and your first step should be extracting only the code you are interested in, before performing further analysis.

To get all of the code inside “.sql” files out into a new table:

#standardSQL

SELECT a.id id, size, content, binary, copies,

sample_repo_name, sample_path

FROM (

SELECT id

, ANY_VALUE(path) sample_path

, ANY_VALUE(repo_name) sample_repo_name

FROM `bigquery-public-data.github_repos.files` a

WHERE PATH LIKE '%.sql'

GROUP BY 1

) a

JOIN `bigquery-public-data.github_repos.contents` b

ON a.id=b.id

This gives us a table representing a total of 2 million SQL files out of GitHub — more than 320 GB of code!

I left an improved public copy of this table in BigQuery —so you can go straight to the next step.

First approach: Count number of lines, files, repos

My first approach using data backfires:

Findings:

There are 50 million SQL lines that end with a comma, while only 1.4 million begin with one.

There are 329,308 unique SQL files with trailing commas, while only 18,312 have leading ones.

There are at least 72,506 repositories with SQL files containing trailing commas, while only 3,360 repositories with leading ones.

Summary: Leading commas lose the popularity race. There are way less projects and files using them.

Side note: There are also 5,647 SQL lines in 1,029 files that have commas at the beginning and the end. At least 360 repositories allow these monstrosities.

Second approach: Incorporate success metrics

Losing the popularity race is not the end of the road. We need to ask ourselves: “Which projects are more successful?”

How do we define success in GitHub? Number of stars? Number of stars last year? Number of stars this year? Number of active users? General activity levels? How about all of that!

The rules:

Each repository gets one vote, regardless of number of lines or files.

Files present in many repositories get assigned to the one with most stars.

Repositories will get assigned to one of four groups: Only leading or trailing commas, no commas at the start of end of a line, or a mix of both styles.

These are my results: