How to find and organize your data from the command-line
By: Robert Yi
Tags: Data Science, Data Catalog, Towards Data Science, Command Line, Productivity

How to find and organize your data from the command-line

Introducing metaframe: a markdown-based documentation tool and data catalog for data scientists.

Repo: https://github.com/rsyi/metaframe

If you are a data scientist, it’s your job to extract value from data. But in your career, you’ll likely spend (and have spent) a non-trivial amount of time looking for and getting context around said data. In my experience, this commonly plays out as a wild goose chase — you ask the most senior person on your team who refers you to someone else, who refers you to someone else, who tells you they don’t know what you’re talking about. You then repeat this process until you find someone who can refer you to a useful table.

person = senior_data_scientist

while person and not_tired:

response = ask(person)

if type(response) == Table

rejoice()

if type(response) == Person

person = response

But this algorithm is really pretty terrible! It’s by no means an exhaustive search, and you’ll often end up with an unmaintained, unused table. If you decide to just risk it and use this in your production pipelines, you could end up costing your company a lot of money when you find out six months down the line that this table doesn’t do what you think it does. What’s more, it’s a huge waste of time! Sometimes you’ll forget what the table does months later and you’ll have to do this again, to the dismay of your coworkers. So what can you do?

A brief intro to data catalogs and their limitations

Luckily, there’s already a solution to this problem: the data catalog. Data catalogs are aptly named, for they simply provide you with a catalog of your data. They let you search and find tables by name, view their columns and metadata, and enrich this metadata with additional documentation. A number of closed source players exist, like Alation and Collibra, and open-source versions have been released by a number of tech companies, including Netflix’s Metacat, Lyft’s Amundsen, and LinkedIn’s Datahub. And tons of companies have been creating these tools for themselves (Airbnb, Uber, Google, Facebook — to name a few).

This begs the question: why not just use/implement one of these solutions?

Well, you might want to seriously consider it, eventually, but there are enormous barriers to entry: catalogs tend to be expensive to set up, or, in the case of open-source tools, difficult to set up. For companies with only a handful of data scientists and analysts, it’s hardly worth upwards of 10s of thousands of dollars or months of work to implement a tool that will, at best, save you the few minutes of turning to your neighbor and asking him/her about that table again.

I’ve talked to a number of friends at smaller startups about this, and while the problem of data documentation and data discovery is salient even at the smallest scales, the cost-benefit tradeoff simply doesn’t make sense. But just because the problem is too expensive to solve with existing methods doesn’t mean that a cost-effective solution doesn’t exist.

So what’s the solution?

I thought, “there has to be an easier way to set up a catalog.” Day-to-day, I just want to look for my data, check the column names, and take some notes. Why is that so hard? I already can do that manually by saving everything in a team-wide spreadsheet.

So I thought I’d try to 80/20 this. The result?

Metaframe.

Metaframe is a CLI-based data catalog with a custom Markdown-based metadata backend, a fzf-based search service, and an ETL pipeline written in bash and python, built off of lyft’s amundsendatabuilder library.

But that’s hard to grok. Here’s a quick demonstration:

At its core, metaframe is just a slightly opinionated documentation tool. Want to just write notes for one table? Easy, just type the following:

mf new table_name

and you’ll enter into a text file located at ~/.metaframe/metadata/table_name.docs.md , where you can add your notes on this table. These will then be accessible in your search, by running

mf

And this isn’t tied to tables — you can create a new doc for SQL queries, ML models, jupyter notebooks, or more. I personally keep a subfolder in ~/.metaframe/ called recipes , where I store named snippets of SQL.

If you want a more full-fledged data catalog experience, metaframe also supports locally-run ETL jobs. All you need to do is specify a connection in yaml by running mf connections edit , then run mf etl , and all your data will be accessible using mf . After searching and finding a table you want to take notes for, you can press enter to edit a doc associated with the selected table, and this doc will be appended to your metadata preview window in future searches.

Because the file structure is entirely text-based, you can even navigate to ~/.metaframe/metadata and version-track it with git. In a company with a number of collaborators wanting to share notes, a simple pattern could be to set up CI/CD (or an airflow job) to populate this data in a git repository automatically and have users push their personal docs to branches in this repo. Because this is version tracked, you should be able to easily see what tables your coworkers know about without even having to ask them by checking the commit history of their personal branch.

Any caveats?

The only issue I’ve noticed with metaframe is that it’ll start to chug at around 100,000 tables. But for companies that have this sort of scale, it may be prudent to (a) not run the ETL job locally and (b) restrict which schemas are being pulled. In metaframe, this is simply a matter of adding a few flags, like included_schemas or excluded_schemas , to your mf connections edit file. See the docs for more details.

Metaframe supports manual documentation and ETL over presto or neo4j (amundsen) out-of-the-box, but we also support custom build scripts to enable custom extraction jobs (from any warehouse, from pre-scraped database stores, etc.). We are also in the process of building out more native connections (e.g. Snowflake and Postgres are in the works). If you’re passionate about getting this to work for your particular setup, feel free to drop an issue and we’ll help out. Or, if you’re so inclined, make a contribution! :)

Summary

Data documentation and data discovery are serious problems, but they’re not things that data scientists typically think about. It’s easy to neglect a problem that wastes less than an hour a week, but I’m here to tell you that there are solutions for this at all scales. Give metaframe a try, and I guarantee you once you have a taste of it, you won’t be able to live without it.

Repo: https://github.com/rsyi/metaframe