ML for “poets” — Abstractions, Conceptual Compression, and Commoditization
By: Osman, Ozzie
Tags: Machine Learning, Software Development, Programming, Computer Science

Two is an abstraction.

One wonderful thing about technology (especially software) is that over time, things that are hard become easy. They become accessible. The hard stuff gets abstracted away at lower-levels, and you can build software faster and more efficiently.

David Heinemeier Hansson (DHH) has a piece where he calls this conceptual compression. I’m not a huge fan of that term (it doesn’t really capture the meaning), but I can’t think of a better one, so I’ll just alternate between “conceptual compression” and “abstraction commoditization”.

Anyway, one example DHH gives, which is probably relatable for most software engineers, is the idea that if you’re building database-backed applications, you used to need to understand your database engine. But over time, databases have been abstracted away and that is no longer the case. He cites memory management as another area that has been abstracted away.

Of course, just because you can get away with not understanding your database doesn’t mean you should. It’s easy to err on both sides — some software purists are insulted that someone can build an application without “understanding the fundamentals”, even if it might be the more pragmatic thing to do. If you are one of those people, I hope you’re writing your code in assembly (or binary, if you’re a true purist). After all, all languages are abstractions on top of 1’s and 0’s. It’s abstraction all the way down.

On the other hand, you can really make a mess of things if you’re completely oblivious about what’s happening at lower levels. But learning when and how to “cross the abstraction barrier” is probably a subject for another post.

This abstraction or commoditization that occurs in software is a really magical thing. A driving force of that is the open source movement. Someone notices they need to do something over and over again, and that thing is difficult. They build a tool to make it easy for themselves to do that thing. They open-source that tool so others can use and improve it. Of course, it’s not always that noble; open-sourcing can happen for a variety of more material reasons, too, but the result is still the same. Commoditization in other industries tends to be a lot less noble, and usually driven more by market forces and competitive dynamics. People don’t often provide tools for free. They realize that it’s better to be the weapons supplier than participate in the war.

In software, the overall trend of things seems to be towards more “compression”. One way to observe this is to look at a particular company’s approach, so we’ll look at Google (and some Jeff Dean-related projects).

For instance, early on, Jeff Dean helped architect Google’s MapReduce tool. MapReduce was published as a paper in 2004, but most of Google’s tooling around MapReduce was proprietary and never really released for open-source or paid public consumption. To fill the vacuum, Hadoop emerged, inspired by MapReduce and Google File System, but developed largely outside of Google (with major support from Google competitors like Yahoo!).

To take another data point, Jeff Dean was also involved in Spanner. Spanner was also intially published as a paper (in 2012). It was never open-sourced, but eventually it was provided as a paid, hosted service within Google’s Cloud ecosystem.

Most recently Jeff Dean has been heavily involved in TensorFlow. Google uses TensorFlow internally, but has also open-sourced it (with libraries in multiple languages), provided it as a cloud service, and even provided custom hardware that can be used for more efficient computation (TPU) as a paidservice. It’s pretty amazing to see this progression from having really awesome tools that are only used internally to tools that are designed for public consumption.

In 2005, unless you worked at a company like Google, if you wanted to run large distributed computing, you really could only read the MapReduce paper and fantasize about what it would be like. You didn’t have access to the libraries or computing power. In 2018, if you want to run deep learning algorithms, it’s all at your fingertips. There’s a Google TensorFlow codelab called “TensorFlow for Poets”.

And it’s happening at a lot of other companies. Microsoft goes from “Linux is a cancer”, the “Halloween Documents”, and “embrace, extend, extinguish” to opensource.microsoft.com and being a top contributor to open-source.

I don’t think companies are doing this out of the goodness of their hearts. There are good business reasons to contribute to this commoditization.

One really interesting area where this abstraction commoditization is happening right now is Machine Learning. It’s getting easier and easier to do Machine Learning, and for someone who isn’t an ML specialist, that’s really empowering. There’s a few things that make the process of ML commoditization really exciting to watch and hard to predict:

The field itself has been around for dozens of years, but has picked up a lot of momentum in the past few years, due to abundance of data and compute, break-throughs in hardware (e.g. GPUs), and a flurry of activity in the field. There’s all sorts of interactions between academia and industry, and people who have been doing this for ages and people with new ideas. This can result in interesting debates and drama.

The field is moving really quickly. For proper commoditization to occur, some level of stability is needed. Otherwise, you’re trying to hit a moving target —and the ML target is moving really, really quickly.

There are a lot of “gorillas” in the room with different interests. Companies like Google, Facebook, Microsoft, and Amazon are battling it out.

ML is critical to their ability to compete and build their own consumer products, so they are investing heavily within ML internally and hiring the best ML engineers they can find.

To do that, they need to prove that they’re an attractive place to work on ML, and so that means they publish papers, give talks, and build tools like TensorFlow. They’re also helping sponsor and develop ML curriculums to build their credibility.

Finally, they are also all competing in the Cloud space, and providing ML tools/libraries (like SageMaker, Tensorflow, or Pytorch) and computing resources (like TPUs or Deep Learning AMIs) to help get developers on their platforms.

In fact, a lot of times these goals might be at odds with each other. For instance, publishing breakthroughs can build credibility within the industry but might give away some “secret sauce”. Releasing open-source tools might be beneficial for the ecosystem, but proprietary tools are more likely to create lock-in to their platforms.

It’s fun to watch, and amazing if the end result is more people can easily do traditionally “difficult” things.