NLP Newsletter: Special Edition! Interview with Erik Dunteman on MLOps and Building ML Platform
Tags: Programming, Machine Learning, Artificial Intelligence, Data Science, Technology

Introduction

Elvis: Briefly introduce yourself (name, experience, education, role)

Erik: My name is Erik Dunteman, I’m a software engineer and founder focused on developer tools and infrastructure.

I have a swiss army knife background — I studied mechanical engineering in undergrad and did internships across agricultural and chemical engineering before teaching myself how to code. My initial inspiration for getting into software was learning about deep learning.

I’ve been full time on angel funded software startups for two years since graduating undergrad, starting with bioinformatics and moving to cloud infra, developer tools, and now to MLOps.

Elvis: Who do you work for?

Erik: I’m a solo founder at Booste https://booste.io

Elvis: What problems are you solving using machine learning and how does it solve the problem?

Erik: My work is more on the Ops side, providing pretrained NLP models via API with simple one-line SDKs to call the model.

As of this week, I’m starting to do custom retraining on GPT2 models, to allow teams to use custom text generation models without worrying about infrastructure and API maintenance.

Elvis: How is the machine learning team structured?

Erik: I’m a team of one

Elvis: Explain with a few words “A day in the life” in the context of your role or the machine learning team?

Erik: On ML-focused days, I’m researching a model or technique that my customers request. For example, today I am working on integrating Uber’s PPLM into my API (https://github.com/uber-research/PPLM/blob/master/run_pplm.py)

I’ll clone the repos, network in my inference servers (if relevant) and run through the demo scripts until I get a grasp for how the data is being processed before and after inference. I’ll then build an API endpoint around this logic. I am generally not retraining.

Product/Technology

Elvis: What is the main service/product your company provides?

Erik: Machine learning inference as an API.

Elvis: Is there an open-source product/effort you are working on? If so, explain what it does.

Erik: The client SDKs are open source: https://github.com/booste-corporation/nodejs_client

Elvis: What is the technology stack (e.g., tools, libraries, etc.) the machine learning (ML) team mostly works with these days?

Erik: TensorFlow Serving as the inference engine, Tensorflow for matrix operations in pre/post processing, ONNX for cross-framework model conversion.

Elvis: What’s a challenging machine learning related task your team has encountered?

Erik: Economically serving the model is a struggle. Especially with large models like GPT2, it’s extremely memory intensive and difficult to justify without a large volume of calls.

Elvis: Are you working with specific cutting-edge techniques like Transformers, GANs, etc.?

Erik: Our work is all tech released within the last four years. We’ve done YoloV3, Transformers (GPT2), and are now working on PPLM which is a single layer inference that tacks onto a transformers model to help guide output to contextually appropriate words.

Elvis: Do you have or are you building your own ML platform/infrastructure? If so, please share how it helps the team?

Erik: Our whole offering is the no-code training platform (to be released) and the API serving infrastructure (currently live with pretrained models). Our customers include even ML-heavy teams who would rather focus on business logic than on model-serving and infra.

Elvis: Are you working with machine learning models in production? What technologies are useful and lessons

Erik: I run everything in Kubernetes (specifically a user-friendly Kubernetes service called https://zeet.co) and would never try to run production without Kubernetes.

RedisAI was my first production tech, and I was very excited by the idea of it. https://oss.redislabs.com/redisai/

It was economically reasonable for multi-model inference systems, since you can load all of the models at once into the server (from TF, torch, or onnx model files) , and they’d be hot and ready in-memory, ready to go. It handled parallel inference jobs really well, and I thought I’d struck gold by finding this relatively new serving tech. Unfortunately, there’s a nasty memory leak somewhere in RedisAI, so my system would crash and restart every hour due to memory overflow.

I’ve moved to Tensorflow Serving. It works well, but I unfortunately need to run each model on a dedicated inference machine, so I lose some economies of scale that the unified RedisAI server gave me.

Elvis: How does the ML team stay informed with the latest machine learning developments/research?

Erik: Word-of-mouth.

Elvis: How does machine learning contribute to the overall business objective?

Erik: It allows us to offer more complex inference options to our users. IE: learning BERT and getting it into production means we can now offer a BERT API as a new feature.

Elvis: Do you often work with data? If so, what are the best practices for working with data (e.g., sanity, cleaning, feature store, etc.)?

Erik: I’ll clean data once a month without using many prebuilt tools, since the focus recently has been just on pretrained NLP models and the Ops side of things.

Opportunities

Elvis: Are you hiring for machine learning related roles? If so, for what roles specifically are you hiring?

Erik: I’m a solo founder and happy to chat with potential cofounders who are more experienced in the training/research side of ML, to complement my infrastructure experience.

Elvis: What level of experience are you looking for in applicants?

Erik: 1 year of ML research experience is ideal, though since it’s a cofounder role: grit, energy, and willingness to learn on the fly is more important than credentials.

Elvis: What incentives/opportunities/activities do the company provide for employees in terms of career growth and learning?

Erik: Learn how to be a founder, ship faster than you ever have before, own a cofounders share of a venture-scaleable company.

Elvis: Are there non-technical AI roles at your company? If so, what are these?

Erik: None

Elvis: How can candidates reach out to you if they are interested in these roles?

Erik: DM me at @erikdoingthings on twitter