How the Data Stole Christmas
Tags: Data Science, Big Data, Dataops, Analytics, DevOps

by Anonymous

The door sprung open and our three little ones joyfully leaped onto the bed, waking my wife and I from peaceful dreams of slumbering sugarplum fairies. Oh Gosh, it was 6:03am, but who could blame them. The kids wait for Christmas day all year.

I sat up as giggles thrashed with reckless excitement on the bed. My feet found slippers and I lumbered down the hall in the direction of the coffee maker. I could hear the tearing of wrapping paper and whoops of joy coming from the living-room tree.

This scene was playing out in countless ways in households all across America. Christmas is a time for family, friends and a relaxing respite from the busy work calendar. Unfortunately, it was all about to come to an abrupt end as my cell phone interrupted our family time with an impatient buzz. “Who could be calling me today?”

I should have powered it off and tossed the damn thing into a snowbank, but as the manager of a data team in a global e-commerce company, I was used to taking calls at odd times. I pressed talk and reluctantly held the phone to my ear.

It was my co-worker. All hell was breaking loose. The back-end for the e-commerce site had crashed. It was all-hands-on-deck to get it back on-line. Our VP wanted hourly status reports — the next one in 47 minutes. I jumped into some clothes, grabbed a coat and mumbled to my disbelieving wife that I would be back in a couple of hours. To make a long story short, I actually didn’t make it home until 2 am the next morning. Yes, we got the site back up, but I missed Christmas that year.

Lessons Learned

The most tragic part of this story is that it didn’t have to be this way. We had talented people on our team, but our approach to data operations in those days was based on a flawed methodology:

· We had one instance of the production environment. When the data team needed to make a change, like for example, updating a schema, we did so directly on the live operational system.

· Making changes was so fraught with risk that we instituted heavy-weight procedures to control it. This slowed innovation to a crawl and even with all the triple checking, outages still occurred.

· We tested new changes to the best of our abilities, but since our development systems used a simplified tools environment, we would encounter unexpected errors when moving code into the more complex production environment.

· Our test data was perfect whereas production data is notoriously messy. Production data always threw unexpected anomalies at our code.

Managing a continuing succession of outages while trying to keep development projects on schedule and under budget is like trying to play “whack a mole” while simultaneously reading a book. Yet, our approach in those days was mainly based on hope and heroism. We release code and “hope” it doesn’t break anything. When there’s an outage, we call in the technical experts (the “heroes”) to work around-the-clock to fix the problem. Looking back, this was no way to run a major operation. It’s not really a surprise that the head of our department got replaced about every two years. Executive management needed to pin the responsibility somewhere.

DataOps — Lessons Applied

It is tempting but simple-minded to blame outages on people. A robust business process eliminates errors and improves efficiency despite the fact that error-prone humans are involved.

When software companies (Netflix, Facebook, …) started executing millions of high-quality code releases per year, it offered an opportunity for data organizations to renew their approach to development and operations. The methodologies used by these software engineering organizations — Agile development, DevOps and lean manufacturing — apply equally well to analytics creation and operations. The data industry refers to this initiative as DataOps.

The fastest way to institute DataOps methods, even assuming the preexistence of a legacy tools and technical environment, is by aligning and automating workflows using a DataOps Platform. A DataOps Platform offers these capabilities:

· Minimizes cycle time — A DataOps Platform aligns production and development environments using virtualization and orchestrates the DataOps testing, qualification and release of new analytics code with a button push. Continuous deployment of new code is how software companies produce such a high volume of releases per year.

· Eliminates errors — The functional, unit and regression testing of code enables new analytics to be deployed with confidence that it will work as promised in operations. In addition, the data that flow through operations is tested and subject to controls at every step of the operations pipeline. Data errors are trapped and remediated before they corrupt charts and graphs.

· Fosters collaboration — DataOps integrates version control and workflow tools like Jira. The DataOps Platform enables team members to share analytics components, encouraging reuse and improving productivity. Geographically dispersed teams can use their own choice of toolchains while fitting into higher-level orchestrations.

The code and data quality supported by DataOps minimizes the unplanned work that can disrupt a data engineer’s weekend or holiday. With a DataOps Platform, enterprises can move away from relying on hope and heroism. Continuous deployment fully tests and deploys new analytics eliminating time-consuming and error-prone manual steps. Tests and statistical controls ensure that data is error-free before it flows into models and analytics.

Have a DataOps Holiday Season

With DataOps in place, tests monitor the data flowing through operational systems 24x7x365. While data scientists are home sipping eggnog, DataOps works overtime to keep operational systems up and running. I may have missed a Christmas celebration with my family that one time, but with DataOps, never again. Happy holidays.

For more information about how a DataOps Platform can compress your new analytics cycle time and eliminate data errors, please give us a shout at datakitchen.io.