The Favorite Part Of My Job As A Data Scientist
By: Adam Ross Nelson
Tags: Cartoon, Data Science, Career Advice, Data Analysis, Careers

The Favorite Part Of My Job As A Data Scientist

The one part of my job as a data scientist that I love above all other parts

TLDR: This article wasn’t designed to provide technical knowledge or insight. Below I tell a story about selecting a stratified sample, calculating sample weights, and sharing that information at a workshop for colleagues and co-workers. This is a story about how data scientists can use their non-technical know-how to help their company build strong data-savvy cultures.

Introduction

I am unreasonably obsessed with the statistic that data scientists report spending 80% of their time finding, cleaning, and organizing data according to an admittedly modest sample analyzed by the Harvard Business Review. It is a good thing I don’t mind finding, cleaning, and organizing data. For me, finding, cleaning, and organizing data is part of the thrill.

But my favorite part of being a data scientist fulfills me more fully than many aspects of being a data scientist. The favorite part of my work is helping others connect in new ways. I have the privilege of serving in a role that tends to get a lot of attention from a broad range of others serving in a wide range of other roles.

Face it, being a data scientist gets the attention of others. My favorite days are the days I get to use that attention to help folks who work in broadly different roles connect. This article tells a story about one of these days.

One Day

A colleague asked me if I would share with her team the steps involved in selecting and analyzing a sample. I said sure, and we planned a brown-bag program. As it turns out having an opportunity to share and explain my work is also a part of the role I enjoy.

Photo Credit: Original attribution unknown. Author sourced and adapted from multiple online publications. An empty conference room. What I thought would happen the day I planned a brown-bag presentation related to selecting a sample.

Good thing too because sharing and explaining your work as a data scientist should be a routine experience. Scientists call sharing and explaining their work, dissemination.

Because selecting and analyzing a sample is a relatively mundane task I was surprised by the request for a brown-bag on this specific topic. There would be pizza for those that attended the presentation so we used a signup sheet to make sure we ordered enough pizza. My surprise grew when the signup sheet was full of personnel from all operational areas throughout the company. Both technical and non-technical colleagues were interested in the topic.

This was an unanticipated opportunity to bring colleagues from areas of operations that wouldn’t usually interact except at company-wide meetings. I was happy and humbled by facilitating that connection.

The steps involved in selecting a sample, in this case, what would be a weighted sample, are not riveting. Having learned from the lessons of one of my worse, albeit relatively harmless, mistakes in data science, I agreed to address this topic even though I regarded it as a topic in which others might not be interested. For better or worse, everyone was about to get an ear full on the steps associated with selecting and analyzing a sample from a population.

The Setup

At the time I worked for an organization that produced a widely used software solution. Its user base was in the low millions. Modest by many software standards.

Despite the modest user base it was important to select a sample of those users for analysis. To select this sample I segmented our userbase into subgroups linked to important demographic characteristics. Then, to ensure we could make conclusions about users from sparsely populated segments I over-sampled some segments and undersampled other segments. Later during analysis, I weighted the results so that we could generalize the findings to our entire user base. Staff and coworkers throughout the company asked to learn, in detail, about these methods and analytical techniques.

The Day Of

In attendance would be both technical and also non-technical personnel. There would be personnel who were years into their career and others still in college.

It would be important to deliver content that would be interesting and meaningful for an audience with broad backgrounds and diverse perspectives.

Bringing audiences together in these formal, and also in informal settings, is one of the favorite roles I perform as a data scientist. Despite the enthusiasm I had for the opportunity and also despite the enthusiasm others demonstrated for the opportunity, I had a few nagging doubts. I worried.

Photo Credit: Author sourced and adapted images from multiple sources. What I thought (or worried) would happen the day I planned a brown-bag presentation related to selecting a sample compared to what actually happened.

I imagined this presentation to be about as interesting as it might be to watch beige paint drying. In the conference room that day there was standing room only.

An Anecdote, When Others Connect

Partway through the presentation, I demonstrated a code block that calculated sample weights. That code block looked something like:

A code example that can be used to calculate sample weights using Python and a Pandas data frame.

This code gets the job done. Also, this code is optimized to be human-readable. I needed everyone in the room to understand, at least at a conceptual level, what this code accomplished. Also, as it turns out, this code isn’t much different from the code I used at the time I conducted the analysis.

In the conference room that day there was standing room only.

Because I optimized the code to be human-readable I sacrificed its ability to accomplish similar tasks without extensive modification on other projects that may also have involved selecting a sample. This sacrifice was appropriate because I wanted the code to be easily interpreted by myself, or others who might later review my work, long after dissemination of the results. This choice to write the code in a manner that would be easily human-readable supported the ability to verify or replicate the work.

I said aloud in the presentation that I did not plan to accommodate ‘edge cases.’ The reason I did not plan for ‘edge cases’ is because I wrote this code for a single user. I explained that I was not concerned about edge cases because I was the edge case. This comment got some laughs. I wasn’t trying to make a joke. But it triggered multiple humorous sensibilities.

One participant asked, “what is an edge case?” I read the room. I got a feeling there were others in the room, who knew what edge cases were, and who wanted to speak up. So I closed my mouth and let the last third of the brown-bag be a conversation between and among those in attendance.

Conclusion

There are many reasons to enjoy working as a data scientist. If you are a data scientist you have to enjoy finding, preparing, and cleaning data. You also have to enjoy the thrill of sifting through a choice of methods. The time spent comparing results and interpreting output is also a fulfilling aspect of working as a data scientist.

For me though, the favorite role I perform as a data scientist is to help others make new connections. I enjoy helping others make new connections to other professionals. I also see my work as an opportunity to help others connect to their past and present work but in new and exciting ways. I leverage the popularity of data science, whether data science deserves that popularity or not, as a means to help others make these new connections.

This was a story about how time I used the methods and techniques associated with selecting a stratified sample, and calculating sample weights, as a common point of interest that brought some of my colleagues together in new, unanticipated, and surprising ways.

This article wasn’t designed to provide a technical overview. If you’re interested in an overview of code samples that I would use to select a sample, calculate the sample weights, and then conduct an analysis shoot me a note on Twitter.com/adamrossnelson. Maybe there would be a collaboration opportunity!

Thanks For Reading

Thanks for reading. Send me your thoughts and ideas. You can write just to say hey. And if you really need to tell me how I got it wrong I look forward to chatting soon. Twitter: @adamrossnelson LinkedIn: Adam Ross Nelson.