Excel for Data Science?. Learn how one of the oldest analysis... | by Fredrik Moeschlin | Towards Data Science

Microsoft Excel is probably the most well-known tool for working with data. Almost everyone knows it, we have our own hate or love relationship to tables, pivots, VBA-scripts and charts. We do our monthly business spreadsheets, calculate vacation costs and even work with contact lists in Excel. The word “Excel” has even become a synonym to “table” or set of tables, as in: “hey, can I send you an Excel?” Even your grandmother has used it. After all, it has been around since 1987 and still looks fairly similar with its grids and sheets. They probably got something right in that design?
But wait. Data Science? In Excel?? Isn’t data science about hyped Jupyter Notebooks in the web browser, Big Data and Hadoop in the Cloud, Deep Learning on the GPU and advanced interactive plots and visualizations, and ...? This may sound far from Excel as you know it. Well, hang on...
After achieving my Master’s degree in Physics I’ve spent over ten years applying “data science” to solve problems. Ranging from self-driving cars in the automotive industries, to risk management in insurance and predictive maintenance in MedTech, Excel has always followed along. Now, I would like to share my experience with you.
It’s time to straighten up and admit:
Excel is a powerful tool for data science.
There are probably plenty of arguments for this (and against). But before starting, let’s just clarify: I don’t have any professional relationship with Microsoft and never have had.
These are my own thoughts and experience — I may be wrong so please correct me!
There’s simply no better editor for 2-dimensional data (i.e. tables). Probably that’s why Excel has had the same layout over the years. The tables are easily edited, formatted, colourized, and shared. Google Sheets is a clear confirmation of the Excel design for editing data — but scaled for multiple users. Allowing multiple users to edit simultaneously is something I miss in Excel (after losing data on SharePoint and OneDrive too many times). The ease of just sharing a link is probably also part of the success behind data science notebooks such as Jupyter.
In Excel, however, one keypress [F11] turns your table into a chart using your company’s visual identity (given that someone took the time to set up the Theme). Add to that the built-in tables, filters, slicers, groupings, window splitting, cell formulas and other features and any alternative table or database editor stands flat. That is, as long as you are fine with a maximum of 1,048,576 rows and 16,384 columns per sheet! Not very impressive for Big Data, but for editing and looking at data in practice, this is often enough.
In a movie on perspectives on AI, DARPA used spreadsheets to illustrate how neural nets work.
Indeed, there are deep learning approaches for Excel but perhaps this isn’t the first tool you would go for. But Excel comes with an Analysis ToolPak that can be activated to unleash more advanced powers. The other day, I heard a Microsoft representative confronted with the question “what distinguishes advanced analytics from normal analytics?”. She answered “the ability to use machine learning”. Since the Analysis ToolPak does machine learning with regression and other complex data analysis, I assume Microsoft sees Excel as one of their platforms for Advanced Analytics.
Anyhow, the ToolPak feels a bit old-school, as using hard-coded functions in SAS or SPSS. Modern data science is about open libraries and transparency. Perhaps you want to tweak your Gaussian Mixture Model for some reason or understand how it is optimized? Doing this in Excel or adapting some open source code in Visual Basic for Applications (VBA) can be cumbersome (how do you even slice a vector in VBA?).
You would, of course, like to use your favourite data science scripting language (Python, R, Scala, Julia, Java, JavaScript, ...) and trusted libraries. That is just fine: you can use Python with Excel. At least if you’re on a Windows machine (which I recommend, as Excel on Mac still isn’t very stable).
To demonstrate Python in Excel, just create a new spreadsheet and enter some data similar to the screenshot here. Select all data and make a Table out of it [Ctrl+T or insert table]. The Table defaults to “Table1” if you are using an English version of Excel. The Python code below uses the package win32com to interact live with Excel’s Object Model (a programmer’s interface to Excel). Notice that you get a live connection interacting with the running program, we’re not just reading or writing to a file on disk.
Get a live connection to the running Excel and active sheet
import win32com.clientimport pandas as pdapp = win32com.client.Dispatch("Excel.Application")ws = app.ActiveSheet
Fetch table contents and headers, into a Pandas data frameNotice that we can use Ranges as well as ListObjects and similar.
x = ws.Range("Table1").Valueheaders = ws.ListObjects("Table1").HeaderRowRange.Value[0]df = pd.DataFrame(list(x), columns=headers)>>> dfProduct     Month       Income0       A   January  7358.0015831       A  February  2218.3867312       B   January  5201.1790323       C   January  1765.4530114       C  February  6261.555208
Calculate median income per product and write back in ExcelNotice that Excel always expects 2D shapes when assigning ranges:
income_med = df.groupby('Product').Income.median()ws.Range("A10:A12").Value = (('A',), ('B',), ('C',))ws.Range("C10:C12").Value = tuple((_,) for _ in income_med)
Let’s demonstrate that Python interacts with Excel liveSet a filter for the Table and select a certain range:
ws.ListObjects("Table1").Range.AutoFilter(        Field=2,        Criteria1="=January")ws.Range("C10:C12").Select()
The example above could easily be expanded to hundreds of thousands of rows without performance issues. Python gets direct access to the internal array storage within Excel. Compared to running VBA, Python and Pandas provide you stronger data manipulation abilities on a much higher abstraction level. You can also easily access scientific tools from SciPy and several other resources — but keep Excel as “frontend” for looking at data.
There are several further analysis possibilities with Excel. The built-in database engine can execute SQL queries on tables and live data sets, and the Power Query editor can fetch data from various data sources (including Azure cloud or Hadoop). There’s a lot to explore if you haven’t already.
Compared to the data science workflow, Excel presents data as it is, at your fingertips. You can do whatever you want, save as many versions as you prefer, and manipulate freely. Data lineage, the ability to track data from origin, through manipulation to a destination or conclusion, is therefore tough. In a data science notebook on the other hand, the entire manipulation process is listed as a script — that others can repeat and verify. Naturally, the step to automate or put in production is much smaller from the notebook. Not to mention dealing with version control and tracking changes.
For a data novice, the simplicity of Excel lowers the entrance barrier and allows to quickly get started with data manipulation. Functions and macros for repetitive tasks can be learnt along the way. The track of data science in a notebook appears tougher initially but will reward you when it comes to larger manipulations and more complex and free analysis.
I think that Microsoft was long before their time when they launched the Office Suite in the 90s. Excel is a splendid example of keeping all in one place: a data repository, graphical user interface, analysis tool, and a scripting environment with VBA. The open object model made it fairly easy to build extensions, add-ins and interact directly with the core of the spreadsheets. Microsoft Access is another similar, but more database-oriented, solution where all is combined. And isn’t it neat, that you can share it all as a file? Microsoft was clearly leading this self-service suite for data analysis. In a way, I think they were even too early. Or was it the work with Excel, Access and similar tools that enabled the Data Science revolution we have seen recently?
Sadly, Excel is way behind when it comes to the scripting language (VBA is built on BASIC, invented in the 60s), not to mention versioning or security of scripts. The data capacity is not impressive (even if data models can carry more than 1 M rows internally) and Excel may never reach Big Data levels (whatever that is). Graphics is another topic where Excel is far behind in comparison with open source libraries such as ggplot and matplotlib. Apparently, Microsoft puts all their efforts into Power BI for visualization of data. A lot has happened since the glorious days when Excel was the one and only tool.
Many superpowers remain. I think Excel is the best editor of virtually any 2D data of smaller scale, it is heavily used and something like a default for sharing spreadsheets. There are many possibilities for scripting and for analytics, even interactively as we saw with Python. Perhaps this is why so many organizations suffer from the “Excel hell”? It is far too easy to by-pass a stubborn business system and build your own Excel solution. (I know, I have been contributing to quite a few ;-))
Modern data science platforms could learn a lot from Excel. Data science is all about learning from data and learning from data means learning from the past. Excel has a lot of past experience to share. Perhaps a Jupyter notebook could find a better way to include data sets for sharing? Could an online data science workbench allow better interfaces to quickly editing and formatting the data visually? There are many possibilities. I think the strength lies in allowing smart storage, editing, visualization, analysis and sharing.
For me, data science has always been about combining the tools best suited to get the job done. Often Excel has turned out to be one of them. I’m continuously trying to migrate to LibreOffice on Linux (to enjoy open software and get rid of Microsoft) but still often find myself launching a virtual Windows machine to get access to the Excel superpowers. Either I’ve gotten addicted to it, or there is some hidden greatness in the design of Excel. In any case, I think a data scientist without Excel experience is comparable with a carpenter who never used a hammer. I’ll probably keep sending people Excels by e-mail for a long time. It still holds as the best format for data exchange allowing recipients to instantly do their own analysis — everyone knows it.
All in all, Excel has been around for decades and is, without doubt, the most popular tool for editing and analysing table data. Many couldn’t live without it in their daily lives. With data at your fingertips and — as we showed — direct access from Python scripts and machine learning packages, what more could you wish for? I recommend to really explore and learn Excel and you will be rewarded with impressing superpowers as a data scientist.
What is your experience?