Replacing VBA with Java in Excel. Fall in love with Excel again with the... | by Tony Roberts | Towards Data Science

Excel is ubiquitous in nearly every workplace. From top tier investment firms and large scale engineering companies right down to individual sole traders, people get work done using Excel.
This article will look at some of the problems and advantages of using Excel, and how using Java embedded in Excel those problems can be overcome.
You don’t have to look far to find criticism of Excel and cases where its mis-use has resulted in heavy losses to companies. Over the last few years there have been many cases where bugs in Excel spreadsheets have been at least partly to blame for embarrassing and costly errors.
www.theregister.co.uk
www.businessinsider.com
With such risks, why do companies still use Excel, and what can they do to prevent similar situations?
The main reason why Excel is so heavily used is worker productivity. Someone using Excel can do an incredible amount of complex work far more effectively than with any other tool.
Software developers often argue that the same can be achieved in their favourite programming language. Technically they may be correct, but that assumes that everyone has the time and appetite for learning to be a developer (which takes most of us many many years).
Most companies don’t have the resources to have developers dedicated to each business user, and even if they did then communicating what’s needed and iterating to get the desired outcome would struggle to compete with an individual ‘knocking together’ an Excel spreadsheet.
What is it about Excel that makes it prone to errors? Nothing in itself, but there are various things that developers are used to that reduce the same sorts of risks in non-Excel based solutions. The following is a list of a few weaknesses in how Excel spreadsheets are typically developed.
Too much complexity. A spreadsheet may start off fairly simple with a few cells and formulas. Then bit by bit it grows and grows. Ranges get duplicated to handle more and more cases or multiple sets of data until it’s hard to reason about what’s going on. The task may still be reasonably simple, but because of the duplication required and the fact that each cell can only hold one unit of data a spreadsheet can sort of explode into something way too complex.
Minimal or no testing. Excel spreadsheets are subject to very little in the way of tests. The are typically no unit tests written for VBA code, and the only functional tests are often by the author of the spreadsheet on a limited set of inputs. When another user has to use the same spreadsheet there’s a high chance because they’re not familiar with the nuances of how it works they’ll stumble across some error that was never tested for.
Stale or outdated data. Connecting Excel to external data sources can be tricky. It can connect directly to a database, but what if the data you need is produced by another system, or you don’t have direct database access to the data you need? Data is often copied and pasted from reports from other systems to Excel, often with no way of knowing when the data was last copied or even if it is complete.
Bugs spread out of control. Even if you know there is an error in a spreadsheet, and you’ve figured out how to fix it, how can you find all the instances of other spreadsheets that have copied and pasted the same bit of VBA code, or even just copies of the exact same spreadsheet? The chances are that you can’t. Spreadsheets get copied and emailed around, and there’s no separation between the spreadsheet, data and code.
Version control hell. When you’re working on a large spreadsheet and you get to the point that it’s stable and working, what do you do? Most likely the answer is you save a copy — maybe with the date added to the file name. That’s about as far as version control goes in Excel. What if a change has unintended consequences? How do you know when that change was introduced? It’s almost impossible!
These problems all stem from taking something that on the surface is quite simple (a grid of related numbers), and pushing it until it’s extremely hard to reason about its behaviour and correctness.
In essence, Excel is a way of expressing relationships between things. A1 is the sum of B1 and C1, for example. Where it starts to go wrong is when those relationships become more and more complex.
If you wanted to compute “A1 is the variance of daily returns of time series X”, what would that look like in Excel? If you are an experienced Excel user you might be imagining a table representing time series B with extra columns for computing the returns and a formula to compute the variance. But what if now we want to compute the returns for another N time series? Copy and paste the formulas for each new time series? This is how errors start to creep in!
Much better is to encapsulate the algorithm of computing the variance of the daily returns of a time series into a function. Then we can call that repeatedly for as many time series as we want without risk of one of the intermediate cells getting edited or not copied correctly.
Now imagine that instead of a table of data, a time series could be represented by a single cell in Excel. If that could be achieved then we’re back at a simple relationship between two cells — “A1 =daily_returns_variance_of(B1)”. Suddenly our spreadsheet starts to look a lot less complex!
We still have the problem that the time series data has to come from somewhere. Rather than copy and paste from another system or database, what if we had a function that loaded a time series from that the system or database directly? That way, each time we calculated the spreadsheet we’d know that the data was up to date and complete! To continue the previous example, we might have “B1 = load_time_series(ticker, start_date, end_date)”. We’ll come on to how exactly we can store a whole data set in a single cell later.
It often won’t just be the person using Excel who writes the functions they use. By providing end users with a solid set of Excel functions, technology teams can support the business more effectively than if they just write applications with only shallow Excel integration, like exporting reports.
By thinking about our spreadsheet and putting algorithms into functions, and by fetching data directly instead of copying and pasting, we’ve addressed a couple of the big issues with Excel spreadsheets. We haven’t yet touched on how those functions can be written, and the problems around testing, bug fixing and version control.
If we were to decide to write all of our functions in VBA (and believe me, many people do!) then we wouldn’t be taking advantage of any of the advances in software development made in the last 20 years!
Java has kept apace with modern software development and has a lot to offer over VBA.
Testing. Java has lots of different testing frameworks, all with different strengths and weaknesses. Whichever you choose though, being able to run automated test suits across your code base gives you confidence that it’s doing the right thing. This simply isn’t possible with VBA.
Extensive Library Support. Writing VBA code is often a case of writing quite standard algorithms found online and converting them to VBA. Want to do something trivial like sort an array of data? In Java that’s no problem but in VBA you will be responsible for making sure your sorting algorithm works, and without any testing. Now imagine writing a complex derivative pricing model!
Keep code outside of Excel. VBA code is usually saved inside the workbook, which is why when sharing workbooks bugs become so hard to track down. If your spreadsheet instead references a compiled Java library (JAR), then that is external to all the spreadsheets that reference it and can be updated easily.
Version Control. Java source code is just text, and so can easily be checked into a version control system. Most Java IDEs have excellent support for this as it is a standard part of modern software development.
Development Environment. The VBA editor (VBE) hasn’t changed in years. It offers little more than a very basic text editor with rudimentary debugging capabilities. Java on the other hand has a range of excellent IDEs to choose from.
That’s true, but Excel has a concept of “add-ins” that allow developers to extend Excel’s functionality. One such add-in is Jinx, the Excel Java Add-In.
Using Jinx, you can completely do away with VBA and write worksheet functions, macros and menus entirely in Java.
Writing a worksheet function in Java is as simple as adding Jinx’s @ExcelFunction annotation to a Java method:
package com.mycompany.xladdin;import com.exceljava.jinx.ExcelFunction;/** * A simple Excel function. */public class ExcelFunctions {    /***     * Multiply two numbers and return the result.     */    @ExcelFunction    public static double multiply(double x, double y) {        return x * y;    }}
You can return all the basic types you’d expect, as well as 1d and 2d arrays. For more complex types, you can write your own type converters, or you can return Java objects directly to Excel as object handles to be passed in to another Java method.
Jinx is free to download. See the Jinx User Guide for more information about how you can use Java as a replacement for VBA.
Jinx functions can return all the standard types you’d expect (ints, doubles, arrays etc.), but it can also return Java objects! When a complex Java object (like a class representing a time series loaded from a database) is returned it will be returned to Excel as an object handle. That object handle can then be passed to other Jinx functions, and the Java method will be passed the original Java object returned from the first function.
This is an extremely useful technique for simplifying spreadsheets to keep the complexity of the data involved away from the spreadsheet. When needed, the object can be expanded to an Excel array using a Jinx array function.
You can read more about these object handles in the Jinx Object Cache section of the user guide.
These techniques aren’t unique to Java.
The same could be said for other JVM languages like Scala or Kotlin. Jinx works with all JVM languages, not just Java.
Another popular language for writing Excel add-ins is Python. This can be achieved using the PyXLL add-in for Excel.