Five Years of Scala, Part I: A Retrospect
By: Guntur Akhmad
Tags: Tech, Scala, Startup, Retrospectives, Engineering

We grow from zero to something with Scala, here comes our retrospection.

Our backend started with Scala monolith back in 2015. As time goes we split it into several microservices — still using Scala — to achieve faster release cycle and every other great things.

In this post I want to do a retrospection of Style Theory’s backend technology that this month marks its fifth year since the first commit; from what we did right, what went wrong, and our current approach to achieving better future.

Happy Days

For years, all of our backend engineers understand Scala, our Tech Leads too. We’ve had enough of engineers and time to translate business and product requirements into Scala code. Releases hit every user story we had.

Using Lagom Scala framework helps — big time — with idiomatic code structure and built-in features most microservices need. Our release cycle becomes faster, hotfixes can be deployed faster and in a non-breaking way.

We crafted a shared library and Scala plugin that consists of reusable functionalities so that each of our Lagom microservices can leverage it to shorten development time.

Inter-service calls are hassle-free, just import other service’s API project in our build.sbt, and call any endpoints easily by invoking functions from the imported API project.

Creating unit and functional tests is a breeze thanks to the syntactic sugars that Scalatest and MockitoScala offer, mocks are easy to do, reviewing pull requests that contains many tests was easy.

What Went Wrong?

We expand our business verticals that was only apparel rental into bags rental and then preloved apparel and luxury bags reselling. We also expand our business from Singapore to Indonesia, and then Hongkong.

With growing business, comes great tech debts.

(1)

Features we need to implement was increased enormously, we basically hit the upper limit on how much we can contribute before we plunge ourselves into the burnout abyss.

Our engineering center is at Jakarta, Indonesia. We were short on backend engineers since hiring was not going smoothly mostly because we went the Scala road. In fact, the supply of backend engineers that have previously work with Scala in Jakarta area is far from enough from our demand.

It is true that everyone can hire backend engineers that did not have a previous experience with a specific language and we do exactly that. It was previously said that new hires can make their pull request within a week of work, but the reality of the situation are:

Such outcome does not apply to all scale of contribution, every person can make pull request but in the end quantity is not what really matters. Since our product goes bigger than ever, all of engineers busy with implementing features and fixing critical bugs, this makes on-boarding process becomes crippled and far from ideal. The steep learning curve of Scala for those who haven’t experienced functional programming made this worse; most of our backend engineers came from non-functional languages. Averaged, it took a month for our freshly-onboarded engineers to be able to comprehend Scala basics. The statement in the previous article is valid at the time it was written, but it doesn’t stand the test of time.

Stand!

There are other languages that are interesting, and sexy, and exciting. Scala is a great example, but trying to understand what happens in a Scala program, it takes a PhD. — Brian Ketelsen, Microsoft.

(2)

With a bigger product and more business verticals to handle, then a huge amount of production bugs, service issues and outages was popping here and there.

Debugging issues in microservices is not easy, it went worse because without a good distributed tracing capabilities in our current microservices implementation, we cannot properly detect which specific service(s) that causes other service(s) to fail. Due to this shortcoming, oftentimes we blamed the incorrect service(s) in the RCA document. This is not healthy.

We are short on time and man power to do research and tinker around with Scala — and Lagom — inner workings to achieve custom functionalities to fix all of distributed tracing issues from the ground up.

All we can do was try our best within our knowledge: optimizing SQL queries, rework our codes, re-allocate service resource budgets, re-assign our containers into their own node pools in Kubernetes. These approaches only fix some issues in a short term manner, while incurring a tech debt in the longer-term.

(3)

Inter-service communication went nuts since we underestimated this by relying on code-based API contract. For example, we import Service X API project to call their defined endpoints from Service Y implementation project.

Some times this approach eases inter-service integration since everyone can just import other service’s api-project to their build.sbt. This approach surely eliminates the need to write additional code to call other services.

Happy dependency.

However, along the way this gave birth to the dreaded dependency hell.

Service X calls on Service Y, while Service Y calls on Service Z, then Service Z calls an endpoint in Service X; while this kind of dependency creates no fundamental problem in microservices design, on the other hand this creates circular dependency in the code-level. All hell breaks loose every time we need to bulk update several api-project in these dependency chain.

An illustration of a cyclic dependency we came through.

This tech debt arises because Scala permits cyclic dependency and we did not aware of this feature. This negligence due to the fact that we need to get things done in a timely manner while there’s no person that can prevent this wrongful action from even happening. I tell you this again: everyone was busy implementing features and fixing bugs.

(4)

Writing tests is an integral part of our backend engineering. We grow our product from just a handful into this huge product now. With every feature we add or change, we modify or create several unit tests, functional tests, and also integration tests.

Writing tests in Scala using ScalaTest and MockitoScala is fun because their feature is complete and they offer syntactic sugars everywhere so our test suites are easy to read and modify.

Back when every microservices we manage only got a handful of features, running test takes no significant amount of time, but today most of our microservices are larger than ever before, test runs take so much time and felt too slow, they also often fail rather mysteriously.

We don’t really know if we wrote our tests incorrectly or just a configuration issue but with our current test configuration, we waste so much time running our tests in our local machine because they are super slow. They are terribly slow that we can arrange a literal ping pong tournament in our Senayan office — of course, it was before COVID-19 happened, now we can only talk to our pets or plants since we go full work from home.