How to migrate your company to a new product overnight
By: Maciej Szkamruk
Tags: Startup, Mergers And Acquisitions, Programming, Product Management, Merger

Photo by Kevin Ku on Unsplash

If the platform you develop is being used by millions of people every month, it’s hard to imagine how challenging undertaking would it be to move it to a completely new version or to merge it with another service that you’re performing fusion with.

Over the web, you can find plenty of articles that most of such projects fail. For more than two years, I took part in one of these projects.

Back then, we had two platforms to serve our users in different countries. Our goal was to migrate them to a single one, without causing a disaster or significant downtime. In the below article, I’ll share with you how we did it and what we have learned during the whole process.

Revolution starts from the business side

Startups’ acceleration process in a nutshell is grabbing more and more money from the investors in a series of financial rounds. If the startup is good enough and grows well, it can count on even millions of dollars of investors’ money.

I’ll never forget one of our all-hands meetings back in 2016, during which, our CEO Mariusz, told us that we just closed another financial round and we will use acquired money to perform a fusion with our then biggest competitor — Spanish Doctoralia.

We used to do fusions of other smaller companies in the past, but we never stood in front of such a great challenge. Soon after sharing the news, our minds were full of not only happiness but also anxiety.

Doctoralia had a great product and a team of outstanding people, including a well-scaled IT department. With ease, they managed to win their home market, a couple of other European ones, LATAM countries, and Australia. At the same time, Docplanner was present on a handful of other European markets as well.

Why joining our products had a lot of sense

Both services allowed to book a doctor’s appointment, write opinions about these doctors, or ask them some questions. Of course, they both had some small differences in the business logic, but key features were basically the same.

Doctoralia was developed in .Net, while Docplanner in PHP. It was hard to imagine further product development in this scenario — it would end up writing the same functionalities twice, solving a doubled amount of bugs. It wouldn’t work at all.

Another problem is data architecture that, in large projects, usually tends to lean towards not only best practices but also a chain of product development decisions. Even if both platforms have entities like user, doctor, appointment, they are represented by totally different data structures. If one wants to migrate them between the platforms, data transformation is needed for every entity.

For the above reasons, a lot of companies restrain from performing a fusion and prefer to just do a takeover of both software and people who support it, expanding the markets as one corporation, but with a handful of independent business entities full of people that do not share the same work culture or even do not know each other at all.

This is not the way we run our business. Because we believe that, in the long term, people and good relations are way more important than accountancy, we didn’t want to just take over another company, earn more money, and move on. We chose to make a fusion of our products, to share our past experiences, and to develop a single, uniform product together.

This change took us almost two years. But from today’s perspective, I’m 100% sure it was worth it. Maybe most of the company mergers fail, but hey, we made it and it was the best decision we could make.

Assumptions and naive estimations

Our product is divided into two parts — one for doctors, which we call SaaS, and the other for patients, called the marketplace. At that time, SaaS was at the beginning of its shiny road, thus didn’t require much effort to make it work in all of the markets we had.

Marketplace, on the other hand, was a huge challenge. Both Doctoralia’s and Docplanner’s implementations were very large, taking literally millions of lines of code and handling huge amounts of data.

To handle the migration with ease, we created a new dedicated team — Merge. We took the following assumptions:

We have 11 markets to migrate Migrate one country at a time, not splitting it into phases Start with smaller markets first. It’s safer in case of failure. If the upcoming market has some functionalities on the Doctoralia side that we lack in the Docplanner marketplace — first let’s implement these before migrating. Aim for as less downtime as possible (a couple of hours max). Merge team offers full support for the migrated market for the first two weeks after the process is finished.

Looks naive? Maybe, but this is exactly what we did. The only thing naive in the end, was our estimations because initially, we thought it’s gonna take a couple of months to finish the job.

Market migration — MVP approach

One of the golden rules for developers is to not reinvent the wheel. While it works best for most of the cases, it was the main reason for our two big failures that we “achieved” during the migration of the first couple of countries.

Our first approach was what startups really love — MVP. We used to shovel data in the CSV format in the past, so why shouldn’t we act differently this time?

From practical experience, it looked quite simple — we extracted data from Doctoralia’s database, transformed it into the format we support in Excel, then pulled it in using importers we created. I must say that for smaller data batches it worked pretty well, giving our PO the ability to import data back and forth in the testing environment whenever he wanted.

The CSV rollercoaster allowed us to migrate data of the first two markets. But, because they were at the same time markets with the lowest amount of data, with the third country we hit our first wall.

It turned out that, even on a relatively powerful machine, there’s a limit of data that Excel is able to process in roughly reasonable amounts of time. When we started to experience more and more freezes, we decided to drop the CSV approach and look for something better.

We still didn’t want to reinvent the wheel though and to write everything from scratch. This is why we started to check what current ETL solutions had to offer.

After a brief search and a couple of meetings, we decided to give Xplenty a shot. This cloud-based tool, apart from the extensive data-manipulation feature set, offers awesome UI to manage the whole process. As backend developers, we’re not “clickers” by nature, but the ability to set up different data flow scenarios in such a way totally bought our hearts. To top that, this tool wasn’t entirely new to our organization, as our Business Intelligence department used it and highly recommended it.

As I just wrote, Xplenty is great. It is so great, that we resigned from it much later than we should — around 2 months of the data-modeling nightmare we led ourselves into. Of course, we gave it a test drive before we opted in, but were simply too optimistic and restrained from performing more complex testing.

A tool that was so simple to use, turned against us when it came down to migrate data with more complex one-to-many relationships. One day, looking at the screen below, we finally opened our eyes and decided to resign from using it.

Importing doctor’s basic data in Xplenty

I’m still too afraid to calculate how much money we lost. I don’t even mean the payrolls of a group of really talented developers, but time impact on the whole company that awaits our fusion to be completed.

Morale dropped a lot, but the ball was still rolling and we didn’t want to let it go.

Third time’s a charm — do reinvent the wheel

The lesson from the above chapters is simple — some special cases require tailored solutions, so if you want to get the job done right, do it yourself from the ground up.

Three main pillars of technology able to complete such a broad task are easy scalability, easy development, and easy monitoring.

Easy scalability

CSV lesson taught us that we have to be prepared for larger and more complex sets of data. With every next market, it’s gonna grow, and we have to keep in mind our promise — to migrate everything overnight.

To give you a sneak peek of what we achieved — with the CSV approach it took us around 1 hour to migrate 16.000 users and 34.000 doctors. With the custom solution I’m about to describe, migration of the Brasil market that held 2 million users and 600.000 doctors, took roughly 16 minutes.

Easy development and maintenance

The only way to built tools like these is to separate them from business code as much as you can. For the whole time, we were working on a living organism — marketplace codebase maintained by dozens of other developers every day. We had to keep in mind that we cannot stop other teams from developing our product while we work on the next market migration.

Easy monitoring

Without proper monitoring of the migration process, you are blind. Of course, testing the app thoughtfully after data migration is super important, but having proper tools to monitor the migration process in realtime is crucial.

Let’s talk about our solution in detail

Below I’m describing key points in our tool’s implementation and overall development strategy, that allowed us to develop it fast and with low-risk in mind.

Stack and environment setup

A proper environment setup requires a hybrid approach. All of the code related to missing features and data importers should be merged to the main repository branch as often as possible. You should use as much production code as possible for importing new entities because it gives you perks related to data persistence like triggers, URL generators, etc for free. Also if models definition changes in the production branch, you can quickly detect it, react, and apply required refactors to the importers.

On the other hand, introducing some configuration changes, like temporary database configuration, services needed solely for the migration process, might easily break production or be hard to rollback. Thus, for these changes, the best strategy is to apply these changes on a separate branch based on the main branch of the repository. Working testing and migration environments should be based on that branch. After the migration is done, there is no simpler way to switch back to the production configuration than just deleting this branch.

To build our migration software, we used technologies that we already know — PHP7, Symfony, RabbitMQ, MongoDB. Can you find technologies that would perform better for such a kind of a task? Surely you can. But picking fresh technologies that you barely know might get you into more trouble than you can imagine in the long term.

Another great advantage of this choice is that you can write the whole thing inside your target app, which is really convenient. You don’t have to create another communication layer for e.g. external service. You can use all of the stuff that you already have. Not only mentioned triggers and URL generators but also, for example, data validators — rewriting them would consume a lot of time and would bring close to none added value to the whole project.

Migration flow and breaking the rules

There are two sides of the migration — source (Doctoralia marketplace) and target (Docplanner marketplace).

To migrate data, we connected directly to the source database. Keep in mind that during the migration process Doctoralia marketplace was still running in production mode. To make a compromise between maintaining the app alive and data integrity, we decided to switch its database to read-only mode, and aimed for late-night hours, which lowered the possibility of new data coming in (new bookings, users).

From the performance side, the best approach is to pull all of the data of a given type at once. Executing select-all query for 1,5 million users might take some time, but on the other hand, you cut down a lot of latency and network time for future data processing, as communication with the source database is no longer needed.

The next step is data validation. If validation for some record didn’t pass, we remove it from the stack, put it aside for future analysis. Then, all of the validated records are converted to RabbitMQ tasks with rule — one record, one task. If for some reason, one task fails, it will not stop other imports. Of course, it is a good practice for rabbit tasks to carry on as small payload as possible (preferably ID), but putting whole data payload for a single record inside the task saves you some time while importing.

Another rule to break here is Don’t Repeat Yourself. For regular software development, it’s a must, but when you want to cut down business and architectural dependencies between different entities, code duplication is indeed what you sometimes need. For example, our diseases, services, and drugs dictionaries share a very similar structure, so it is tempting to write one importer to handle them all. In a perfect world, I would agree, but in reality, when you should expect business change on every corner, I strongly advise you to not follow that path.

The above steps can be developed as a pretty straightforward structure: for each type of data, you need a single generator (for example Symfony Command), which will handle querying for data, its validation tasks publishment. On the opposite side, you need a single handler for each data type, that will do the actual import from the RabbitMQ task. This way structure is kept single and it’s dead easy to scale.

These huge return buttons are pretty comfy

Grouping imports into stages

A great strategy to optimize and automate imports a bit is to group them into import stages, grouped and ordered in a way that reflects data relations. If you have to import 100 different types of entities, out of which 20 are independent dictionaries, there’s nothing against importing them simultaneously.

In our case, we used the Symfony feature that makes it possible to programmatically run predefined commands. We created configs that described stages, their relationships, and the import data types they were responsible for. It lowered the number of steps needed to migrate the full dataset from over 100 to about 20. Less clicking, less space for human errors.

Ensuring data integrity

One of the problems with migrations chunked into stages is maintaining data integrity and its relationships in a new application. Assuming that the ID of every entity might change, we have to come up with a mechanism that will make it happen.

Let’s take a look at the following example — a doctor and his offices. These are two different entities in a one-to-many relationship. To migrate this dataset in a proper way, we first need to determine the correct order. Of course, we will pick the Doctor entity first, as in this data set it remains independent. The doctor will be imported and will be given a new ID that is compatible with the target application.

The next step would be to migrate the doctors’ offices data. But how to maintain the relation, when previously imported doctor entities have new IDs? There are two ways out of this problem. The first one is creating a new column in the doctor entity definition, that will hold ID from the source system. I do not recommend this one as this approach will force you to modify the schema of every entity you import, which will result in more cleaning up afterward, plus it is not transparent to other developers in the organization.

A better solution is to go into a bit more “hacky” way. Create a single table, that will have a fieldset of the source entity identifier, target entity identifier, and entity type. If you migrate data from multiple systems into one, you can add another column determining the source.

I called this approach “hacky” because this way you lose what most RDBMSs offer you — relation checking. In other words, you cannot have external keys. On the other hand, what you gain is having a single place where all of the mappings belong, fully transparent of the production code of your target system. Therefore, while importing doctors’ offices, just exchange the old doctor id for the new one using this table, and you are good to go.

Maximizing amount of properly imported data

Chances that you will manage to migrate all of the data between systems that have a lot of differences in business logic are, sadly, low. It’s definitely achievable, but pragmatically it’s better to find a compromise between the amount of imported data and time spent to normalize it.

Let’s take a look at another example — the phone number field. In Docpanner’s marketplace, we were always very strict when it comes to phone numbers handling. We always checked if they are real, then formatted them to uniform format before persisting. Doctoralia on the other hand was a bit more tolerant in this matter — the system allowed any input that looked like a phone number. It could be a single phone, a couple of them separated by spacebar or comma, combined with internal PBX numbers and so on.

To some extent, you can define patterns that will define how most people tend to write their phone numbers, but if you have a dataset of a couple of millions of records, it’s unlikely that you will cover every case. Sometimes it’s even impossible not for technical reasons, but for business ones — you received three phone numbers in your input, while the target system can handle only two. In these situations, you have to aim for maximization of the number of situations you are able to handle but never target for 100% coverage, unless it’s crucial for the business.

Just a reminder here to use what you already have in your target system — if you already have pretty decent data validators, use them instead of building new ones.

Migration monitoring in practice

When your timeframe to import millions of records is limited to a couple of night hours and you want to have the best control possible over the situation, traditional data logging is not the thing that you should look for.

This is why we created our own monitoring system, which is both pretty easy and efficient.

When a new task is created by the generator, we create a new record corresponding to it — it contains ID and type of imported item. When the task is being processed later by the handler, it updates this record according to the import result — success or failure. If the import failed, we also persist information about why it failed, so we have a quick overview of the situation.

Having in mind heavy database loads during the import procedure, we use separate MongoDB instance to store these logs.

To calculate the overall estimations and performance, we also log separately some metadata about the whole import batch itself — like type, date started.

To view logs in realtime, we created a command-line tool, that displays multiple import stats and estimates in the realtime. If something is wrong, the red flag starts to blink and we can react instantly.

Our homemade migration process monitor

Important steps after successful data migration

After the migration is done, there’s a bunch of actions required to perform before switching production to the new app.

The first move is to run a set of crucial tests to determine if all of the main functionalities work fine. If everything is fine, we rollback the temporary changes that I mentioned in the earlier stages. At the same time, we start the migration of the freshly populated database to the production servers.

The last step is to update the DNS records, warm up the cache, and populate ElasticSearch.

Congratulations, your migration process is now complete :)

One of the mornings after another country successfully migrated

Tips & tricks

Test period before the migration

The extensive testing phase before the migration is your (and your organization’s) best friend. That’s why we always aimed for 2–3 month testing phases instead of 2–3 weeks ones. Plan this process in advance the way to have some time to not only test the importers themselves but also:

cooperating with the product folks, prepare a comprehensive set of test scenarios, and conduct them across the whole service, from different aspects — user, doctor, employee,

make sure that all of the people that will be working on a new app, will get proper training on test data sets before production migration,

show a new version of your service to a narrowed group of your clients to gather more feedback and detect more potential bugs.

Talking a little bit more about employee training. What turned out to be really cool is what happened during the migration of the Brasil market — we used gamification methods to conduct it. In our office in Curitiba, there was always a couple of computers with test environment running accessible for everyone, so each employee could check it out, ask some questions to the product expert we sent to the site. For most active employees, there were small gifts to encourage them a little bit more.

While planning test migrations, plan to do at least one of them with full power available. This way you can estimate if you gonna fit into the planned migration timeframe, as well as to test if scaling works properly. In our case, this practice quickly revealed performance problems with the RabbitMQ instance, which sometimes refused to handle too heavy batches of incoming tasks.

What’s the best moment to start the whole process?

It’s best to conduct migration during the night of your end-users. So for us, it was sometimes during our night hours, while sometimes, when we were migrating Latin America countries, it happened during the day.

If your migration estimation is not as optimistic as it could be, it’s best to find a reasonable compromise between limiting functionalities to as little group of users as possible, at the same time maintaining a safety margin, so if something goes wrong, you still have time to react and fix it.

Cost optimization

Watch out for potential money traps. There are, in general, two places to optimize — things that cost a lot to compute or gather that don’t change much during every migration. Also, consider the costs of unused resources in the test environment.

For instance — we have around 600.000 doctors in Brazil. Most of them have profile pictures attached. There is no point in converting these pictures on every import — it will both burn time and money related to computing power that you have to pay for. Do the conversion once instead, correlate new assets with corresponding doctor ids from the source system, then store for further iterations. On the next iterations, detect and recompute only things that have changed.

Another example is optimizing the cost of the test environment. While final migration needs as much power as possible, during the test period the app is being used by so small group of people, that it makes no sense to have the best performing servers running it.

Maintaining a good SEO

It’s almost sure that both of the systems share different URL schemas for the same resource, for example, doctor profile. Don’t forget to let Google know that URL schemas changed, so your SEO will remain good and healthy. How to do that? Just create a bunch of HTTP 301 redirect routings from the old scheme to the new one, using what you already have (e.g. mappings of id’s from the source system). It’s pretty straightforward in Symfony and should be also easy to achieve in any other framework.

Be prepared for errors

Chances that migration will go totally smooth are close to none. Be prepared, that you will have to face unexpected problems both during and after the migration process. Make sure that you have the right set of people, who can handle stressful situations easily. Build your team out of people who know almost everything about both of the apps you’re dealing with. Apart from the knowledge itself, they should have proper access rights, a lot of experience, and a steady hand, when it comes to doing something that might lead to a terrible crash.

And if, by any chance, you use some asynchronous communication or processing in your service, keep in mind that errors related to this topic are the hardest to detect and most painful to fix.

Mathias Verraes at Twitter.com

Team setup

To add a little bit more about choosing the right people for the job. What kind of people you should look for while building your team? Not only the ones who have 10+ years of experience in coding or product management but mostly the ones that are patient, long runners, and are able to handle stressful situations easily.

20 hours run in the office without sleep? Hold my tea

Our fusion was definitely a long-term undertaking, filled both with interesting challenges, but also the ability to interact with enormous computing power, as for some markets we used to have hundreds of worker machines running. But’s that’s just a grain of sand compared to the number of hard decisions we had to make, repetitive and boring tasks, work under pressure, stress, and responsibility.

What really matters are the people, not the code

The last two years taught me a lot of things. For example, for a lot of problems you have to approach with a proper distance, make some cold-blood decisions when needed, and do not let emotions control you.

Jumping into the car and heading to the office in the middle of the night I wasn’t thinking that I’m stressed or sleepy. What I always felt is that I’m going to do something great with all these amazing people I had a chance to work with. All of the procedures, fuckups, and issues, despite huge amounts of stress, I tended to hide behind a thick layer of abstraction.

Our team consisted of around a dozen folks, but what we were doing impacted the whole ~1,5k organization. We used to cooperate with a lot of other people from other teams, departments, and countries — product owners, country managers, experts, customer care specialists, moderators — basically everyone.

The abstraction that I mentioned, gave me a little shock kick at the end of our journey. After migrating the last market, our Product Owner arranged a gathering for everyone involved to spend some time together, have some common activity outside of the office. When I stepped outside of the building, I’ve seen more than 50 people. Some of them I met the first time in my life, even though I was working at Docplanner for many years.

Our team having some good time after the last migration is done. It was totally worth it!

What was common between all these folks was happiness, openness, joy. It wasn’t about a feeling of relief that “it’s finally over”. On a daily basis, all of these people are experts in what they do and they love cooperating together, generating a great atmosphere every day.

What makes me happy even more is that thanks to this project, we are able to grow faster and serve a better product to our patients and doctors. I personally believe that what we do changes our lives in a really good way.

That’s all folks

That day had to come eventually. Around a year after the last migration, we removed all of the migration code from our codebase.

If you consider conducting a similar process in your company, feel free to contact me — we will try to share more of our experiences and help.