Migrating Data from Legacy Systems Using Akka Streams and Alpakka
By: Samuel So
Tags: Software Development, Akka Streams, Scala, Database, Alpakka

Migrating data from old applications can be a pain. This is especially true for legacy applications with a lot of data. Although data extraction is straightforward using database backup programs or an application’s built-in data export feature, shaping the extracted data into the format you need can be difficult, especially for large data sets. The extracted data may be incomplete, or may not be packaged in a structure required by your migration workflow. To address these challenges we created a utility that makes the data migration process as efficient and painless as possible. This blog describes how we did this using Akka Streams and Alpakka.

Use Case

Our legacy product (which we’ll call Source App ) is an on-premise Java EE application that contains a large amount of data in a SQL database. Our goal is to extract that information and transfer it over to our cloud-based solution (which we’ll call Target App ). The product’s main feature is to analyze business data and detect fraud and anomalies on a scheduled basis. Analysis data is modelled as Analytic Tables which belong to one or more Analytic Projects. Also belonging to Analytic Projects are Analytic Scripts which contain sets of SQL-like operations to be executed on Analytic Tables.

An Analytic Project contains Analytic Tables and Analytic Scripts

These objects are stored in the Source App database in a single denormalized table — audititems , with every item having a reference to its parent item, which itself is in this same table. Each table record has metadata columns describing the item’s ID, name, type, parent ID, etc. The relationship of these objects and their metadata are the things we’re interested in transferring into its successor product ( Target App ).

Source App’s audititems table

To import this data, we will use Target App ’s import feature which can ingest data from compressed files with data laid out in a predefined structure. This structure organizes data per Analytic Project with its name and contents derived from Source App 's database.

Example folder structure for Target App’s import feature

Source App has an archive feature available that can be used to extract data, but its output format does not match what’s needed for Target App ’s import feature. In order to migrate data from the old application, customers usually engage our consulting team to conduct a migration process which involves manually archiving every Analytic Project and placing the archive outputs in folders representing the relationship of the extracted items. This is a time-consuming process and can take up a large portion of time in the consulting engagement that could otherwise be spent on adding more value to the customer.

What we wanted was a standalone data extraction and transformation utility that can be used to get data out from Source App to be loaded into Target App without a lot of manual work. This tool should make the extraction and packaging of old application data much easier and reduce the overall time needed to complete the migration.

Options Considered

First we considered modifying the existing archive feature from Source App to export data in the format we need for the migration process. However, development and testing can be challenging for a legacy application — software frameworks and libraries become outdated and existing functionality can be hard to extend. The process to deploy this new feature is also time consuming as the upgrade involves running installer software directly on the server, often requiring additional planning and access rights.

Another option we considered was to extract the data directly using database backup programs or dump utilities and format the data with custom shell/SQL scripts. This would be an efficient way to extract the needed metadata, but a fair amount of work would be needed to derive the relationship of the extracted objects (some of which are stored as XML data that must first be parsed) and place them in the correct paths in the required structure. Also, much of this work is specific to the customer and may not be easily reusable.

A Streaming Solution

To cut out all the custom work, we decided on building a simple standalone streaming solution utilizing Alpakka to connect to the legacy application database with the Slick JDBC connector — Slick is a database access library for Scala. We’ll use basic linear stages from the Akka Streams library to process elements and construct the destination output files that we need.

Each step in the migration process is expressed by a processing stage. These are put together to compose a processing pipeline capable of extracting and transforming large data sets into files of the expected format, which are then compressed. By using the database connectors from Alpakka and the fluent DSL provided by Akka Streams, we get many of the benefits of streaming including ease of use, reusability, and composability. Best of all, we do not need to worry about running out of memory processing large data sets as we would with a non-streaming solution. By using Akka’s high-level, explicit API we enjoy these benefits without having to manage the complexity of streaming ourselves. Aside from the aforementioned benefits of going with this approach, this was a good opportunity to work with a fun and new way of programming a solution that doesn’t involve working with large objects in memory.

Here are the steps — to be described in detail with code later:

Establish a connection to Source App ’s database using Alpakka’s Slick connector for PostgresSQL. Create a Source with Slick.source from the results of a database query to extract the records we’re interested in as AuditItem elements from Source App . Use Akka Stream’s linear stage processing to transform elements into our desired data type Project ( Source App ’s Analytic Project). Create a Flow that creates directories from the paths constructed from Project elements and writes the project content out into the directories using Akka Streams’ FileIO. Compress the entire output folder.

The Code

The steps to our solution can be expressed simply with a single statement of code that ties everything together. We’ll go over the implementation by covering each of these steps in detail next.

Establish a Connection

Before we can run the code shown above, we need to first configure a database session. In our standalone tool written in Scala, we create a database session with an instance of SlickSession defined in the application.conf with profile and db properties for declaring database connection settings. The following is for PostgreSQL, but other database connection types can be added by substituting the profile and driver :

In our case, database connection settings are available from Source App in the form of an encrypted XML file. Username and password entries are decrypted and used to create a database session. Note that the constructed config overrides settings loaded from application.conf . This gives us flexibility in how we can create this config object.

Create a Source

Once a database session is available, we can create a Slick Source that emits elements from the results of a SQL query. We now define the auditItems value that you saw in the first code block. The SQL query retrieves data from Source App ’s audititems and audititemsdata tables using Slick’s plain SQL / String Interpolation. This returns a result set of the AuditItem class representing the data we’re extracting from the database. The data for our pipeline comes from this Source.

Transform Elements

Using the Source map operator, we transform elements from the auditItems Source defined above into the expected data type — from AuditItem elements to Project elements. The following code illustrates how .map(auditItem => Project(auditItem)) performs the conversion using the apply factory method.

Create a Flow

Then, using an Akka Streams Flow Stage, we process all of the Project elements received from the auditItems Source after the transformation step above. In this stage, we’ll first create the file along with its full path based on the attributes of the Project element and then write to this file with data contained in the Project element’s content field representing the Analytic Project. The writing of data is done using Akka Stream’s FileIO with IOResult as the return type. Each element is processed asynchronously and in parallel using the mapAsync operator with parallelism set to 4.

Compress the Results

Finally, we construct the processing pipeline and materialize the stream. For simplicity, the IOResult elements emitted downstream after the steps above are discarded in Sink.ignore, though you can make use of them using other Sink operators. The extract and transform phases are expressed in a clean and concise manner. Upon completion, the expected files and folders have been created and the remaining task is to compress the whole path in the expected layout for importing into Target App .

Summary

We demonstrated how the integration connectors from Alpakka and the operators provided by Akka Streams can make implementation of a streaming data extraction and transformation tool quite simple and straightforward. With much of the complexity of streaming managed by Akka, we can easily build applications capable of handling large amounts of data that are efficient and easily extensible. Adding support for customers with Oracle databases, for example, only requires the use of a different Alpakka integration connector with slight changes to the config. This tool has a small memory footprint for the amount of data that it can process and its implementation was made simple by the libraries that Akka provides. There’s a lot of potential to extend the capabilities of this tool as well, as Akka Streams offers a vast array of operators that we can add to our processing pipeline to manipulate elements as needed.