JSON Array Optimization
By: Mykhailo Romaniuk
Tags: Json, Optimization, Arrays

JSON Array Optimization

This technique can be called “Data Transpose” (in some way it’s similar to the transpose of a matrix). And basically, it’s about the changing of data structure before sending the response to the client in order to reduce data size.

The Problem

Let's imagine:

We are sending HTTP request from our front-end application to the back-end API to get a list of products

Back-end returns an array of products in JSON format

This array contains 10 000 items — which results in 750 Kb of gzipped traffic that must be loaded to the user’s browser, let’s say, over 3G

Sound like a nightmare, right?

An example of a communication flow

How to improve this?

The obvious answer — use server-side pagination or even special protocol like OData that has built-in pagination. And it’s the correct way! If it applies to your scenario — go with that approach.

However, sometimes it’s not possible to use pagination (e.g. you need data for offline mode) but you still want to optimize your network traffic consumption. And I have an interesting solution for you.

The optimization trick consists of two steps:

Before return, the response from the server should be modified in a specific way — transpose Then, on the client, transform the response back to get the original result — reverse transpose.

Transpose Data

In order to transpose data in our application, we must change it in the way, so instead of an array of products, we will have an object that contains arrays of values for each property of the products.

Looks scary? Let me illustrate this:

Visual representation of data transpose

So, by getting values from object.name[0] and object.price[0] we can get a name and a price of the first product; by getting object.name[1] and object.category[1] — get info about the second product; and so on.

What it gives us?

First of all, we are reducing the overhead that comes with the JSON structure, because we don’t have to repeat the property’s names for each product in the array.

The second benefit — now it’s more likely that similar data (like product.category ) will be placed together and the compression algorithm will be able to compress it more efficiently.

Implementation

That kind of transformation isn’t complex and even can be done in a generic way. See examples of code for JavaScript and C#:

JavaScript implementation of the generic transpose

.NET implementation of the generic TransposedCollection

The thing I want to recommend when talking about API— it’s better to control with an HTTP header if a transpose should be executed. So, if some specific header present in the request — transpose the response and provide a corresponding response header, otherwise — leave it unchanged (implementation of this approach varies on the technology, so I leave it up to you).

Control transpose with HTTP headers

Reverse Transpose

If you are good at algorithms you already can imagine how the opposite data transformation looks like. Here is an illustration:

Visual representation of reverse data transpose

Implementation

JavaScript and C# examples:

JavaScript implementation of revert transpose

.NET implementation of revert transpose

Sorting is Important

As I previously said — placing similar data together can increase the efficiency of the compression.

So, we can take advantage of sorting our data before the transformation itself. The best option is to sort by string properties that have low entropy.

In our example a great candidate for sorting is product.category since there is a limited count of different categories and they are likely to be repeated in the response.

.NET sorting example

The algorithm is data-dependent — for different datasets, the best sorting options will be different. So, don’t hesitate to sort by few fields with a corresponding sort order to get even better results. But make sure to do measurements along the way to get the best results for your use-cases.

Speed

There are two aspects that need to be pointed:

How data size decreased after using the algorithm?

What does it take to execute the algorithm?

Data Size

Within my app, we had an object with around 40 properties. We tested the transpose approach on a bunch of typical datasets. Data size was reduced by x1.4–x2.1 times (in a case when gzip used). Here is an example with one of the measurements:

A real-life example of optimization measurements

You may see that the compression ratio is smaller for a transposed array. The reason for this — transpose reduces the entropy of the result which leads to a smaller ratio.

As I said — the algorithm efficiency is data-dependant. So, my results aren’t etalon. However, it can bring you the idea of what you can get.

Algorithm Speed

If going mathematical way and describe the algorithm’s time complexity in terms of Big O notation, algorithm running time can be described as O(n) . It can give you an idea of how the algorithm work at scale.

In real life, speed can depend on technology and implementation. For example, dynamic languages like JavaScript can do transformations quicker, when .NET requires reflection for execution.

Comparison of speed between Node and .NET (CPU: i5–8250U 1.60 GHz, 4 Cores; RAM: 16 GB)

When to Use

The described technique isn’t a silver bullet. This is optimization that should be used wisely in certain cases. So, you can use it if:

You have a big amount of structured data to transfer over the network;

You cannot use other optimization techniques like server-side paging for some reason;

You can guarantee that all your clients are aware of your new data structure and can handle it (as an option can be used special HTTP Header to turn-on transpose optimization).

Or:

You are doing it for fun on your pet-project (experimenting in a sandbox is great).

And always TEST AND MEASURE your results before going live with this solution.

Have fun!