How do professionals handle thousands, hundreds-of-thousands, or potentially millions of JSON objects?
Tags: Performance, Nodejs, Json, Elasticsearch, Data

I saw this question on StackOverflow and as I was working towards and answer the question was deleted. I am not sure why it was deleted but it was an interesting question which I would like to have answered and have debated over with other StackOverflow users

I recently completed a small app that was requesting a dozen or so JSON objects (they were movie objects generated by me). Because I was working with few objects there was no need for me to code in an efficient manner when I wanted to parse and search the JSON for specific objects.

But if I were working on a real professional app and it received 100k JSON objects how would I even handle:

Storing all the objects so that the back-end can use it

Searching for specific objects in an efficient manner

For example, let's say those 100k objects were movies where each had a type and list of actors. Would I really parse these 100K into an array? What if my front-end wants the list of movies that include a particular actor. Would I really have to go through the entire JSON every time the front end sends back a different actor? What if the filter changed to all movies of a specific type?What if instead of 100k we have a million movie JSON objects in the back-end? It would seem like having a million entry array or going through the entire million objects often would be really inefficient and could slow the front-end.

For my small app, I just saved all 22 JSON objects to a global array called “allMovies” and I could do a simple linear search to find what I needed, but again if I have a million movie objects I do not see how my app (in its current state) would scale.

I’m still very new to this but that’s essentially my question, how to efficiently store a large number of JSON objects (that the back-end has received) and search them efficiently. I’m looking for guidance on disciplines or data structures that I could implement.

The little app I made was in node.js.

Answer

Hi, Interesting question. There is no correct or single answer to this. Each of us will provide an answer which is based on how well a particular solution has worked.

Let me try and provide a solution and a way to deice.

- Your data from your description is a set of movie objects which will be searched on frequently. A large number of reads.

— And this will be entered once and updated a few times.

— Involves a lot of text search and sorting on the frontend

— You are using nodes and a client-server type architecture for your app, so this will involve REST API

So above are some characteristics which I could point out, which we will use for our solution.

Since we can see that the data you have should be highly searchable and in realtime, it consists mostly of JSON object data, which is dynamic and subject to change. we can go with elasticsearch or MongoDB or any other text search supporting dynamic NoSQL database.

Now that we have database figured out. The next thing is how we do design the flow.

Now that we have a database decided we can proceed to design the data flow.

An important step here is database design and how you can effectively create references and the only person that can do this is you as you have a better understanding of the domain. reference 1 reference 2 reference 3

Step 1 — The movie object first has to parsed and inserted in the database and/or elastic search indexes. This I guess something you have done already on a smaller scale by storing the objects in all movies an array which can act as a client-side buffer. When the buffer is full you can offload the allMovies array to the backend by making a REST API a call using XHR or AJAX from your app.

// Incoming movies

var newMovie = {your data from forms, other source, etc}

allMovies.push(newMovie);

if(allMovies.length >= 20 )

{

//make API call to backend.

//empty the buffer

allMovies = [];

}

//else wait for new movie

Step 2: On the backend just store the data to your database, index the most searched fields. Here is the reference for the 2 databases I mentioned.

note that elasticsearch indexing is also inserting data where MongoDB has 2 step insert and index operations.

Step 3: This can be the part where you show and let users search through your movie database. This is where you will have to create a new API which will provide the ability to perform a custom search to your users and thus the App frontend as well. There can be multiple API or a single API endpoint that accepts multiple parameters like search and sort.

This is how a sample API looks like GET /twitter/_search?q=tag:wow Here are some reference resources for API design, your result may vary as per your requirement.

Here are some reference resources for API design, your result may vary as per your requirement. Get Users API

Search API for searching music tracks

The final step is to integrate all of this to deliver results to your app in a seamless fashion. I will try the illustrate this flow via a simple diagram.