How To Build a Command-Line JSON Splitter
By: Jonathan Hsu
Tags: Technology, Coding, Python, Data Science, Programming

Getting Started

First things first, create a file named json-splitter.py . We'll need three external modules for this project...

os : responsible for opening and reading the file

: responsible for opening and reading the file json : responsible for the decoding and encoding of json data

: responsible for the decoding and encoding of json data math: necessary for calculating the number of files when splitting the JSON file

Our first step will be importing these three modules and printing a welcome message.

import os

import json

import mathprint('Welcome to the JSON Splitter')

print('First, enter the name of the file you want to split')

Requesting the JSON File

In order to split a JSON file we need to ask the user for one. We’re going to use a try/except block to prompt the user for the file, open the file, and check for a JSON Array. If any of these three things fail then the script cannot work and we’ll exit.

try:

# request file name

file_name = input('filename: ')

f = open(file_name)

file_size = os.path.getsize(file_name)

data = json.load(f)if isinstance(data, list):

data_len = len(data)

print('Valid JSON file found')

else:

print("JSON is not an Array of Objects")

exit()except:

print('Error loading JSON file ... exiting')

exit()

Some highlights from the code above:

The input() function is used to prompt the user for text input

function is used to prompt the user for text input We use os.path.getsize() to get the file size which is needed later when splitting.

to get the file size which is needed later when splitting. The isinstance() function is used to make sure the JSON file is an Array, presumably of Objects

function is used to make sure the JSON file is an Array, presumably of Objects If any of the code in the try section causes an error, the except block will execute

Defining the Chunk Size

Now that the JSON file has been loaded into the variable data, it’s time to find out how to split the file. We’re going to split our file based on a maximum file size for each chunk. If the chunk size is smaller than the file then we’ll prompt the user and gracefully exit the script.

# get numeric input

try:

mb_per_file = abs(float(input('Enter maximum file size (MB): ')))

except:

print('Error entering maximum file size ... exiting')

exit()# check that file is larger than max size

if file_size < mb_per_file * 1000000:

print('File smaller than split size, exiting')

exit()# determine number of files necessary

num_files = math.ceil(file_size/(mb_per_file*1000000))

print('File will be split into',num_files,'equal parts')

The most important aspect of the above code is that we convert the String input to a Float (and take the absolute value for good measure). To finish up, we’ll calculate, store and print the number of files that will be created based on the maximum file size.

Converting JSON to a Tabled Structure

Okay, next is to set up the data structure for holding the split pieces and the cutoffs for each piece. We’ll use a 2D Array — also known as an Array of Arrays. Think of a 2D Array as a spreadsheet with an Y-axis (number of nested arrays) and an X-axis (size of each nested array). We’ll create the correct number of nested arrays by using a for loop with the calculated number of chunks. To find the cutoff points lets divide the length of the JSON Array by the number of files. Last, add the length to the end of the array of indices.

# initialize 2D array

split_data = [[] for i in range(0,num_files)]# determine indices of cutoffs in array

starts = [math.floor(i * data_len/num_files) for i in range(0,num_files)]

starts.append(data_len)

Looping Through Each Chunk

We’re all set to slice up our array. For every chunk to create, we’ll loop through the JSON array starting at the current cutoff index and stopping at the next cutoff index. As we complete each chunk we’ll make a file and write the chunk accordingly. Print a section complete message then finally once the loop is done a message letting the user know the entire script has completed.