Repository: weecology/retriever

Issue:
JSON file generated using install json is not pretty
The .json file generated when installing datasets into json is not pretty.

For example, consider abalone-age-

The output file generated when using 'retriever install json abalone-age` looks like:
![current_look](https://user-images.githubusercontent.com/21103831/54385322-5aa3f000-46bc-11e9-8ccd-6679462f4f37.png)

The size of this file is 726.5 kB.

The output file when setting the indentation of json to 2:
![Screenshot from 2019-03-15 00-54-56](https://user-images.githubusercontent.com/21103831/54385582-fa617e00-46bc-11e9-919c-f076c186fe46.png)

The size of this file is 810.1 kB.

It would be great if we have an option of setting the indentation and sorting the keys
 `retriever install json abalone-age --sort-keys --indent 2` to increase the readability of the json file.
I am aware that we keep the json files the way it is right now to reduce the file size but again these new options are optional. When installing the `abalone-age` to `xml` it takes 1.3 MB.
It would be nice if we let the user decide upon the tradeoff between file size and readability.

@ethanwhite @henrykironde Your views on this?
If you like it I will send a PR for this.


Discussion:
Thanks @apoorvaeternity. I think this is an important point. Are most data produced by other folks that are stored in JSON stored in the pretty format?
I personally don't know about good JSON dataset providers.
I just googled `datasets in json` and found some of these sites and most of their datasets were indented:
https://catalog.data.gov/dataset?res_format=JSON
https://json-datasets.zeef.com/jdorfman
https://data.world/datasets/json

Even `Django` allows dumping of data from database in JSON format with the option of setting the indentation.
`python manage.py dumpdata table_name --indent 2 > table_name.json`

Similarly we can have that option here too
`retriever install json abalone-age --indent 2`

Again by default we will store the data like we do it right now but for JSON format this is something that is good to have.
I think if we add something like that we should use a default indent spacing and no need to provide a 2.
Thanks @apoorvaeternity - I definitely agree that this should at least be optional. I asked because if most folks using JSON for data are already using pretty formatting then I think there's an argument for just making everything pretty. I agree with @henrykironde that we should skip the customization of the indent and using something like  `retriever install json --pretty abalone-age`

Resolving patch:
diff --git a/retriever/engines/jsonengine.py b/retriever/engines/jsonengine.py
--- a/retriever/engines/jsonengine.py
+++ b/retriever/engines/jsonengine.py
@@ -12,7 +12,7 @@
 
 
 class engine(Engine):
-    """Engine instance for writing data to a CSV file."""
+    """Engine instance for writing data to a JSON file."""
 
     name = "JSON"
     abbreviation = "json"
@@ -110,10 +110,14 @@ def insert_statement(self, values):
         else:
             newrows = values
         json_dumps = []
+        pretty = True if "pretty" in self.opts and self.opts["pretty"] is True else False
         for line_data in newrows:
             tuples = (zip(keys, line_data))
             write_data = OrderedDict(tuples)
-            json_dumps.append(json.dumps(write_data, ensure_ascii=False) + ",")
+            if not pretty:
+                json_dumps.append(json.dumps(write_data, ensure_ascii=False) + ",")
+            else:
+                json_dumps.append(json.dumps(write_data, ensure_ascii=False, indent=2) + ",")
         return json_dumps
 
     def table_exists(self, dbname, tablename):
diff --git a/retriever/lib/get_opts.py b/retriever/lib/get_opts.py
--- a/retriever/lib/get_opts.py
+++ b/retriever/lib/get_opts.py
@@ -100,6 +100,10 @@
         engine_parser.add_argument('-b', '--bbox', nargs=4,
                                    help='Set bounding box xmin, ymin, xmax, ymax',
                                    required=False)
+        if engine.name == "JSON":
+            engine_parser.add_argument('-p', '--pretty',
+                                       help='Add indentation to json file', action='store_true',
+                                       required=False)
 
     abbreviations = set('h')
 
diff --git a/retriever/lib/install.py b/retriever/lib/install.py
--- a/retriever/lib/install.py
+++ b/retriever/lib/install.py
@@ -125,14 +125,15 @@ def install_msaccess(dataset, file='access.mdb',
 
 def install_json(dataset,
                  table_name='{db}_{table}.json',
-                 data_dir=DATA_DIR, debug=False, use_cache=True):
+                 data_dir=DATA_DIR, debug=False, use_cache=True, pretty=False):
     """Install datasets into json."""
     args = {
         'command': 'install',
         'dataset': dataset,
         'engine': 'json',
         'table_name': table_name,
-        'data_dir': data_dir
+        'data_dir': data_dir,
+        'pretty': pretty
     }
     return _install(args, use_cache, debug)
 


Test patch: