Repository: pulp/pulpcore

Issue:
Json parsing library used for imports "naya" is quite slow
**Describe the bug**

In a number of different profiles of import tasks on large and small systems, the "naya" json parsing library is responsible for about 2/3rds of the execution time during the repository version import tasks.

The reason we use Naya is that it has an API close to what we needed at the time, and its pure-Python nature made it easy to package for immediate distribution to resolve https://github.com/pulp/pulpcore/issues/2307

It was the right choice at the time.  Going forwards we could probably find something better.  Alternatively, we could attempt to optimize Naya to better suit our needs.  Currently Naya processes one character at a time and has a huge tree of if statements:

https://github.com/danielyule/naya/blob/master/naya/json.py#L54

There's a limit to what could be achieved here (it's Python after all, and faster parsers including the stdlib ones are written in native code) but it's probably possible to reduce the overhead by restructuring that code to require less conditional checks per character.

**To Reproduce**

Perform an import with pyinstrument profiling enabled, observe that Naya is responsible for a huge portion of the runtime.

I also manually reverted the PR for  https://github.com/pulp/pulpcore/issues/2307 and found that the execution time dropped from 1533 seconds to 567 seconds.  We cannot simply do that as bulk processing like this has the potential to use too much memory during import - on my system this inflated from 350mb memory used to 4.2gb memory used for one task.  More than one such task executing, or the use of larger repos such as container repos, could blow out the 20gb minimum requirement we'd like to keep well within if possible.

But it demonstrates that a library with lower overhead while maintaining the same requirements would yield significant benefits.



Discussion:

Resolving patch:
diff --git a/pulpcore/app/tasks/importer.py b/pulpcore/app/tasks/importer.py
--- a/pulpcore/app/tasks/importer.py
+++ b/pulpcore/app/tasks/importer.py
@@ -7,10 +7,10 @@
 from gettext import gettext as _
 from logging import getLogger
 
+import json_stream
 from django.conf import settings
 from django.core.files.storage import default_storage
 from django.db.models import F
-from naya.json import stream_array, tokenize
 from io import StringIO
 from pkg_resources import DistributionNotFound, get_distribution
 from rest_framework.serializers import ValidationError
@@ -76,17 +76,14 @@ def _impfile_iterator(fd):
     we yield the result of json.dumps() for that batch. Repeat until all rows have been
     called for.
     """
-    eof = False
+    data = json_stream.load(fd)
     batch = []
-    rows = stream_array(tokenize(fd))
-    while not eof:
-        try:
-            while len(batch) < IMPORT_BATCH_SIZE:
-                batch.append(next(rows))
-        except StopIteration:
-            eof = True
-        yield json.dumps(batch)
-        batch.clear()
+    for row in data:
+        batch.append(json_stream.to_standard_types(row))
+        if len(batch) >= IMPORT_BATCH_SIZE:
+            yield json.dumps(batch)
+            batch.clear()
+    yield json.dumps(batch)
 
 
 def _import_file(fpath, resource_class, retry=False):


Test patch: