Package: django-retriever
Django RetrieverThe Simple Way to Interface JSON and DjangoOverviewThe retriever is an interface used to define how to map
a JSON object to a Django model. The retriever takes care
of mutating the JSON structure, connecting foreign objects,
and creating or updating the resulting Django model.There are several components that must be defined in the
retriever to complete the job,modelthe Django model with which the retriever
is intended to interfaceidthe list of field names in the Django model
that should be used to determine whether an object is
already in the database. These can be thought of unique
fields. If they are not actually unique, the retriever will
react by not saving the JSON object.structuresthe critical component of the retriever. Thestructuresattribute is the interface between the JSON
document and the Django model. The interface is a nested structure
of lists and dictionaries, and must nest to a series of tuple/list
objects in the following format,namethe name of the key in the JSON documentforeign_structures(defined below) the list of foreign structures to
define how a key in the JSON document must be used to relate
Django modelsstructures(defined below) the list of mappings to determine how the
JSON document value must be mutated and saved to the Django
modelDefinitionsForeignStructuremodelthe foreign modelidthe primary key field name in the foreign modelid2the field name in the current model that is mapped
to the primary key in the foreign model (id)Structurethe normal structure definition for the JSON
document, because the value might very well map to another
field on the foreign modelStructurenamethe new name of the JSON document, in the case
the JSON and the Django model name the field differentlyfuncthe mutation to perform on the JSON document. Can
be one ofint,str,bool, or a custom functionInstallationpip install django-retrieverContributingBefore contributing, please install thetestanddevversions of
the library,pip install django-retriever[dev,test]Please run tests using (in the root project directory)python -m pytest testsOnce the package is ready to be uploaded, please complete the
following in order,Increment theversiontemplate inpyproject.tomlSet thePYPI_TOKENenvironment variableRun./bin/twinefrom the root project directoryUsageThe best explanation is going to be through an implementation, so please
consider the following case. The JSON object includes information
about a product image, and is taken from a public API.Consider the JSON document,{
  "results": [
    {
      "raw": {
        "ec_sku": "333333",
        "image": "https://image3.jpeg"
      }
    }
    {
      "raw": {
        "ec_sku": "333333",
        "image": "https://image3.jpeg"
      }
    }
    {
      "raw": {
        "ec_sku": "333333",
        "image": "https://image3.jpeg"
      }
    }
  ]
}The JSON document should be loaded into a python object,
using a parser of your choice. The following code inmodels.pyandretriever.pyis a sample that could be
used to parse the JSON document into some database objects
through the Django ORM.# models.py

from django.db import models


class Product(models.Model):

    id = models.AutoField(
        primary_key=True)
    sku = models.CharField(
        max_length=64)


class Image(models.Model):

    product = models.ForeignKey(
        Product,
        on_delete=models.CASCADE,
        related_name="images")
    image = models.ImageField(
        # storage, etc.
        ...)
    source_url = models.CharField(
      max_length=512)# retrievers.py

from retriever import Retriever
from .models import (
    Image,
    Product
)


class ProductRetriever(Retriever):

    model = Product
    id = ["sku"]
    structures = [
        {
            "results": [
                {
                    "raw": [
                        [
                            "ec_sku",
                            [],
                            [
                                "sku",
                                None
                            ]
                        ]
                    ]
                }
            ]
        }
    ]


class ImageRetriever(Retriever):

    model = Image
    id = ["product_id", "source_url"]
    structures = [
        {
            "results": [
                {
                    "raw": [
                        [
                            # JSON name
                            "ec_sku",
                            # foreign structures
                            [
                                # foreign model
                                Product,
                                # id and foreign id field name
                                ["id"],
                                ["product_id"],
                                # normal structure, the `ec_sku` field is
                                # clearly a map to the unique field `sku`, not `id` # itself. Note that `sku` should be unique or the
                                # structure will not be created and an error should
                                # be propagated
                                [
                                    "sku",
                                    None
                                ],
                            ],
                            # structures. Note `sku` is not in the `Image` model,
                            # we are not interested in mapping it
                            [],
                        ],
                        [
                            # JSON name
                            "image",
                            # no foreign structures
                            [],
                            # map it to `source_url` field name, no
                            # mutation required
                            [
                                "source_url",
                                None,
                            ],
                        ],
                    ]
                },
            ]
        }
    ]If you've loaded in the JSON document and defined the retrievers
correctly, you can now simply save the JSON document to
the database,from .retrievers import *

json_object = {
    "results": [
        ...
    ]
}

ProductRetriever(
    batch_size=5,
    default=[],
    strict=True
).save(json_object)

ImageRetriever(
    batch_size=5,
    default=[],
    strict=True
).save(json_object)The retrievers have used theec_skufield in the JSON
document to find theProductobject that corresponds to
the correctImageobject. In this way, a JSON document can
be decomposed into several retrievers, and the definitions
can be isolated. Note, theProductRetrievershould be called
before theImageRetriever, or there will be noProductobjects
in the database to find.