Package: django-import-data
Django import datais a command-line tool for importing XML, HTML or
JSON data to django models via XSLT mapping.Source code is located here -https://github.com/lev-veshnyakov/django-import-data.Basic featuresDjango import data can take any XML, HTML or JSON source file or URL as
an input and save entities from it to the django models without need to
modify an existing code.It also supports saving of a related data in form one-to-many and
many-to-many.DependenciesIt useslxmllibrary for all XML manipulations anddicttoxmllibrary for
the transformation from JSON to XML.InstallationFirst you need to install dependencies for lxml library:sudoapt-getinstalllibxml2-devlibxslt-devpython-devThen install django-import-data using pip:pipinstalldjango-import-dataIf you want the latest version you can install it from Github:pipinstallgit+https://github.com/lev-veshnyakov/django-import-dataAdd import_data to INSTALLED_APPS:INSTALLED_APPS=[...'import_data',]UsageDjango import datais a management command-line tool, that can be
used from the code as well.Too see the list of console commands type:pythonmanage.pyhelpIn the output you will find import_data section like below:[import_data]process_xsltvalidate_xmlTo get help for the particular command type:pythonmanage.pyhelpprocess_xsltpythonmanage.pyhelpvalidate_xmlpythonmanage.pyhelpjson_to_xmlTo call console commands from your code usedjango.core.management.call_command:fromdjango.core.managementimportcall_commandcall_command('process_xslt','http://stackoverflow.com/','transform.xslt','--save')How it worksIn a few words it takes a source in either XML or HTML, then takes
provided by you XSLT file, transforms the source into the specific XML
representation, and then saves the data from this XML to the database
using models.The point is, that you don’t need to write procedural code for saving
data. You only need to write XSLT files, which is actually XML. One file
for one source. By the source I mean a range of XML or HTML files in the
same format. For example all google search result pages have one schema.
That means that you can write only one XSLT transformation file to
import all search pages data.The difficult moment is that you have to be familiar with XSLT and
Xpath.XSLT and XPathXSLT is a language for transforming XML documents into XHTML documents
or to other XML documents.XSLT uses XPath to find information in an XML document. XPath is used to
navigate through elements and attributes in XML documents.If you are not familiar with that I reccomend you to read ashort
tutorial on
www.w3school.com.Moreover, you have to know what an XML Schema is and a particular schema
language RELAX NG.XML Schema and RELAX NGDjango import datauses RELAX NG to validate resuls of
transformations. That means if you write XSLT file wrong, it wouldn’t be
accepted.But you dont have to write RELAX NG schema yoursef, it’s alreadyincluded in the
module.Resulting XMLAfter XSLT transformation and schema validation the resulting XML file
should be like following:<?xml version="1.0" encoding="UTF-8"?><mapping><modelmodel="app.Author"><itemkey="1"><fieldname="name">AndrewTanenbaum</field></item><itemkey="2"><fieldname="name">DonaldKnuth</field></item></model><modelmodel="app.Book"><itemkey="1"><fieldname="name">ComputerNetworks</field><fieldname="ISBN">0130661023</field><fkmodel="app.Author"key="1"/></item><itemkey="2"><fieldname="name">TheArtofComputerProgramming</field><fieldname="ISBN">0321751043</field><m2mkmodel="app.Author"key="2"/></item></model></mapping>This XML can be automatically saved to the models.It contains the root element<mapping/>. Into it are nested<model/>elements. Each model element represents a particular django
model. You must providemodel=""attributes, in which specify a
related model. Path to the model is in following format:
application_name.ModelName, the same format likemanage.py dumpdatauses.Model elements don’t have to be unique. If you specify several model
elements with the same model attribute, they will be merged together.
This concerns to item elements as well.Model elements contain<item/>elements, representing particular
records in the database. They have only one required attributename="", which sets the name of a related model field.Foreign keysDjango import datasupports import of related entities in the form
one-to-many and many-to-many. To save such entities your models should
have appropriate foreign keys.In a resulting XML you can use<fk/>and<m2m/>elements (see
above). They havemodel=""andkey=""attributes, pointing to
the related<item/>elements.Setting key attributeThekey=""attribute of<item/>elements must be unique by each
unique record. It has not to be the same as a primary key value in the
database. It even will not be stored (if you want to store a primary key
value, use<field/>element).Therefore, the value of thekey=""attribute not obliged to be
integer. You can use any sring. Often it’s convenient to use an URL as
the key.You can even omit filling that attribute if you don’t have related
items.But one case is special. That’s when you don’t have any unique
attributes in the source. In that case you can usegenerate-id(..)XPath function. It will generate unique IDs for every separate XML node
in the source.Using JSON sourcesIt’s possible to use JSON sources. Because the transformation is
XSLT-based, JSON is converted to the appropriate XML.For example the following JSON code:{"firstName":"John","lastName":"Smith","age":25,"address":{"streetAddress":"21 2nd Street","city":"New York","state":"NY","postalCode":"10021"},"phoneNumber":[{"type":"home","number":"212 555-1234"},{"type":"fax","number":"646 555-4567"}],"gender":{"type":"male"}}will be converted to this XML:<?xml version="1.0" encoding=""?><root><firstNametype="str">John</firstName><lastNametype="str">Smith</lastName><agetype="int">25</age><addresstype="dict"><postalCodetype="str">10021</postalCode><citytype="str">NewYork</city><streetAddresstype="str">212ndStreet</streetAddress><statetype="str">NY</state></address><phoneNumbertype="list"><itemtype="dict"><typetype="str">home</type><numbertype="str">212555-1234</number></item><itemtype="dict"><typetype="str">fax</type><numbertype="str">646555-4567</number></item></phoneNumber><gendertype="dict"><typetype="str">male</type></gender></root>That XML is supposed to be used for writing an XSLT transformation.If you use some JSON source and want to find out which XML is related
for it, then use the command:pythonmanage.pyjson_to_xml<URL>After writing an XSLT transformation file you can useprocess_xsltspecifying the URL of the JSON source.JSON to XML transformations is performed by dicttoxml library written by
Ryan McGrealhttps://github.com/quandyfactory/dicttoxml.ExamplesSave data to one modelIn this simple example we will parse the main page ofstackoverflow.comand save titles of
recent questions to this model:fromdjango.dbimportmodelsclassQuestion(models.Model):title=models.CharField(max_length=255)First we need to write an XSLT file:<?xml version="1.0" encoding="UTF-8"?><mappingxsl:version="1.0"xmlns:xsl="http://www.w3.org/1999/XSL/Transform"><modelmodel="test_app.Question"><xsl:for-eachselect="//a[@class='question-hyperlink']"><itemkey=""><fieldname="title"><xsl:value-ofselect="."/></field></item></xsl:for-each></model></mapping>Name it transform.xslt and perform the following command:pythonmanage.pyprocess_xslthttp://stackoverflow.com/questionstransform.xslt--validateThe output will be like this (but longer):<?xml version="1.0" encoding="utf-8"?><mapping><modelmodel="xml_json_import.Article"><itemkey=""><fieldname="title">customizingsoapresponseattributeformat</field></item><itemkey=""><fieldname="title">Secondfragmentloadedbutnotvisibleonscreen</field></item><itemkey=""><fieldname="title">django-oscar:firsttimeuse"pythonmanage.pymigrate"getserror</field></item><itemkey=""><fieldname="title">JTablefireTableDataChanged()methoddoesn'trefreshtable</field></item><itemkey=""><fieldname="title">whythedynamicnodesdontrespondtoclickinjstree?</field></item><itemkey=""><fieldname="title">Connectingkdb+toR</field></item></model></mapping>Parameter--validateadds to outputDocument is valid.To save the result add the parameter--saveto the command above.Save data to related modelsUse the same source and add two other models with foreign keys:classQuestion(models.Model):title=models.CharField(max_length=255)user=models.ForeignKey('User',null=True)tags=models.ManyToManyField('Tag')classTag(models.Model):title=models.CharField(max_length=255)classUser(models.Model):title=models.CharField(max_length=255)The XSLT file will be like following:<?xml version="1.0" encoding="UTF-8"?><mappingxsl:version="1.0"xmlns:xsl="http://www.w3.org/1999/XSL/Transform"><modelmodel="test_app.Question"><xsl:for-eachselect="//div[@class='question-summary narrow']"><itemkey=""><fieldname="title"><xsl:value-ofselect=".//a[@class='question-hyperlink']"/></field><fkmodel="test_app.User"><xsl:attributename="key"><xsl:value-ofselect="generate-id(.//div[@class='started']/a[2])"/></xsl:attribute></fk><xsl:for-eachselect=".//a[@class='post-tag']"><m2mkmodel="test_app.Tag"><xsl:attributename="key"><xsl:value-ofselect="generate-id(.)"/></xsl:attribute></m2mk></xsl:for-each></item></xsl:for-each></model><modelmodel="test_app.Tag"><xsl:for-eachselect="//a[@class='post-tag']"><item><xsl:attributename="key"><xsl:value-ofselect="generate-id(.)"/></xsl:attribute><fieldname="title"><xsl:value-ofselect="."/></field></item></xsl:for-each></model><modelmodel="test_app.User"><xsl:for-eachselect="//div[@class='started']/a[2]"><item><xsl:attributename="key"><xsl:value-ofselect="generate-id(.)"/></xsl:attribute><fieldname="title"><xsl:value-ofselect="."/></field></item></xsl:for-each></model></mapping>Take notice how calculated attributes are set and how to use generate-id
function. In tis example we use both types of relationship: one-to-many
and many-to-many. This means that one question can have several tags,
but only one related user (which has changed the question last).The output will be like following (but essential longer):<?xml version="1.0" encoding="utf-8"?><mapping><modelmodel="test_app.Question"><itemkey=""><fieldname="title">CTCPServerdoesn'tsenddatabeforeclosing</field><fkmodel="test_app.User"key="idp1407668180"/><m2mkmodel="test_app.Tag"key="idp1407657924"/><m2mkmodel="test_app.Tag"key="idp1407659948"/><m2mkmodel="test_app.Tag"key="idp1407660732"/><m2mkmodel="test_app.Tag"key="idp1407661540"/></item><itemkey=""><fieldname="title">NinjectFactoryExtension</field><fkmodel="test_app.User"key="idp1407665492"/><m2mkmodel="test_app.Tag"key="idp1407676788"/><m2mkmodel="test_app.Tag"key="idp1407674900"/><m2mkmodel="test_app.Tag"key="idp1407678572"/><m2mkmodel="test_app.Tag"key="idp1407678508"/><m2mkmodel="test_app.Tag"key="idp1407652988"/></item><modelmodel="test_app.Tag"><itemkey="idp1407657924"><fieldname="title">c</field></item><itemkey="idp1407659948"><fieldname="title">linux</field></item><itemkey="idp1407660732"><fieldname="title">sockets</field></item><itemkey="idp1407661540"><fieldname="title">tcp</field></item><itemkey="idp1407676788"><fieldname="title">c#</field></item><itemkey="idp1407674900"><fieldname="title">dependency-injection</field></item><itemkey="idp1407678572"><fieldname="title">ninject</field></item><itemkey="idp1407678508"><fieldname="title">ninject.web.mvc</field></item><itemkey="idp1407652988"><fieldname="title">ninject-extensions</field></item></model><modelmodel="test_app.User"><itemkey="idp1407668180"><fieldname="title">user3809727</field></item><itemkey="idp1407665492"><fieldname="title">user2119597</field></item></model></mapping>