Package: scrapy-djangoitem
scrapy-djangoitemis an extension that allows you to defineScrapy itemsusing existingDjango
models.This utility provides a new class, namedDjangoItem, that you can use as a
regular Scrapy item and link it to a Django model with itsdjango_modelattribute. Start using it right away by importing it from this package:from scrapy_djangoitem import DjangoItemInstallationStarting withv1.1bothPython 2.7andPython 3.4/3.5are
supported. ForPython 3you needScrapy v1.1or above.Latest tested Django version isDjango 1.9.Install fromPyPIusing:pip install scrapy-djangoitemIntroductionDjangoItemis a class of item that gets its fields definition from a
Django model, you simply create aDjangoItemand specify what Django
model it relates to.Besides of getting the model fields defined on your item,DjangoItemprovides a method to create and populate a Django model instance with the item
data.UsageDjangoItemworks much like ModelForms in Django, you create a subclass
and define itsdjango_modelattribute to be a valid Django model. With this
you will get an item with a field for each Django model field.In addition, you can define fields that aren’t present in the model and even
override fields that are present in the model defining them in the item.Let’s see some examples:Creating a Django model for the examples:from django.db import models

class Person(models.Model):
    name = models.CharField(max_length=255)
    age = models.IntegerField()Defining a basicDjangoItem:from scrapy_djangoitem import DjangoItem

class PersonItem(DjangoItem):
    django_model = PersonDjangoItemworks just like Scrapy items:>>> p = PersonItem()
>>> p['name'] = 'John'
>>> p['age'] = '22'To obtain the Django model from the item, we call the extra methodDjangoItem.save()of theDjangoItem:>>> person = p.save()
>>> person.name
'John'
>>> person.age
'22'
>>> person.id
1The model is already saved when we callDjangoItem.save(), we
can prevent this by calling it withcommit=False. We can usecommit=FalseinDjangoItem.save()method to obtain an unsaved model:>>> person = p.save(commit=False)
>>> person.name
'John'
>>> person.age
'22'
>>> person.id
NoneAs said before, we can add other fields to the item:import scrapy
from scrapy_djangoitem import DjangoItem

class PersonItem(DjangoItem):
    django_model = Person
    sex = scrapy.Field()>>> p = PersonItem()
>>> p['name'] = 'John'
>>> p['age'] = '22'
>>> p['sex'] = 'M'And we can override the fields of the model with your own:class PersonItem(DjangoItem):
    django_model = Person
    name = scrapy.Field(default='No Name')This is useful to provide properties to the field, like a default or any other
property that your project uses. Those additional fields won’t be taken into
account when doing aDjangoItem.save().CaveatsDjangoItemis a rather convenient way to integrate Scrapy projects with Django
models, but bear in mind that Django ORMmay not scale wellif you scrape a lot
of items (ie. millions) with Scrapy. This is because a relational backend isoften not a good choice for a write intensive applications(such as a web
crawler), specially if the database is highly normalized and with many indices.SetupTo use the Django models outside the Django application you need to set up theDJANGO_SETTINGS_MODULEenvironment variable and –in most cases– modify
thePYTHONPATHenvironment variable to be able to import the settings
module.There are many ways to do this depending on your use case and preferences.
Below is detailed one of the simplest ways to do it.Suppose your Django project is namedmysite, is located in the path/home/projects/mysiteand you have created an appmyappwith the modelPerson. That means your directory structure is something like this:/home/projects/mysite
├── manage.py
├── myapp
│   ├── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
└── mysite
    ├── __init__.py
    ├── settings.py
    ├── urls.py
    └── wsgi.pyThen you need to add/home/projects/mysiteto thePYTHONPATHenvironment variable and set up the environment variableDJANGO_SETTINGS_MODULEtomysite.settings. That can be done in your
Scrapy’s settings file by adding the lines below:import sys
sys.path.append('/home/projects/mysite')

import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'Notice that we modify thesys.pathvariable instead thePYTHONPATHenvironment variable as we are already within the python runtime. If everything
is right, you should be able to start thescrapy shellcommand and import
the modelPerson(i.e.from myapp.models import Person).Starting withDjango 1.8you also have to explicitly set upDjangoif using
it outside amanage.pycontext
(seeDjango Docs):import django
django.setup()DevelopmentTest suite from thetestsdirectory can be run usingtoxby running:tox…using the configuration intox.ini. ThePythoninterpreters
used have to be installed locally on the system.Changelogv1.1.1 (2016-05-04)Distribute as universal wheelFix README’s markupv1.1 (2016-05-04)Python 3.4/3.5supportMaking tests work withDjango 1.9againv1.0 (2015-04-29)Initial version