Package: django-pipe2db
django-pipe2dbContentsdjango-pipe2dbContentsConceptsFeaturesInstall and ImportQuick Start1. Using django orm as standalone2. Using with django projectUseageArgument of pipe decorator as contextmodelunique_keymethodrename_fieldsexclude_fieldsforeignkey_fieldsmanytomany_fields- See complicate context and data nested level examplecontentfile_fieldsConceptsA decorator that written by wrapping orm method of django modelsIt maps the relationship between the models and data via nested dictionaryFeaturesIt bridges Python functions and django modelsCreate and update data to database via modelsAutomatically create and modify tables by wrapping manage.py commands from django as makemigrations and migrateLoad minimum django settings for can use django orm as standalone that without using the django projectInsertion of data with the same relationship as foreignkey and manytomany fieldsInserting a content file object as an image fieldInstall and Importpipinstalldjango-pipe2db# crawler.pyfrompipe2dbimportpipefrompipe2dbimportsetupdbQuick Start1. Using django orm as standaloneCreate models.py in the directory that will be used as the Django appexample for minimum project directory structure.seeProject
│__main__.py
│
└─bookstore│insert.py│└─dbmodels.py# models.pyfromdjango.dbimportmodelsclassAuthor(models.Model):email=models.EmailField('Email',unique=True)first_name=models.CharField(max_length=100)last_name=models.CharField(max_length=100)date_of_birth=models.DateField(null=True,blank=True)date_of_death=models.DateField('Died',null=True,blank=True)classMeta:db_table='author'# insert.pyfrompipe2dbimportpipe,setupdbsetupdb()# find models automatically# setupdb('bookstore.db') # or more explicitly# The key of the data and the field names of the model are matchedauthor1={'email':'xman1@google.com','first_name':'charse','last_name':'javie','date_of_birth':'1975-07-25','date_of_death':'1995-07-11'}author2={'email':'yman1@google.com','first_name':'jin','last_name':'gray','date_of_birth':'1925-07-25','date_of_death':'1999-01-21'}@pipe({'model':'db.Author','unique_key':'email',# unique values of table as pk# 'method': 'update' # If uncomment, works in update mode})definsert(*args,Author,**kwargs):# You Can get model class via argumenting at generator function# from django.apps import apps # or via get_model method of django# Author = apps.get_model('db.Author')queryset=Author.objects.all()yield from[author1,author2,author3]run examplespythonbookstore/insert.py2. Using with django projectSince DJANGO_SETTINGS_MODULE is already setted, it's not need to call setupdbdjango site examplerun via shell which excuted by 'python manage.py shell' command of django managepythonmanage.pyshellIn[1]:fromyourpackage.insertimportinsertIn[2]:insert()idemailfirst_namelast_namedate_of_birthdate_of_death1xman1@google.comcharsejavie1975-07-251995-07-112yman1@google.comjingray1925-07-251999-01-213batman1@google.comwolverinjack1988-07-25NULLUseageArgument of pipe decorator as contextA context is a dictionary that describes the relationship between the model and the dataIn the following examples, the elements that make up the context are explained step by stepmodeldjango model to pipe data written as string literals# some_crawler.pyfrompip2dbimportpipe@pipe({'model':'db.Author'# 'model': 'yourapp.YourModel' on django project})defabc_crawler():...yieldrowIt is also a good way to assign and use a variable to increase reusability
When expressing nested relationships in relational data, not assigning them as variables can result in repeatedly creating the same context.# assign to variable crawler.py# It seems to better waycontext_author={'model':'db.Author'}@pipe(context_author)defabcd_crawler(*args,**kwargs):yield..It is also possible to specify the model by directly importing it, but in the case of standalone, you must declare setupdb before importing the model# dose not look good.pyfrompipe2dbimportsetupdb,pipesetupdb()from.db.modelsimportAuthorcontext_author={'model':Author}@pipe(context_author)defabc():yield..Another way to refer to the model classUsing Django's apps modulefromdjango.appsimportappsAuthor=apps.get_model('db.Author')Specify the model name as an argument to the generator function# An example of controlling a generator based on data in a database@pipe(context_author)defabc_crawler(rows,response,Author):visited=Author.objects.values_list('review_id',flat=True)forrowinrows:ifrow['id']invisited:breakyieldrowunique_keykey to identify data like as primary keyIf you don't specify it, creating data will be duplicatedTo identify data with one or several keys as unique_together# models.py# unique key modelclassAuthor(models.Model):...first_name=models.CharField(max_length=100,unique=True)...# uniqufy_by_one.pycontext_author={'model':'db.Author','unique_key':'first_name'}If uniqueness is not guaranteed with one key, add another# models.py# unique together modelclassAuthor(models.Model):...first_name=models.CharField(max_length=100)last_name=models.CharField(max_length=100)classMeta:unique_together=['first_name','last_name']...#unique_together.pycontext_author={'model':'db.Author','unique_key':['first_name','last_name']}methodCreates or updates data with a unique key specifiedDefaults is createIn create mode, data is inserted based on unique.In update mode as wrapper update_or_create of django method, creates records if they don't exist, otherwise modifies existing records# incorrect create.pyfrompipe2dbimportpipeauthor_incorrect={'email':'batman1@google.com','first_name':'who',# incorrect'last_name':'jackman','date_of_birth':'1988-07-25',# incorrect'date_of_death':None}context={'model':'db.Author','unique_key':'email',# 'method': 'create' no need to specify if create}@pipe(context)defgen_author(...):yieldauthor_incorrectresult tableidemailfirst_namelast_namedate_of_birthdate_of_death3batman1@google.comwhojackman1988-07-25NULL# correct as update.pyfrompipe2dbimportpipeauthor_corrected={'email':'batman1@google.com','first_name':'Hugh',# correct'last_name':'jackman','date_of_birth':'1968-10-12',# correct'date_of_death':None}context={'model':'db.Author','unique_key':'email','method':'update',# for update record by corrected data}@pipe(context)defgen_author(...):yieldauthor_correctedresult tableidemailfirst_namelast_namedate_of_birthdate_of_death3batman1@google.comHughjackman1968-10-12NULLrename_fieldsDictionary of between data and model as key:field mappingUsed when the data key and the model field name are different# models.pyfromdjango.dbimportmodelsclassAuthor(models.Models):......classBook(models.Model):title=models.CharField(max_length=200)isbn=models.CharField('ISBN',max_length=13,unique=True)classMeta:db_table='book'# book_crawler.pycontext={'model':'db.Book','unique_key':'isbn','rename_fields':{'header':'title','book_id':'isbn',}}# map header -> title, book_id -> isbn@pipe(context)defbook_crawler(abc,defg,jkl=None):book_list=[{'header':'oh happy day',# header to title'book_id':'1234640841',},{'header':'oh happy day','book_id':'9214644250',},]yield frombook_listexclude_fieldsList of keys to excludsUsed when the data has a key that is not in the field names in the modelFilter too much information from data that model cannot consume# bookcrawler.pyfrompipe2dbimportpipe......context={'model':'db.Book','unique_key':'isbn','rename_fields':{'header':'title','book_id':'isbn',},'exclude_fields':['status']# exclude}@pipe(context)defbook_crawler(abc,defg,jkl=None):book_list=[{'header':'oh happy day',# header to title'book_id':'1234640841','status':'on sales',# status is not needed in Book model},{'header':'oh happy day','book_id':'9214644250','sstatus':'no stock',},]yield frombook_listMapping of Relative Dataforeignkey_fieldsCreat records by generation according to the foreign key relationship between tablesRecursively nest parent dict to children dictThere are two way of create relationship data# models.py# two models of related with foreign keyfromdjango.dbimportmodelsclassAuthor(models.Model):email=models.EmailField('Email',unique=True)name=models.CharField(max_length=100)classMeta:db_table='author'classBook(models.Model):author=models.ForeignKey('Author',on_delete=models.CASCADE,null=True)# fkisbn=models.CharField('ISBN',max_length=13,unique=True)title=models.CharField(max_length=200)classMeta:db_table='book'# some crawler.pyfrompipe2dbimportpipe# 1. Generate data of book author nestedcontext_author={'model':'db.Author','unique_key':'email','method':'update'}context_book={'model':'db.Book','unique_key':'isbn','foreignkey_fields':{'book':context_author}}# author data is nested in book data@pipe(context_book)defparse_book():author1={'email':'pbr112@naver.com','name':'hs moon',}book={'author':author1,'title':'django-pipe2db','isbn':'291803928123'}yieldbook# some crawler.pyfrompipe2dbimportpipe# 2. Generate data of author and book sequentially@pipe(context_author)defparse_author():author1={'email':'pbr112@naver.com','name':'hs moon',}yieldauthor1# create author firstauthor1=parse_author()# create book after and connect fk relation to author@pipe(context_book)defparse_book():book={'author':author1['email'],# Since the author has already been created, it possible to pass email as pk of author only# 'author': author1, # or same as above'title':'django-pipe2db','isbn':'291803928123'}yieldbookmanytomany_fieldsCreate data for manytomany relationshipsGenerate data with nesting the children m2m data in the parent data key in the form of a list# models.pyfromdjango.dbimportmodelsclassBook(models.Model):title=models.CharField(max_length=200)isbn=models.CharField('ISBN',max_length=13,unique=True)genre=models.ManyToManyField('db.Genre')classMeta:db_table='book'classGenre(models.Model):name=models.CharField(max_length=200,unique=True)classMeta:db_table='genre'# m2m_generator.pyfrompipe2dbimportpipecontext_genre={'model':'db.Genre','unique_key':'name'}context_book={'model':'db.Book','unique_key':'isbn','manytomany_fields':{'genre':context_genre}}@pipe(context_book)defgen_book_with_genre():genre1={'name':'action'}genre2={'name':'fantasy'}book1={'title':'oh happy day','isbn':'2828233644','genre':[genre2],# nest genres to list}book2={'title':'python','isbn':'9875230846','genre':[genre1,genre2],}book3={'title':'java','isbn':'1234640841',# has no genre}yield from[book1,book2,book3]See complicate context and data nested level exampleCreate record with contentfilescontentfile_fieldsSaving file via ContentFile class from django.core.files modulesource_url_field is specified as meta data for determinding file name# models.pyfromdjango.dbimportmodelsclassBookImage(models.Model):img=models.ImageField()classMeta:db_table='bookimage'frompipe2dbimportpipe@pipe({'model':'db.BookImage','contentfile_fields':{'img':{'source_url_field':'src',}},'exclude_fields':['src']# when model dose not need src data})defimage_crawler(response):image_data={'img':'response_content','src':response.url#  needed for extracting filename as source_url_field}yieldimage_data