Package: django-csvimport
Django CSV ImportEd Crewe - December 2023Overviewdjango-csvimport is a generic importer tool to allow the upload of CSV files for
populating data. The egg installs an admin csvimport model that has a file upload field.
Add a new csvimport and upload a comma separated values file or MS Excel file.The upload triggers the import mechanism which matches the header line of the files
field names to the fields in the selected model. Importing any rows that include all required fields.
Optionally required fields can be specified as part of the upload.
By default duplicate value rows are not inserted.The import can also be run as a custom command, ie manage.py importcsv filename
for possible use via cronjob etc.For CSV files import where their schema is unknown, and there is no existing model to import to, there
is another command, inspectcsv, to generate the model code from the CSV file, guessing data types from the data
using code fromhttps://messytables.readthedocs.orgThe core import code was based onhttp://djangosnippets.org/snippets/633/by Jonathan Holst.
It adds character encoding handling, model field and column autodetection, admin interface,
custom command etc.Version 3 - Dec 2023Test with Django 5 and Python 3.12Incorporate messytables relevant code, the types.py file, since its no longer supported for Python 3.12Use dateparser for auto-handling a much wider range of date formatsVersion 2 - Sept 2014New management command csvinspect to generate models from CSV filesGeneral code refactorManagement command renamed from csvimport to importcsvMore features to cope with bad encoding and date typesVersion Compatibilityversion 3.0  tested with Django 5.0 Python 3.12version 2.16 tested with Django 3.2.16 on Python 3.9.6version 2.14 tested with Django 3.0.5 on Python 3.7.6, 3.8.2version 2.13 was tested with Django 2.2.5 on Python 3.7.3version 2.6 was tested with Django 1.7, 1.8, 1.9, 1.10, 1.11 on Python 2.7.13, Python 3.5.6Please use version 2.1, eg. pip install django-csvimport==2.1
for Django versions prior to 1.7This Django >= 1.7 requirement is because django-csvimport uses the newly added AppConfig for versions > 2.1
(NB: To fix this issue you could install django-appconf to django 1.6 or earlier
and tweak csvimport to use it in csvimport.app)For really old Django versions < 1.4 you may have to dial back the versions until it works!Note that only versions > 2.2 are compatible with Python 3.4Installation instructionsAdd the following to the INSTALLED_APPS in the settings.py of your project:>>>  pip install django-csvimport
...
...  INSTALLED_APPS = (
...  ...
...  'csvimport.app.CSVImportConf',  # use AppConfig for django >=1.7 csvimport >=2.2
...  )
...
...  python manage.py migrate  (or syncdb if django < 1.9)Note that migrate has the core tables in 0001_initial migration and test tables in 0002 so
rm migrations/0002_test_models.py if you do not want these cluttering your databaseCustom commandsINSPECTCSVmanage.py inspectcsv importfile.csv > models.pyThis returns the code for a new models file with a guesstimated model for the CSV file.
Add it to your app then run>>> makemigrations your_app
>>> migrateYou can then run the import to that model for importfile.csvNB: As it says its a guesstimate, you may have to manually tweak the generated models.py to get
the import to work better.If there are no headings in the CSV file, then it just uses automated ones col_1, col_2 … etc.IMPORTCSV(Please note this command used to be csvimport but that caused name clash issues with the module)manage.py importcsv –mappings=’’ –model=’app_label.model_name’ –delimiter=’t’ importfile.csvFor mappings enter a list of fields in order only if you dont have a header row
with matching field names - or you want to override it, eg.–mappings = ‘1=shared_code,2=org(otherapp.Organisation|name),3=date’where (model|foreign key field) is used to specify relations if again, you want to
override what would be looked up from your models.If you have no real field names in your csv file, then you can use
–mappings=’none’ and it will assume the fields are named col_1, col_2 … etc.Note that if you have a header row and specify mappings then it will treat the header as a data row, so delete it first.Admin interface importJust add a csvimport item, fill in the form and submit.
Failed import rows are added to the log field.Demonstration installation instructionsTo see how it works, you can install a demo easily enough eg. via virtual environment,
then use the tests settings to have some sample models for importing data, and the fixtures are sample csv files.Run the following in your shell:>>> virtualenv mysite
... cd mysite
... pip install django
... pip install django-csvimport
...
... cat > bin/django-admin.py << EOF
... #!/usr/bin/env python
... from django.core import management
... import os
... os.environ["DJANGO_SETTINGS_MODULE"] = "csvimport.settings"
... if __name__ == "__main__":
...     management.execute_from_command_line()
... EOF
...
... django-admin.py migrate
... django-admin.py runserverGo tohttp://127.0.0.1:8000/admin/in your browser - pay attention to the trailing / !Click on add CsvimportPick the django-csvimport/csvimport/tests/fixtures/countries.csv [1] and upload itCheck to see if the Country model is now populated.[1] also available fromhttps://raw.github.com/edcrewe/django-importcsv/master/importcsv/tests/fixtures/countries.csvAlternatively you can use the command line to uploaddjango-admin.py importcsv –model=’csvimport.Country’ django-csvimport/csvimport/tests/fixtures/countries.csv –settings=’csvimport.settings’TestsTo run the django-csvimport tests use:>>> bin/python3 -m django test --settings='csvimport.settings' csvimport.testsForeign KeysIt is not viable for csvimport to determine complex table relations.
However if it finds something marked as an ForeignKey with a lookup field in its model mappings, then it checks if the data exists already for the related model and pulls back an id for the field or creates a new entry if possible in the fkey model and pulls back a new id.For this to be useful then you need a related table that has a unique and more meaningful field that is being used in your data than a numeric primary key.eg. for an organisation column, org, that holds the unique name of the organisation from a separate table, you can addcolumn2=org(Organisation|name)to the mappings, so it knows that the org field relates to a separate Organisation table with a unique name field to be used for it to lookup and replace with org_id FKeyMore complex relationsFor any more sophisticated relations, eg. multiple keys, many to many fields etc.
The recommended approach is to create a temporary or intermediate import table that holds the data from your CSV file
with the lookup data in as columns, you can useinspectcsv importfile.csv > models.pyto automatically generate the import model from your CSV file.Whenever you do an import to that table you would use a bulk insert database query to take the data in it and populate complex relations of the final model tables appropriately.
If imports are happening repeatedly, eg. once a day, you retain your import CSV format table, and can add a database trigger for the table to automatically run your stored data conversion synchronisation query into the target tables.DateTime  dataNote that the importer uses dateparser to try to convert any datetime types you have in your CSV file.
Seehttps://dateparser.readthedocs.io/en/latest/settings.htmlfor DATEPARSER_SETTINGS env variable as a JSON map.AcknowledgementsThis package was created as part of a django dash at the House of Omni, Bristol UK, organised
by Dan Fairs and my local django users group, #DBBUG. It was a core component for an application
for aid agency supply chain sharing, prompted by Fraser Stephens of the HELIOS foundation
and developed by Ed Crewe and Tom Dunham.Other Django CSV importershttps://github.com/uptick/django-model-importhttps://pypi.org/project/csvImporter/https://pypi.org/project/django-csv-import/ChangelogContributors listed as [github.com username] where they are on github3.2 - Use dateparser for import - 26 Dec 2023Use dateparser for importing a much wider range of mixed date formats without specifying a formatRemove old tz monkeypatch3.1 - Add messytables for django 5 - 23 Dec 2023Use dateparser rather than python-dateutil and add to setup.py dependencies3.0 - Add messytables for django 5 - 23 Dec 2023Upgrade to Django 5, hence Python 3.12Add csvimport/messytables with two files to provide functionality used from OKN messytables, since the whole package is no longer maintained for Python 3.12 [OKN]Autoformat the code with black2.17 - Fix for django 4 - 2 Jan 2023Fix incompatible methods for django 4Add a test for single row CSV and fix issue of trailing line returnsMinor issue PRs [edouard-gv]2.16 - Fix and add tests - 28 June 2020Move test fixture migrations to separate migration folder picked up by custom testrunnerAdd regression tests for all reported issuesAdd admin UI testsEnable and fix all existing testsUse regex match group for failover CSV splitter2.15 - Fix compatiblity issue for Django 3 - 27 April 20202.13 - Fix issue with inspectcsv for different header and data line number of cols - 26th Sept 20192.12 - Fix for latest django required on_delete and duplicated log msgs - 24 April 20192.11 - Fix command line summary messages - 14 July 20172.10 - Bug with models var not set if not bulk import - 6 July 20172.9 - Fix for using foreign keys from other apps - 4 July 2017Handle boolean types actually being ints for sqliteAllow dots in –mappings so app labels of foreign keys can be set by the user, since model lookup may assume wrong label.2.8 - Minor fixes - 19 June 2017Force datetimes to local timezone to prevent warning on insertRebuild initial migrations fileAdd option to do bulk create with –bulk to speed up import2.7 - Merge in pull requests for minor fixes and test with Django 1.11 - 30 May 2017Fix uninitiated variable bug if type not detectedFix command line arg usage for Django 1.10 or 1.11 and make it backwards compatible down to 1.7For multiple django version compatibility code try excepts make sure new syntax is first to avoid deprecation warningsAdd test shell script to make it easy to run test suite and a command line testReport back matched and unmatched cols in CSV rather than just showing list of all of themAllow clean=false option to not fix numeric values to match allowed rangeUse atomic to wrap row inserts so we can still report count of inserts Ok after a transactional exception2.5 - Merge in various pull requests and test with Django 1.7 to 1.10 - 15 Nov 2016Fix for all recent versions of Django and PythonAdd in delimiter option eg. for tab separated valuesAdd migrationsMake importcsv parsing more tolerant of unparsable rows in CSV2.4 - Fix app config path so that management commands are found - 11 Nov 2014Fix log print for commands2.3 - Make compatible with Python 3 - 29 October 20142.2 - Move to core appconfig usage - 28 October 2014Add appconfig path to cater for module install test runningCheck for SMALLINT_DBS including sqlite - since can fail big int imports2.1 - Fix for django 1.7 - 27 October 20142.0 - Added model creation and datetime clean up - 29th Sept 2014Generate django model files based on CSV data type sniffing with messytablesUse flag to generate model code eg. –model=create_new_model.modelname and add testUse the settings date formats and add clean up for themFactor out type clean up method for data valuesAllow defaults to be passed as a command line argumentUse –mappings=none for CSV created models with no column name rowHandle badly encoded files as rawSet date limit in case of strftime bug for pre-1900 dates[edcrewe]1.1 - Added mappings and defaults tests - 18 March 2014Allow mapping format to be simple header row styleRefactored heading parsing and added tests for optional args[edcrewe]1.0 - Added signals and code cleanup - 29th January 2013Fixed number overflow test and tested with django 1.5.1 [edcrewe]Added signals to the app, to allow the model to custom format a row.added appconf to allow the app to be more customizable - eg. select models for import.Removed except all and return database errors.[cwood]0.9 - Missed import of re for change to display of error log - 9th Dec 2012Import re in csvimport/models.py [blorenz]0.8 - Tidy up logging and Boolean handling - 9th Dec 2012Fix display of error log so there are newlines for each entry [smeyfroi]Log processed rows to the csvimport.management.commands.csvimport logger [smeyfroi]Fix for boolean data [smeyfroi]Add setting for using standard logging machinery for Admin UI log lines [edcrewe]Add test to check creation and content of csvimport log file [edcrewe]0.7 - Fixes and merge in changes for running up the test app - 24th Nov 2012Merge in changes for 1.4 templates in settings [elena]Merge in doc changes [elena, valhallasw]Fix bug with columns that are floats [imposeren]Fix imports via the admin not having charset encoding argument [edcrewe]0.6 - Handle text not number or special float to integer - 7th March 2012Handle numeric columns with text in - zero if not nan or inf[edcrewe]0.5 - Add command line encoding option and error return - 6th March 2012Make sure command line usage returns errors and warningsAdd the option to force –charset=utf-8 for example rather than chardetAdd numeric type handling for empty fields or other issues plus tests[Tessa Alexander, edcrewe]0.4 - Add settings to allow demo site install from tests - 4th October 2011Add demo countries.csv fileAdd MEDIA_ROOT for file uploads[edcrewe]0.3 - Fix issue with adding in admin - 25th Sept 2011Fix empty mapping submittedRemove non-generic filename country processing codeUse get_models to populate drop down for admin import formAdd upload message about whether mapping is from CSV header rowEnsure header row only used if mapping is not supplied.[edcrewe]0.2 - Initial beta release version - 29th July 2011Build it as a django-csvimport eggCreate csvimport management commandImprove character set detection and unicode handlingAdd admin csvimport model to upload CSV files with logging of importUse file upload save to trigger csvimport commandAutodetect column mappings by matching model fields and CSV header textAdd a deduplicate featureStart to add test suite with data / app for HELIOS cross agency supply chain
data sharinghttp://www.helios-foundation.org/cbha-project/index_html[edcrewe]0.1 - UnreleasedUse django snippet as starting pointhttp://djangosnippets.org/snippets/633/[Jonathan Holst]