tnc-ba/strongTNC — pull request #159 by dbrgn (2014-05-04 13:43:58+00:00)
Title: Importing tags with >999 files fails

When using the `importswid` command on `Dropbox/BA/swid-tags-full-jfu.txt`, it fails on several packages with the following trace:

```python
Traceback (most recent call last):
File "./manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/mnt/data/Projects/ba/strongTNC/apps/swid/management/commands/importswid.py", line 38, in handle
tag = utils.process_swid_tag(tag_xml)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/transaction.py", line 371, in inner
return func(*args, **kwargs)
File "/mnt/data/Projects/ba/strongTNC/apps/swid/utils.py", line 73, in process_swid_tag
tag.files = file_pks
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/fields/related.py", line 842, in __set__
manager.add(*value)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/fields/related.py", line 583, in add
self._add_items(self.source_field_name, self.target_field_name, *objs)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/fields/related.py", line 658, in _add_items
new_ids = new_ids - set(vals)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/query.py", line 96, in __iter__
self._fetch_all()
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/query.py", line 854, in _fetch_all
self._result_cache = list(self.iterator())
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/query.py", line 1065, in iterator
for row in self.query.get_compiler(self.db).results_iter():
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
for rows in self.execute_sql(MULTI):
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/backends/util.py", line 69, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/home/danilo/.virtualenvs/strongtnc/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 451, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: too many SQL variables
```

This appears to be an SQLite issue. Apparently SQLite does not allow more than 999 parameters to be present. If more than 999 files are assigned to a tag, all those PKs are sent to the DB in a single query.

We could start splitting the queries into smaller chunks. Although I still think that SQLite should be avoided and that the docs and README should discourage its use.