Repository: simonw/sqlite-utils

Issue:
insert_all(..., alter=True) should work for new columns introduced after the first 100 records
Is there a way to make `.insert_all()` work properly when new columns are introduced outside the first 100 records (with or without the `alter=True` argument)?

I'm using `.insert_all()` to bulk insert ~3-4k records at a time and it is common for records to need to introduce new columns.  However, if new columns are introduced after the first 100 records, `sqlite_utils` doesn't even raise the `OperationalError: table ... has no column named ...` exception; it just silently drops the extra data and moves on.

It took me a while to find this little snippet in the [documentation for `.insert_all()`](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts) (it's not mentioned under [Adding columns automatically on insert/update](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts)):

> The column types used in the CREATE TABLE statement are automatically derived from the types of data in that first batch of rows. **_Any additional or missing columns in subsequent batches will be ignored._**

I tried changing the `batch_size` argument to the total number of records, but it seems only to effect the number of rows that are committed at a time, and has no influence on this problem.

Is there a way around this that you would suggest?  It seems like it should raise an exception at least.

Comments:

Patch:
diff --git a/docs/python-api.rst b/docs/python-api.rst
index 9697fd3..5860397 100644
--- a/docs/python-api.rst
+++ b/docs/python-api.rst
@@ -410,7 +410,7 @@ Use it like this:
         "is_good_dog": True,
     }], pk="id", column_order=("id", "twitter", "name"))
 
-The column types used in the ``CREATE TABLE`` statement are automatically derived from the types of data in that first batch of rows. Any additional or missing columns in subsequent batches will be ignored.
+The column types used in the ``CREATE TABLE`` statement are automatically derived from the types of data in that first batch of rows. Any additional columns in subsequent batches will cause a ``sqlite3.OperationalError`` exception to be raised unless the ``alter=True`` argument is supplied, in which case the new columns will be created.
 
 The function can accept an iterator or generator of rows and will commit them according to the batch size. The default batch size is 100, but you can specify a different size using the ``batch_size`` parameter:
 
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index a8791c3..75599f6 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -1074,6 +1074,14 @@ class Table(Queryable):
                 all_columns = list(sorted(all_columns))
                 if hash_id:
                     all_columns.insert(0, hash_id)
+            else:
+                all_columns += [
+                    column
+                    for record in chunk
+                    for column in record
+                    if column not in all_columns
+                ]
+
             validate_column_names(all_columns)
             first = False
             # values is the list of insert data that is passed to the


Test patch:
diff --git a/tests/test_create.py b/tests/test_create.py
index a84eb8d..fc8edc0 100644
--- a/tests/test_create.py
+++ b/tests/test_create.py
@@ -707,13 +707,24 @@ def test_insert_thousands_using_generator(fresh_db):
     assert 10000 == fresh_db["test"].count
 
 
-def test_insert_thousands_ignores_extra_columns_after_first_100(fresh_db):
+def test_insert_thousands_raises_exception_wtih_extra_columns_after_first_100(fresh_db):
+    # https://github.com/simonw/sqlite-utils/issues/139
+    with pytest.raises(Exception, match="table test has no column named extra"):
+        fresh_db["test"].insert_all(
+            [{"i": i, "word": "word_{}".format(i)} for i in range(100)]
+            + [{"i": 101, "extra": "This extra column should cause an exception"}],
+        )
+
+
+def test_insert_thousands_adds_extra_columns_after_first_100_with_alter(fresh_db):
+    # https://github.com/simonw/sqlite-utils/issues/139
     fresh_db["test"].insert_all(
         [{"i": i, "word": "word_{}".format(i)} for i in range(100)]
-        + [{"i": 101, "extra": "This extra column should cause an exception"}]
+        + [{"i": 101, "extra": "Should trigger ALTER"}],
+        alter=True,
     )
     rows = fresh_db.execute_returning_dicts("select * from test where i = 101")
-    assert [{"i": 101, "word": None}] == rows
+    assert [{"i": 101, "word": None, "extra": "Should trigger ALTER"}] == rows
 
 
 def test_insert_ignore(fresh_db):