Repository: spotify/luigi

Issue:
Lets use Text type for value column of TaskParameter table
## Description

Change `value` column type from `string(256)` to `text` because of the issues on insert to task parameters table.

[error](https://gist.github.com/oivoodoo/0fb78f5a041bb08939f30f4af10c00c6) 
## Motivation and Context

Found this issue from the log file in our production system.
## Have you tested this? If so, how?

Going to test on 17.10.2016 in the next our deploy.

luigi parameter value column is set to limit size of 256
<!---
We use GitHub issues mainly for tracking bugs and feature requests.
Questions for how to use luigi can be sent to the mailing list.

Currently, there are no strict procedures or guidelines for submitting issues.
In short, please just use common sense.

Common sense includes this at bare-minimum:

 * search for similar issues posted before creating a new issue.
 * Use markdown to format all code/logs. Issues which are hard to read
   when rendered on GitHub might be closed with a friendly reminder of this.
 * If applicable, reading relevant parts of the documentation.

Also, add steps to reproduce the bug, if applicable. Sample code would be nice too :)

For more information on how to submit valuable contributions,
see https://opensource.guide/how-to-contribute/#how-to-submit-a-contribution
-->

But we need values longer than that. Can the limit be raised? In postgres this can just be `TEXT` type.


Discussion:
@oivoodoo, thanks for your PR! By analyzing the history of the files in this pull request, we identified @steenzout, @erikbern and @ystopia to be potential reviewers.

LGTM but if somebody else could review that would be good.

Seems like you need to add an upgrade step to `_upgrade_schema` to migrate old schemas on a `luigi` upgrade?

https://github.com/spotify/luigi/blob/68fa7bc467ae9cb142206e202a105252bddb6517/luigi/db_task_history.py#L243


Resolving patch:
diff --git a/luigi/db_task_history.py b/luigi/db_task_history.py
--- a/luigi/db_task_history.py
+++ b/luigi/db_task_history.py
@@ -196,7 +196,7 @@ class TaskParameter(Base):
     __tablename__ = 'task_parameters'
     task_id = sqlalchemy.Column(sqlalchemy.Integer, sqlalchemy.ForeignKey('tasks.id'), primary_key=True)
     name = sqlalchemy.Column(sqlalchemy.String(128), primary_key=True)
-    value = sqlalchemy.Column(sqlalchemy.String(256))
+    value = sqlalchemy.Column(sqlalchemy.Text())
 
     def __repr__(self):
         return "TaskParameter(task_id=%d, name=%s, value=%s)" % (self.task_id, self.name, self.value)
@@ -247,10 +247,35 @@ def _upgrade_schema(engine):
     :param engine: SQLAlchemy engine of the underlying database.
     """
     inspector = reflection.Inspector.from_engine(engine)
-    conn = engine.connect()
-
-    # Upgrade 1.  Add task_id column and index to tasks
-    if 'task_id' not in [x['name'] for x in inspector.get_columns('tasks')]:
-        logger.warn('Upgrading DbTaskHistory schema: Adding tasks.task_id')
-        conn.execute('ALTER TABLE tasks ADD COLUMN task_id VARCHAR(200)')
-        conn.execute('CREATE INDEX ix_task_id ON tasks (task_id)')
+    with engine.connect() as conn:
+
+        # Upgrade 1.  Add task_id column and index to tasks
+        if 'task_id' not in [x['name'] for x in inspector.get_columns('tasks')]:
+            logger.warning('Upgrading DbTaskHistory schema: Adding tasks.task_id')
+            conn.execute('ALTER TABLE tasks ADD COLUMN task_id VARCHAR(200)')
+            conn.execute('CREATE INDEX ix_task_id ON tasks (task_id)')
+
+        # Upgrade 2. Alter value column to be TEXT, note that this is idempotent so no if-guard
+        if 'mysql' in engine.dialect.name:
+            conn.execute('ALTER TABLE task_parameters MODIFY COLUMN value TEXT')
+        elif 'oracle' in engine.dialect.name:
+            conn.execute('ALTER TABLE task_parameters MODIFY value TEXT')
+        elif 'mssql' in engine.dialect.name:
+            conn.execute('ALTER TABLE task_parameters ALTER COLUMN value TEXT')
+        elif 'postgresql' in engine.dialect.name:
+            conn.execute('ALTER TABLE task_parameters ALTER COLUMN value TYPE TEXT')
+        elif 'sqlite' in engine.dialect.name:
+            # SQLite does not support changing column types. A database file will need
+            # to be used to pickup this migration change.
+            for i in conn.execute('PRAGMA table_info(task_parameters);').fetchall():
+                if i['name'] == 'value' and i['type'] != 'TEXT':
+                    logger.warning(
+                        'SQLite can not change column types. Please use a new database '
+                        'to pickup column type changes.'
+                    )
+        else:
+            logger.warning(
+                'SQLAlcheny dialect {} could not be migrated to the TEXT type'.format(
+                    engine.dialect
+                )
+            )


Test patch: