Repository: dbt-labs/dbt-core

Issue:
Write a script to migrate pre-0.14 archive tables to 0.14 archive tables
We should provide a migration tool/script to help users migrate from < 0.14.0 to >= 0.14.0 archives.

**Overview of changes to archival:**
- Code migration:
  - use `archive` block in a .sql file instead of yml in `dbt_project.yml`
  - specify the `timestamp` strategy in the block config
  - note: archives are now nodes, and so could collide with existing models
- Database migration:
  - rename old-style column names to new-style column names

**Details:**

We should furnish a script (or similar) that:
1. migrates archive tables in the database
2. modifies a dbt project to represent archives as blocks in .sql files

### Migrating archive tables
This script should migrate the following column names in-place (without changing their types)!

| old column name | new column name | notes |
| --- | --- | --- |
| "valid_from" | dbt_valid_from | preserve type |
| "valid_to" | dbt_valid_to | preserve type |
| "updated_at" | dbt_updated_at | preserve type |
| "scd_id" | dbt_scd_id | |

Resources:
1. [rename columns on Snowflake](https://docs.snowflake.net/manuals/sql-reference/sql/alter-table-column.html)
2. [rename columns on Redshift (+ Postgres?)](https://docs.aws.amazon.com/redshift/latest/dg/r_ALTER_TABLE_examples_basic.html#r_ALTER_TABLE_examples_basic-rename-a-column)
3. [rename columns on BigQuery](https://cloud.google.com/bigquery/docs/manually-changing-schemas#changing_a_columns_name)
  - Columns can't be renamed
  - Tables can't be renamed
  - The workaround is to write a select statement that projects column names
  - The workflow (unpleasant):
    - Make a copy of the table with a temporary suffix
    - Drop the original table
    - Copy from the temp table back to the original table, substituting correct column names

Considerations:
1. The old column names will be quoted, but the new column names _should not_ be quoted
2. It's probably a good idea to make a backup table and not drop it in this script
  - The script can provide instructions around dropping the backup tables after validating the migration

Ways that this can go wrong:
1. Views exist which select from the columns we want to migrate
  - Failure mode: Redshift/Postgres specific: the alter column queries may fail
  - Solution: Inform the users that they must drop dependent views to use the script
2. The user has implemented their own overrides for part (or all) of the archive materializations
  - Failure mode: undefined behavior
  - Solution: Really good documentation about who should use this script
3. Not all dbt users upgrade to 0.14.0 simultaneously, and someone runs `dbt archive` with <0.14.0
  - Failure mode: dbt fails with an error like `column "valid_from" does not exist`
  - Solution: be grateful that this doesn't cause destructive behavior!
4. The table that is being archived already contains columns called `dbt_valid_from`, `dbt_valid_to`, etc (unlikely)
  - Failure mode: the migration script will fail when renaming columns
  - Solution: the user will need to reconcile this manually for archival to work in 0.14.0

### Migrating archive definitions
The provided script should generate archive blocks which implement the same logic as the definitions in `dbt_project.yml`. The script _may_ delete the entries from `dbt_project.yml` after re-creating blocks in an `archives/` directory. It can alternatively provide instructions for the users to remove these configs manually.

Considerations:
1. Create one file per archive in the `archives/` directory
2. Name each archive file `archives/{{ target_table }}.sql`
3. Each block should be named `target_table`
4. Each block should supply the following `config` values where present in the original archive spec:
 - `alias` via `target_table`
 - `target_schema`
 - `target_database` (if provided in original spec)
 - `updated_at`
 - `unique_key`
 - `strategy='timestamp'` (implicitly)
5. Each block should archive the following select statement:
```
-- If source_database is defined:
select * from "{{ source_database }}"."{{ source_schema }}"."{{ source_table }}"

-- If source_database is not defined:
select * from "{{ source_schema }}"."{{ source_table }}"
```

**Note:**: If we were exceedingly clever, we'd generate schema.yml files that defined these source tables. That is out of scope for this migration script.
**Note 2:**: Is `source()` available in the `archive` block context?

Ways this can go wrong:
1. The `archives/` directory already exists
  - Failure mode: The script should fail with a helpful error message
  - Solution: The user should delete or rename their `archives/` directory to use the script
2. Values of `target_table` are duplicated across archives (eg. with different target schemas)
  - Failure mode: The script should fail with a helpful error message
  - Solution: The user should create their archive blocks manually


### The script
I imagine this script working broadly like [upgrade_dbt_schema_tests_v1_to_v2.py](https://github.com/fishtown-analytics/dbt/blob/dev/wilt-chamberlain/core/scripts/upgrade_dbt_schema_tests_v1_to_v2.py). This may be challenging, as the script will need to execute database-specific queries against each of the core databases that dbt supports.

Does it make sense to build this right into dbt as a subcommand? I can imagine:
```
$ dbt migrate upgrade-archives-to-0.14.0
```

I really dislike the idea of doing this, but the alternative would be to import dbt as a library and then invoke adapter functions, right? Can you think of a good way to handle this? Forgetting implementation, I imagine the spec is:

```
<script>
  --logfile-path <path to logfile>
  --migrate=<code|tables|both>
  --apply
```

When `apply` is provided, this script should actually migrate the target tables and generate archive blocks. When `apply` is not provided, this script should just log a series of actions that it _would_ undertake (possibly verifying that everything looks as expected along the way, IE that the target table is an old-style archive).

```
$ <script> --apply --migrate both

Migrating <count> archives tables on <database>

Migrating archive table <x> of <count>: <database>.<schema>.<table>
 - Verifying table <database>.<schema>.<table>
 - Making a backup of <database>.<schema>.<table>
 - Migrating <database>.<schema>.<table>
 - Table successfully migrated
 - Migrating archive code to archives/<table>.sql

Migrating archive table <x> of <count>: <database>.<schema>.<table>
 - Verifying table <database>.<schema>.<table>
 - Making a backup of <database>.<schema>.<table>
 - Migrating <database>.<schema>.<table>
 - Table successfully migrated
 - Migrating archive code to archives/<table>.sql

OK: <count> tables migrated.
```


Discussion:

Resolving patch:
diff --git a/core/dbt/main.py b/core/dbt/main.py
--- a/core/dbt/main.py
+++ b/core/dbt/main.py
@@ -23,6 +23,7 @@
 import dbt.task.freshness as freshness_task
 import dbt.task.run_operation as run_operation_task
 from dbt.task.list import ListTask
+from dbt.task.migrate import MigrationTask
 from dbt.task.rpc_server import RPCServerTask
 from dbt.adapters.factory import reset_adapters
 
@@ -624,6 +625,38 @@ def _build_run_operation_subparser(subparsers, base_subparser):
     return sub
 
 
+def _build_snapshot_migrate_subparser(subparsers, base_subparser):
+    sub = subparsers.add_parser(
+        'snapshot-migrate',
+        parents=[base_subparser],
+        help='Run the snapshot migration script'
+    )
+    sub.add_argument(
+        '--from-archive',
+        action='store_true',
+        help=('This flag is required for the 0.14.0 archive to snapshot '
+              'migration')
+    )
+    sub.add_argument(
+        '--apply-files',
+        action='store_true',
+        dest='write_files',
+        help='If set, write .sql files to disk instead of logging them'
+    )
+    sub.add_argument(
+        '--apply-database',
+        action='store_true',
+        dest='migrate_database',
+        help='If set, perform just the database migration'
+    )
+    sub.add_argument(
+        '--apply',
+        action='store_true',
+        help='If set, implies --apply-database --apply-files'
+    )
+    sub.set_defaults(cls=MigrationTask, which='migration')
+
+
 def parse_args(args):
     p = DBTArgumentParser(
         prog='dbt',
@@ -709,6 +742,7 @@ def parse_args(args):
     _build_debug_subparser(subs, base_subparser)
     _build_deps_subparser(subs, base_subparser)
     _build_list_subparser(subs, base_subparser)
+    _build_snapshot_migrate_subparser(subs, base_subparser)
 
     snapshot_sub = _build_snapshot_subparser(subs, base_subparser)
     archive_sub = _build_snapshot_subparser(subs, base_subparser, 'archive')
diff --git a/core/dbt/task/migrate.py b/core/dbt/task/migrate.py
new file mode 100644
--- /dev/null
+++ b/core/dbt/task/migrate.py
@@ -0,0 +1,293 @@
+import itertools
+import os
+
+from dbt.task.base import BaseTask
+
+from dbt.config import RuntimeConfig
+from dbt.adapters.factory import get_adapter
+from dbt.logger import GLOBAL_LOGGER as logger
+from dbt.clients import system
+from dbt.exceptions import RuntimeException
+
+
+_UNQUOTE_RENAME_COLUMNS = (
+    ('valid_from', 'dbt_valid_from'),
+    ('valid_to', 'dbt_valid_to'),
+    ('scd_id', 'dbt_scd_id'),
+)
+
+
+SNAPSHOT_TEMPLATE = '''
+{{% snapshot {name} %}}
+    {{{{
+        config({kwargs})
+    }}}}
+    select * from {source_relation}
+{{% endsnapshot %}}
+'''
+
+
+class Migrator(object):
+    """Migrate a single archive config"""
+    def __init__(self, manager, adapter, archive_def):
+        self.manager = manager
+        self.archive_def = archive_def
+        self.adapter = adapter
+        self.relation = adapter.Relation.create(
+            database=self.archive_def['target_database'],
+            schema=self.archive_def['target_schema'],
+            identifier=self.archive_def['target_table'],
+            quote_policy=self.manager.config.quoting
+        )
+        self.backup = self.append_relation_name()
+        self.snapshot_path = os.path.join(self.manager.snapshot_root,
+                                          self.relation.identifier + '.sql')
+
+    def get_renamed_columns(self, quote=True):
+        columns = []
+
+        unquote = _UNQUOTE_RENAME_COLUMNS
+        if self.adapter.type() == 'snowflake':
+            unquote = itertools.chain(
+                _UNQUOTE_RENAME_COLUMNS,
+                [('dbt_updated_at', 'dbt_updated_at')]
+            )
+
+        for old, new in unquote:
+            if quote:
+                old = self.adapter.quote(old)
+            columns.append((old, new))
+        return columns
+
+    def append_relation_name(self, end='_dbt_archive_migration_backup'):
+        new_name = self.relation.identifier + end
+        backup = self.relation.incorporate(
+            path={'identifier': new_name}, table_name=new_name
+        )
+        return backup
+
+    def migrate_archive_ctas(self, dest):
+        # get the columns
+        columns = self.adapter.get_columns_in_relation(self.relation)
+        if len(columns) == 0:
+            # the archive target must not exist? Continue, that is ok.
+            logger.info('  - Table {} does not exist, nothing to migrate.'
+                        .format(self.relation))
+            return
+        cols = {c.name.lower(): c.name for c in columns}
+        renames = self.get_renamed_columns()
+        select_parts = []
+        select_as_parts = []
+        for old, new in renames:
+            key = old.strip('"').lower()
+            if key not in cols:
+                raise Exception(
+                    'expected column like {} not but it is not in the table!'
+                    .format(key)
+                )
+            del cols[key]
+            select_as_parts.append('{} as {}'.format(old, new))
+
+        for column in columns:
+            name = column.name
+            if name.lower() in cols:
+                select_parts.append(self.adapter.quote(name))
+
+        selections = ', '.join(itertools.chain(select_parts, select_as_parts))
+        ctas = 'create table {!s} as (select {} from {!s})'.format(
+            dest, selections, self.relation
+        )
+        self.adapter.execute(ctas)
+
+    def migrate_archive_postgres(self):
+        """Migrate the archive using "alter table" commands to rename columns.
+        """
+        self.adapter.connections.add_begin_query()
+        tmp = self.append_relation_name('_dbt_archive_migration_tmp')
+        logger.debug('  - Making new archive at {}'.format(tmp))
+        self.migrate_archive_ctas(dest=tmp)
+        logger.info('  - Backing up table to {}'.format(self.backup))
+        self.adapter.rename_relation(self.relation, self.backup)
+        logger.debug('  - Renaming temp archive to final')
+        self.adapter.rename_relation(tmp, self.relation)
+        self.adapter.connections.add_commit_query()
+
+    def migrate_archive_snowflake(self):
+        """Migrate the archive by create table as select ..."""
+        logger.debug('  - Making new archive at {}'.format(self.backup))
+        self.migrate_archive_ctas(dest=self.backup)
+        logger.info('  - Backing up table to {}'.format(self.backup))
+        self.adapter.execute('alter table {!s} swap with {!s}'.format(
+            self.relation, self.backup
+        ))
+
+    def migrate_archive_bigquery(self):
+        """Migrate the archive by select * EXCEPT(...) into itself
+        """
+        logger.info('  - Backing up table to {}'.format(self.backup))
+        self.adapter.execute(
+            'create table {} as (select * from {})'
+            .format(self.backup, self.relation)
+        )
+        columns = self.get_renamed_columns()
+        except_str = ', '.join(o for o, _ in columns)
+        rename_str = ', '.join('{} as {}'.format(o, n) for o, n in columns)
+        sql = 'select * EXCEPT({}), {} from {!s}'.format(
+            except_str, rename_str, self.relation
+        )
+        self.adapter.connections.create_table(
+            database=self.relation.database,
+            schema=self.relation.schema,
+            table_name=self.relation.identifier,
+            sql=sql
+        )
+
+    def migrate_archive_table(self):
+        logger.info('  - Starting table migration')
+
+        if self.adapter.type() == 'bigquery':
+            self.migrate_archive_bigquery()
+        elif self.adapter.type() == 'snowflake':
+            self.migrate_archive_snowflake()
+        else:
+            self.migrate_archive_postgres()
+        logger.info('  - Finished table migration')
+
+    def build_archive_data(self):
+        source_relation = [
+            self.adapter.quote_as_configured(
+                self.archive_def['source_database'], 'database'
+            ),
+            self.adapter.quote_as_configured(
+                self.archive_def['source_schema'],
+                'schema'
+            ),
+            self.adapter.quote_as_configured(
+                self.archive_def['source_table'],
+                'identifier'
+            ),
+        ]
+
+        kwargs = {
+            'target_database': self.archive_def['target_database'],
+            'target_schema': self.archive_def['target_schema'],
+            'updated_at': self.archive_def['updated_at'],
+            'strategy': 'timestamp',
+            'unique_key': self.archive_def['unique_key'],
+        }
+
+        return SNAPSHOT_TEMPLATE.format(
+            source_relation='.'.join(source_relation),
+            kwargs=repr(kwargs),
+            name=self.archive_def['target_table']
+        )
+
+    def write_file(self):
+        logger.debug('  - Writing snapshot to {}'.format(self.snapshot_path))
+        contents = self.build_archive_data()
+
+        wrote = system.make_file(path=self.snapshot_path, contents=contents)
+        if wrote:
+            logger.info('  - Wrote new snapshot file to {}'
+                        .format(self.snapshot_path))
+        else:
+            logger.error('  - Error: Could not write new snapshot file to {}'
+                         .format(self.snapshot_path))
+
+
+class ArchiveOkConfig(RuntimeConfig):
+    @classmethod
+    def from_args(cls, args, allow_archive_configs=True):
+        supercls = super(ArchiveOkConfig, cls)
+        return supercls.from_args(
+            args=args,
+            allow_archive_configs=allow_archive_configs
+        )
+
+
+class MigrationTask(BaseTask):
+    ConfigType = ArchiveOkConfig
+
+    def __init__(self, args, config):
+        if not args.from_archive:
+            raise RuntimeException(
+                'The --from-archive paramteter is required!'
+            )
+        if args.apply:
+            args.write_files = True
+            args.migrate_database = True
+        super(MigrationTask, self).__init__(args, config)
+
+        self.snapshot_root = os.path.normpath(self.config.snapshot_paths[0])
+        system.make_directory(self.snapshot_root)
+
+        self.backups_made = []
+        self.snapshots_written = []
+
+    def archive_definitions(self, adapter):
+        default_database = self.config.credentials.database
+        for archive in self.config.archive:
+            target_database = archive.get('target_database', default_database)
+            target_schema = archive['target_schema']
+            source_database = archive.get('source_database', default_database)
+            source_schema = archive['source_schema']
+            for table in archive['tables']:
+                table_copy = table.copy()
+                table_copy['target_database'] = target_database
+                table_copy['target_schema'] = target_schema
+                table_copy['source_database'] = source_database
+                table_copy['source_schema'] = source_schema
+
+                yield Migrator(self, adapter, table_copy)
+
+    def perform_migration_with(self, adapter):
+        archive_defs = list(self.archive_definitions(adapter))
+        num_defs = len(archive_defs)
+        logger.info('Found {} archive{} to migrate'.format(
+            num_defs, '' if num_defs == 1 else 's'
+        ))
+        logger.info('')
+        for idx, migrator in enumerate(archive_defs, start=1):
+            logger.info('Archive {} of {}: {}'
+                        .format(idx, len(archive_defs), migrator.relation))
+            if self.args.migrate_database:
+                migrator.migrate_archive_table()
+                self.backups_made.append(migrator.backup)
+            else:
+                logger.info('  - Skipping migration in dry-run mode')
+            if self.args.write_files:
+                migrator.write_file()
+                self.snapshots_written.append(migrator.snapshot_path)
+            else:
+                logger.info('  - Skipping new snapshot file in dry-run mode')
+            logger.info('')
+
+    def run(self):
+        adapter = get_adapter(self.config)
+        with adapter.connection_named('migration'):
+            self.perform_migration_with(adapter)
+
+        if self.backups_made:
+            logger.info('The following backup tables were created:')
+            for backup in self.backups_made:
+                logger.info('  - {!s}'.format(backup))
+            logger.info('')
+
+        if self.snapshots_written:
+            logger.info('The following snapshot files were created:')
+            for written in self.snapshots_written:
+                logger.info('  - {!s}'.format(written))
+            logger.info('')
+
+        applied_something = any((self.args.apply,
+                                 self.args.migrate_database,
+                                 self.args.write_files))
+        if applied_something:
+            msg = (
+                'After verifying the migrated tables in the database, please '
+                'drop the backup\ntables and remove any archive configs from '
+                'your dbt_project.yml file.'
+            )
+        else:
+            msg = 'Re-run this script with `--apply` to apply these migrations'
+        logger.info(msg)


Test patch: