Repository: dbt-labs/dbt-core

Issue:
Support for postgres index creation?
Is there a way, using `dbt`, to create an index in postgres as part of a run? Given that everything is a `select`, it's not clear where this functionality would live, and I couldn't find anything in the docs or issues about it. 

Or, is the answer to just get out of postgres and move to BigQuery etc :) ?


Discussion:
Hey @ryantuck - you can accomplish this with a [post-hook](https://docs.getdbt.com/docs/using-hooks#section-configuring-hooks). You'll probably want to do something like

```sql
{{
  config({
    "materialized" : "table",
    "post-hook": 'create index if not exists {{ this.name }}__index_on_FIELD on {{ this }} ("FIELD")'
    })
}}

select ...
```

^ assuming the index you want to add is on a column called FIELD. If you find yourself doing this a lot, you can create a [macro](https://docs.getdbt.com/docs/macros) to add an index to an parameterized column. We also have a nascent [postgres package](https://github.com/fishtown-analytics/postgres) that does exactly this :)


I think in general, dbt works best with Redshift/Snowflake/BigQuery, but sounds like there are at least a few folks using it on postgres with some success! Hope this helps :)
An issue I've found with the suggested hook is it seems to only add an index every second run. But if I remove the `if not exists`, then it fails on it already existing (after the first run).

Is this something to do with how dbt handles the cleverness of the table creation and renames? Is there a fix that will solve the immediate problem?

Thank you!
Discussion for that at https://github.com/fishtown-analytics/dbt/issues/1945
Let's add first-class support for an `index` config to models running on Postgres. We support the analogues for Redshift/Snowflake/BigQuery -- we might as well support these natively on Postgres too.

We don't intend to prioritize this directly (we always recommend using a proper data warehouse with dbt like Redshift, Snowflake, or BigQuery), but I would be _very_ happy to work with a community contributor to implement this functionality in dbt!
Hi all. Just wanted to update this issue with the workaround (Proposed by drewbanin in https://github.com/fishtown-analytics/dbt/issues/1945#issuecomment-576826021) we are going to use @ DoSomething.org. For context: our former data lead created this related issue https://github.com/fishtown-analytics/dbt/issues/1945#issue-526684234.

### Macro

```
{% macro get_index_name(model, idx_name) %}
    {{ model.schema ~ '_' ~ model.table ~ '_' ~ idx_name ~ '_' ~ run_started_at.strftime('%Y%m%d_%H%M%S') ~ ' ON ' ~ model }}
{% endmacro %}
```

### Usage

Creates indexes consistently on every run for the given table in the pattern `<schema>_<table_name>_<idx_name>_<yyyymmdd_hhmmss>`, e.g. "public_post_actions_unique_20200804_235704"

```
models:
    ds_dbt:
        ...
        model_name:
            table_mame:
                alias: post_actions
                materialized: table
                 post-hook:
                    - "CREATE UNIQUE INDEX {{ get_index_name(this, 'unique') }} (created_at, id)"
```
@drewbanin I've gone through a few iterations of my own macro to create an index in Postgres. I'd love to contribute it. It handles the timestamp naming above as well as supports indexes on multiple columns (and chooses the best index name such that the index name is a unique function of the columns the index is on).

Should I contribute it to the main repo or to the [Postgres package](https://github.com/fishtown-analytics/postgres)? I'm a first time contributor and am happy to follow the process [here](https://github.com/fishtown-analytics/dbt/blob/dev/margaret-mead/CONTRIBUTING.md).
nice one @arzavj ! @jtcohen6 what do you reckon the right home for a macro like this would be?

Resolving patch:
diff --git a/plugins/postgres/dbt/adapters/postgres/impl.py b/plugins/postgres/dbt/adapters/postgres/impl.py
--- a/plugins/postgres/dbt/adapters/postgres/impl.py
+++ b/plugins/postgres/dbt/adapters/postgres/impl.py
@@ -1,21 +1,62 @@
+from datetime import datetime
 from dataclasses import dataclass
-from typing import Optional, Set
+from typing import Optional, Set, List, Any
 from dbt.adapters.base.meta import available
 from dbt.adapters.base.impl import AdapterConfig
 from dbt.adapters.sql import SQLAdapter
 from dbt.adapters.postgres import PostgresConnectionManager
 from dbt.adapters.postgres import PostgresColumn
 from dbt.adapters.postgres import PostgresRelation
+from dbt.dataclass_schema import dbtClassMixin, ValidationError
 import dbt.exceptions
+import dbt.utils
 
 
 # note that this isn't an adapter macro, so just a single underscore
 GET_RELATIONS_MACRO_NAME = 'postgres_get_relations'
 
 
+@dataclass
+class PostgresIndexConfig(dbtClassMixin):
+    columns: List[str]
+    unique: bool = False
+    type: Optional[str] = None
+
+    def render(self, relation):
+        # We append the current timestamp to the index name because otherwise
+        # the index will only be created on every other run. See
+        # https://github.com/fishtown-analytics/dbt/issues/1945#issuecomment-576714925
+        # for an explanation.
+        now = datetime.utcnow().isoformat()
+        inputs = (self.columns +
+                  [relation.render(), str(self.unique), str(self.type), now])
+        string = '_'.join(inputs)
+        return dbt.utils.md5(string)
+
+    @classmethod
+    def parse(cls, raw_index) -> Optional['PostgresIndexConfig']:
+        if raw_index is None:
+            return None
+        try:
+            cls.validate(raw_index)
+            return cls.from_dict(raw_index)
+        except ValidationError as exc:
+            msg = dbt.exceptions.validator_error_message(exc)
+            dbt.exceptions.raise_compiler_error(
+                f'Could not parse index config: {msg}'
+            )
+        except TypeError:
+            dbt.exceptions.raise_compiler_error(
+                f'Invalid index config:\n'
+                f'  Got: {raw_index}\n'
+                f'  Expected a dictionary with at minimum a "columns" key'
+            )
+
+
 @dataclass
 class PostgresConfig(AdapterConfig):
     unlogged: Optional[bool] = None
+    indexes: Optional[List[PostgresIndexConfig]] = None
 
 
 class PostgresAdapter(SQLAdapter):
@@ -42,6 +83,10 @@ def verify_database(self, database):
         # return an empty string on success so macros can call this
         return ''
 
+    @available
+    def parse_index(self, raw_index: Any) -> Optional[PostgresIndexConfig]:
+        return PostgresIndexConfig.parse(raw_index)
+
     def _link_cached_database_relations(self, schemas: Set[str]):
         """
         :param schemas: The set of schemas that should have links added.


Test patch:
diff --git a/test/integration/065_postgres_index_tests/data/seed.csv b/test/integration/065_postgres_index_tests/data/seed.csv
new file mode 100644
--- /dev/null
+++ b/test/integration/065_postgres_index_tests/data/seed.csv
@@ -0,0 +1,4 @@
+country_code,country_name
+US,United States
+CA,Canada
+GB,United Kingdom
diff --git a/test/integration/065_postgres_index_tests/models-invalid/invalid_columns_type.sql b/test/integration/065_postgres_index_tests/models-invalid/invalid_columns_type.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/065_postgres_index_tests/models-invalid/invalid_columns_type.sql
@@ -0,0 +1,10 @@
+{{
+  config(
+    materialized = "table",
+    indexes=[
+      {'columns': 'column_a, column_b'},
+    ]
+  )
+}}
+
+select 1 as column_a, 2 as column_b
diff --git a/test/integration/065_postgres_index_tests/models-invalid/invalid_type.sql b/test/integration/065_postgres_index_tests/models-invalid/invalid_type.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/065_postgres_index_tests/models-invalid/invalid_type.sql
@@ -0,0 +1,10 @@
+{{
+  config(
+    materialized = "table",
+    indexes=[
+      {'columns': ['column_a'], 'type': 'non_existent_type'},
+    ]
+  )
+}}
+
+select 1 as column_a, 2 as column_b
diff --git a/test/integration/065_postgres_index_tests/models-invalid/invalid_unique_config.sql b/test/integration/065_postgres_index_tests/models-invalid/invalid_unique_config.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/065_postgres_index_tests/models-invalid/invalid_unique_config.sql
@@ -0,0 +1,10 @@
+{{
+  config(
+    materialized = "table",
+    indexes=[
+      {'columns': ['column_a'], 'unique': 'yes'},
+    ]
+  )
+}}
+
+select 1 as column_a, 2 as column_b
diff --git a/test/integration/065_postgres_index_tests/models-invalid/missing_columns.sql b/test/integration/065_postgres_index_tests/models-invalid/missing_columns.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/065_postgres_index_tests/models-invalid/missing_columns.sql
@@ -0,0 +1,10 @@
+{{
+  config(
+    materialized = "table",
+    indexes=[
+      {'unique': True},
+    ]
+  )
+}}
+
+select 1 as column_a, 2 as column_b
diff --git a/test/integration/065_postgres_index_tests/models/incremental.sql b/test/integration/065_postgres_index_tests/models/incremental.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/065_postgres_index_tests/models/incremental.sql
@@ -0,0 +1,18 @@
+{{
+  config(
+    materialized = "incremental",
+    indexes=[
+      {'columns': ['column_a'], 'type': 'hash'},
+      {'columns': ['column_a', 'column_b'], 'unique': True},
+    ]
+  )
+}}
+
+select *
+from (
+  select 1 as column_a, 2 as column_b
+) t
+
+{% if is_incremental() %}
+    where column_a > (select max(column_a) from {{this}})
+{% endif %}
diff --git a/test/integration/065_postgres_index_tests/models/table.sql b/test/integration/065_postgres_index_tests/models/table.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/065_postgres_index_tests/models/table.sql
@@ -0,0 +1,14 @@
+{{
+  config(
+    materialized = "table",
+    indexes=[
+      {'columns': ['column_a']},
+      {'columns': ['column_b']},
+      {'columns': ['column_a', 'column_b']},
+      {'columns': ['column_b', 'column_a'], 'type': 'btree', 'unique': True},
+      {'columns': ['column_a'], 'type': 'hash'}
+    ]
+  )
+}}
+
+select 1 as column_a, 2 as column_b
diff --git a/test/integration/065_postgres_index_tests/snapshots/colors.sql b/test/integration/065_postgres_index_tests/snapshots/colors.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/065_postgres_index_tests/snapshots/colors.sql
@@ -0,0 +1,29 @@
+{% snapshot colors %}
+
+    {{
+        config(
+            target_database=database,
+            target_schema=schema,
+            unique_key='id',
+            strategy='check',
+            check_cols=['color'],
+            indexes=[
+              {'columns': ['id'], 'type': 'hash'},
+              {'columns': ['id', 'color'], 'unique': True},
+            ]
+        )
+    }}
+
+    {% if var('version') == 1 %}
+
+        select 1 as id, 'red' as color union all
+        select 2 as id, 'green' as color
+
+    {% else %}
+
+        select 1 as id, 'blue' as color union all
+        select 2 as id, 'green' as color
+
+    {% endif %}
+
+{% endsnapshot %}
diff --git a/test/integration/065_postgres_index_tests/test_postgres_indexes.py b/test/integration/065_postgres_index_tests/test_postgres_indexes.py
new file mode 100644
--- /dev/null
+++ b/test/integration/065_postgres_index_tests/test_postgres_indexes.py
@@ -0,0 +1,134 @@
+import re
+
+from test.integration.base import DBTIntegrationTest, use_profile
+
+
+INDEX_DEFINITION_PATTERN = re.compile(r'using\s+(\w+)\s+\((.+)\)\Z')
+
+class TestPostgresIndex(DBTIntegrationTest):
+    @property
+    def schema(self):
+        return "postgres_index_065"
+
+    @property
+    def models(self):
+        return "models"
+
+    @property
+    def project_config(self):
+        return {
+            'config-version': 2,
+            'seeds': {
+                'quote_columns': False,
+                'indexes': [
+                  {'columns': ['country_code'], 'unique': False, 'type': 'hash'},
+                  {'columns': ['country_code', 'country_name'], 'unique': True},
+                ],
+            },
+            'vars': {
+                'version': 1
+            },
+        }
+
+    @use_profile('postgres')
+    def test__postgres__table(self):
+        results = self.run_dbt(['run', '--models', 'table'])
+        self.assertEqual(len(results),  1)
+
+        indexes = self.get_indexes('table')
+        self.assertCountEqual(
+            indexes,
+            [
+              {'columns': 'column_a', 'unique': False, 'type': 'btree'},
+              {'columns': 'column_b', 'unique': False, 'type': 'btree'},
+              {'columns': 'column_a, column_b', 'unique': False, 'type': 'btree'},
+              {'columns': 'column_b, column_a', 'unique': True, 'type': 'btree'},
+              {'columns': 'column_a', 'unique': False, 'type': 'hash'}
+            ]
+        )
+
+    @use_profile('postgres')
+    def test__postgres__incremental(self):
+        for additional_argument in [[], [], ['--full-refresh']]:
+            results = self.run_dbt(['run', '--models', 'incremental'] + additional_argument)
+            self.assertEqual(len(results),  1)
+
+            indexes = self.get_indexes('incremental')
+            self.assertCountEqual(
+                indexes,
+                [
+                  {'columns': 'column_a', 'unique': False, 'type': 'hash'},
+                  {'columns': 'column_a, column_b', 'unique': True, 'type': 'btree'},
+                ]
+            )
+
+    @use_profile('postgres')
+    def test__postgres__seed(self):
+        for additional_argument in [[], [], ['--full-refresh']]:
+            results = self.run_dbt(["seed"] + additional_argument)
+            self.assertEqual(len(results),  1)
+
+            indexes = self.get_indexes('seed')
+            self.assertCountEqual(
+                indexes,
+                [
+                  {'columns': 'country_code', 'unique': False, 'type': 'hash'},
+                  {'columns': 'country_code, country_name', 'unique': True, 'type': 'btree'},
+                ]
+            )
+
+    @use_profile('postgres')
+    def test__postgres__snapshot(self):
+        for version in [1, 2]:
+            results = self.run_dbt(["snapshot", '--vars', 'version: {}'.format(version)])
+            self.assertEqual(len(results),  1)
+
+            indexes = self.get_indexes('colors')
+            self.assertCountEqual(
+                indexes,
+                [
+                  {'columns': 'id', 'unique': False, 'type': 'hash'},
+                  {'columns': 'id, color', 'unique': True, 'type': 'btree'},
+                ]
+            )
+
+    def get_indexes(self, table_name):
+        sql = """
+            SELECT
+              pg_get_indexdef(idx.indexrelid) as index_definition
+            FROM pg_index idx
+            JOIN pg_class tab ON tab.oid = idx.indrelid
+            WHERE
+              tab.relname = '{table}'
+              AND tab.relnamespace = (
+                SELECT oid FROM pg_namespace WHERE nspname = '{schema}'
+              );
+        """
+
+        sql = sql.format(table=table_name, schema=self.unique_schema())
+        results = self.run_sql(sql, fetch='all')
+        return [self.parse_index_definition(row[0]) for row in results]
+
+    def parse_index_definition(self, index_definition):
+        index_definition = index_definition.lower()
+        is_unique = 'unique' in index_definition
+        m = INDEX_DEFINITION_PATTERN.search(index_definition)
+        return {'columns': m.group(2), 'unique': is_unique, 'type': m.group(1)}
+
+class TestPostgresInvalidIndex(DBTIntegrationTest):
+    @property
+    def schema(self):
+        return "postgres_index_065"
+
+    @property
+    def models(self):
+        return "models-invalid"
+
+    @use_profile('postgres')
+    def test__postgres__invalid_index_configs(self):
+        results, output = self.run_dbt_and_capture(expect_pass=False)
+        self.assertEqual(len(results), 4)
+        self.assertRegex(output, r'columns.*is not of type \'array\'')
+        self.assertRegex(output, r'unique.*is not of type \'boolean\'')
+        self.assertRegex(output, r'\'columns\' is a required property')
+        self.assertRegex(output, r'Database Error in model invalid_type')