Repository: dbt-labs/dbt-core

Issue:
Make pg/redshift database verification case-insensitive
Database names are case sensitive in Postgres (and other databases).
When running DBT I got the following:
`ERROR: Cross-db references not allowed in postgres (stg_rammeraal_82136 vs STG_rammeraal_82136)`

To avoid this, please apply the following patch:
```diff --git a/plugins/postgres/dbt/adapters/postgres/impl.py b/plugins/postgres/dbt/adapters/postgres/impl.py
index af27aef2..f442281e 100644
--- a/plugins/postgres/dbt/adapters/postgres/impl.py
+++ b/plugins/postgres/dbt/adapters/postgres/impl.py
@@ -23,10 +23,8 @@ class PostgresAdapter(SQLAdapter):
     def verify_database(self, database):
         if database.startswith('"'):
             database = database.strip('"')
-        else:
-            database = database.lower()
         expected = self.config.credentials.database
-        if database != expected:
+        if database.lower() != expected.lower():
             raise dbt.exceptions.NotImplementedException(
                 'Cross-db references not allowed in {} ({} vs {})'
                 .format(self.type(), database, expected)```

This may be an issue with other database vendors as well.


Discussion:
Thanks for the report @rpammeraal! Please feel free to submit a pull request for this change -- it's a very good one :)

We've recently added a [CLA](https://docs.getdbt.com/docs/contributor-license-agreements) to the dbt contribution process. This CLA ensures that contributions to the dbt codebase are "original work that does not violate any third-party license agreement." As such, we're not well-suited to copy/paste code into patches to the dbt-core repository. If you're able to submit a PR for this, I'd be very happy to merge it! If not, we can retitle this issue to "Make Postgres database comparisons case-insensitive" and prioritize it on our own accord. Let us know!
I looked at requirements on how to create a PR -- it is quite a bit of work.
Also, some other changes need to be made in the underlying postgres modules
to ensure that database (and other entity) names are always compared in
lower case in postgres  -- however, I don't have the bandwidth to set up,
debug and test all this.

Thanks,
Roy


On Mon, Oct 7, 2019 at 6:22 PM Drew Banin <notifications@github.com> wrote:

> Thanks for the repot @rpammeraal <https://github.com/rpammeraal>! Please
> feel free to submit a pull request for this change -- it's a very good one
> :)
>
> We've recently added a CLA
> <https://docs.getdbt.com/docs/contributor-license-agreements> to the dbt
> contribution process. This CLA ensures that contributions to the dbt
> codebase are "original work that does not violate any third-party license
> agreement." As such, we're not well-suited to copy/paste code into patches
> to the dbt-core repository. If you're able to submit a PR for this, I'd be
> very happy to merge it! If not, we can retitle this issue to "Make Postgres
> database comparisons case-insensitive" and prioritize it on our own accord.
> Let us know!
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/fishtown-analytics/dbt/issues/1800?email_source=notifications&email_token=AGE35HHAEGLTVB7OOZUNI4DQNPOEFA5CNFSM4I43HUT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEASKEWQ#issuecomment-539271770>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AGE35HAZ2MSEY2UWCYOVHITQNPOEFANCNFSM4I43HUTQ>
> .
>

I don't believe the fix in #1918 fully addresses this issue. While the `cross-db` error message has indeed been addressed, dbt now fails with an error like:
```
Compilation Error in model debug2 (models/debug2.sql)
  When searching for a relation, dbt found an approximate match. Instead of guessing
  which relation to use, dbt will move on. Please delete "mydb"."test_schema"."debug2", or rename it to be less ambiguous.
  Searched for: "MyDB"."test_schema"."debug2"
  Found: "mydb"."test_schema"."debug2"
```

Here, my database is named `MyDB` on Postgres. We can reproduce this failure mode with:

```
-- models/my_model.sql
select 1 as id
```

And a profile configured with:
```
debug:
  target: dev
  outputs:
    dev:
      type: postgres
      host: localhost
      user: drew
      pass: password
      port: 5433
      dbname: MyDB
      schema: test_schema
      threads: 8
```

Followed by:
```
dbt run
dbt run
```

The first `dbt run` will succeed while the second `dbt run` will fail.

Resolving patch:
diff --git a/core/dbt/adapters/base/impl.py b/core/dbt/adapters/base/impl.py
--- a/core/dbt/adapters/base/impl.py
+++ b/core/dbt/adapters/base/impl.py
@@ -138,17 +138,18 @@ def schemas_searched(self):
     def flatten(self):
         new = self.__class__()
 
-        database = None
-        # iterate once to look for a database name
+        # make sure we don't have duplicates
         seen = {r.database.lower() for r in self if r.database}
         if len(seen) > 1:
             dbt.exceptions.raise_compiler_error(str(seen))
-        elif len(seen) == 1:
-            database = list(seen)[0]
 
         for information_schema_name, schema in self.search():
+            path = {
+                'database': information_schema_name.database,
+                'schema': schema
+            }
             new.add(information_schema_name.incorporate(
-                path={'database': database, 'schema': schema},
+                path=path,
                 quote_policy={'database': False},
                 include_policy={'database': False},
             ))


Test patch:
diff --git a/test/integration/001_simple_copy_test/shouting_models/ADVANCED_INCREMENTAL.sql b/test/integration/001_simple_copy_test/models-shouting/ADVANCED_INCREMENTAL.sql
similarity index 100%
rename from test/integration/001_simple_copy_test/shouting_models/ADVANCED_INCREMENTAL.sql
rename to test/integration/001_simple_copy_test/models-shouting/ADVANCED_INCREMENTAL.sql
diff --git a/test/integration/001_simple_copy_test/shouting_models/COMPOUND_SORT.sql b/test/integration/001_simple_copy_test/models-shouting/COMPOUND_SORT.sql
similarity index 100%
rename from test/integration/001_simple_copy_test/shouting_models/COMPOUND_SORT.sql
rename to test/integration/001_simple_copy_test/models-shouting/COMPOUND_SORT.sql
diff --git a/test/integration/001_simple_copy_test/shouting_models/DISABLED.sql b/test/integration/001_simple_copy_test/models-shouting/DISABLED.sql
similarity index 100%
rename from test/integration/001_simple_copy_test/shouting_models/DISABLED.sql
rename to test/integration/001_simple_copy_test/models-shouting/DISABLED.sql
diff --git a/test/integration/001_simple_copy_test/shouting_models/EMPTY.sql b/test/integration/001_simple_copy_test/models-shouting/EMPTY.sql
similarity index 100%
rename from test/integration/001_simple_copy_test/shouting_models/EMPTY.sql
rename to test/integration/001_simple_copy_test/models-shouting/EMPTY.sql
diff --git a/test/integration/001_simple_copy_test/shouting_models/GET_AND_REF.sql b/test/integration/001_simple_copy_test/models-shouting/GET_AND_REF.sql
similarity index 100%
rename from test/integration/001_simple_copy_test/shouting_models/GET_AND_REF.sql
rename to test/integration/001_simple_copy_test/models-shouting/GET_AND_REF.sql
diff --git a/test/integration/001_simple_copy_test/shouting_models/INCREMENTAL.sql b/test/integration/001_simple_copy_test/models-shouting/INCREMENTAL.sql
similarity index 100%
rename from test/integration/001_simple_copy_test/shouting_models/INCREMENTAL.sql
rename to test/integration/001_simple_copy_test/models-shouting/INCREMENTAL.sql
diff --git a/test/integration/001_simple_copy_test/shouting_models/INTERLEAVED_SORT.sql b/test/integration/001_simple_copy_test/models-shouting/INTERLEAVED_SORT.sql
similarity index 100%
rename from test/integration/001_simple_copy_test/shouting_models/INTERLEAVED_SORT.sql
rename to test/integration/001_simple_copy_test/models-shouting/INTERLEAVED_SORT.sql
diff --git a/test/integration/001_simple_copy_test/shouting_models/MATERIALIZED.sql b/test/integration/001_simple_copy_test/models-shouting/MATERIALIZED.sql
similarity index 100%
rename from test/integration/001_simple_copy_test/shouting_models/MATERIALIZED.sql
rename to test/integration/001_simple_copy_test/models-shouting/MATERIALIZED.sql
diff --git a/test/integration/001_simple_copy_test/shouting_models/SCHEMA.yml b/test/integration/001_simple_copy_test/models-shouting/SCHEMA.yml
similarity index 100%
rename from test/integration/001_simple_copy_test/shouting_models/SCHEMA.yml
rename to test/integration/001_simple_copy_test/models-shouting/SCHEMA.yml
diff --git a/test/integration/001_simple_copy_test/shouting_models/VIEW_MODEL.sql b/test/integration/001_simple_copy_test/models-shouting/VIEW_MODEL.sql
similarity index 100%
rename from test/integration/001_simple_copy_test/shouting_models/VIEW_MODEL.sql
rename to test/integration/001_simple_copy_test/models-shouting/VIEW_MODEL.sql
diff --git a/test/integration/001_simple_copy_test/models-trivial/model.sql b/test/integration/001_simple_copy_test/models-trivial/model.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/001_simple_copy_test/models-trivial/model.sql
@@ -0,0 +1 @@
+select 1 as id
diff --git a/test/integration/001_simple_copy_test/test_simple_copy.py b/test/integration/001_simple_copy_test/test_simple_copy.py
--- a/test/integration/001_simple_copy_test/test_simple_copy.py
+++ b/test/integration/001_simple_copy_test/test_simple_copy.py
@@ -1,3 +1,6 @@
+import os
+from pytest import mark
+
 from test.integration.base import DBTIntegrationTest, use_profile
 
 
@@ -329,7 +332,7 @@ def test__snowflake__incremental_overwrite(self):
 class TestShouting(BaseTestSimpleCopy):
     @property
     def models(self):
-        return self.dir('shouting_models')
+        return self.dir('models-shouting')
 
     @property
     def project_config(self):
@@ -351,3 +354,42 @@ def test__postgres__simple_copy_loud(self):
         self.assertEqual(len(results),  7)
 
         self.assertManyTablesEqual(["seed", "VIEW_MODEL", "INCREMENTAL", "MATERIALIZED", "GET_AND_REF"])
+
+
+# I give up on getting this working for Windows.
+@mark.skipif(os.name == 'nt', reason='mixed-case postgres database tests are not supported on Windows')
+class TestMixedCaseDatabase(BaseTestSimpleCopy):
+    @property
+    def models(self):
+        return self.dir('models-trivial')
+
+    def postgres_profile(self):
+        return {
+            'config': {
+                'send_anonymous_usage_stats': False
+            },
+            'test': {
+                'outputs': {
+                    'default2': {
+                        'type': 'postgres',
+                        'threads': 4,
+                        'host': self.database_host,
+                        'port': 5432,
+                        'user': 'root',
+                        'pass': 'password',
+                        'dbname': 'dbtMixedCase',
+                        'schema': self.unique_schema()
+                    },
+                },
+                'target': 'default2'
+            }
+        }
+
+    @property
+    def project_config(self):
+        return {}
+
+    @use_profile('postgres')
+    def test_postgres_run_mixed_case(self):
+        self.run_dbt()
+        self.run_dbt()
diff --git a/test/setup_db.sh b/test/setup_db.sh
--- a/test/setup_db.sh
+++ b/test/setup_db.sh
@@ -40,4 +40,7 @@ psql -c "CREATE ROLE noaccess WITH PASSWORD 'password' NOSUPERUSER;"
 psql -c "ALTER ROLE noaccess WITH LOGIN;"
 psql -c "GRANT CONNECT ON DATABASE dbt TO noaccess;"
 
+psql -c 'CREATE DATABASE "dbtMixedCase";'
+psql -c 'GRANT CREATE, CONNECT ON DATABASE "dbtMixedCase" TO root WITH GRANT OPTION;'
+
 set +x