Repository: dbt-labs/dbt-core

Issue:
dbt is creating schemas when they already exists because of case-sensitive comparisons
## Issue
In Snowflake, DBT is trying to create a schema that already exists.   Because my snowflake role does not have create schema privs, I get an `Insufficient privileges to operate on database 'foo'` error when dbt tries to create the schema.


### Issue description

Stacktrace:
```
2019-08-01 18:05:01,735 (MainThread): On master: select distinct schema_name
    from CENTRAL_DATA.information_schema.schemata
    where catalog_name ilike 'CENTRAL_DATA'
2019-08-01 18:05:04,211 (MainThread): SQL status: SUCCESS 9 in 2.48 seconds
2019-08-01 18:05:04,223 (MainThread): Creating schema "CENTRAL_DATA"."cdm".
2019-08-01 18:05:04,227 (MainThread): Using snowflake connection "master".
2019-08-01 18:05:04,227 (MainThread): On master: BEGIN
2019-08-01 18:05:04,417 (MainThread): SQL status: SUCCESS 1 in 0.19 seconds
2019-08-01 18:05:04,417 (MainThread): Using snowflake connection "master".
2019-08-01 18:05:04,417 (MainThread): On master: create schema if not exists CENTRAL_DATA.cdm
2019-08-01 18:05:04,727 (MainThread): Snowflake error: 003001 (42501): 018dec4d-0307-9857-0000-0379146884aa: SQL access control error:
Insufficient privileges to operate on database 'CENTRAL_DATA'
2019-08-01 18:05:04,728 (MainThread): On master: ROLLBACK
2019-08-01 18:05:05,336 (MainThread): Connection 'master' was left open.
2019-08-01 18:05:05,337 (MainThread): On master: Close
2019-08-01 18:05:05,953 (MainThread): Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1080a42e8>, <snowplow_tracker.self_describi
ng_json.SelfDescribingJson object at 0x106d629b0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x106d62748>]}
2019-08-01 18:05:06,031 (MainThread): Flushing usage events
2019-08-01 18:05:06,031 (MainThread): /Users/edmund.yan/.pyenv/versions/3.6.2/envs/central-data/lib/python3.6/site-packages/dbt/main.py:97: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

2019-08-01 18:05:06,031 (MainThread): Encountered an error:
2019-08-01 18:05:06,031 (MainThread): /Users/edmund.yan/.pyenv/versions/3.6.2/envs/central-data/lib/python3.6/site-packages/dbt/main.py:98: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

2019-08-01 18:05:06,031 (MainThread): Database Error
  003001 (42501): 018dec4d-0307-9857-0000-0379146884aa: SQL access control error:
  Insufficient privileges to operate on database 'CENTRAL_DATA'
2019-08-01 18:05:06,050 (MainThread): Traceback (most recent call last):
  File "/Users/edmund.yan/.pyenv/versions/3.6.2/envs/central-data/lib/python3.6/site-packages/dbt/adapters/snowflake/connections.py", line 77, in exception_handler
    yield
  File "/Users/edmund.yan/.pyenv/versions/3.6.2/envs/central-data/lib/python3.6/site-packages/dbt/adapters/sql/connections.py", line 60, in add_query
    cursor.execute(sql, bindings)
  File "/Users/edmund.yan/.pyenv/versions/3.6.2/envs/central-data/lib/python3.6/site-packages/snowflake/connector/cursor.py", line 558, in execute
    errvalue)
  File "/Users/edmund.yan/.pyenv/versions/3.6.2/envs/central-data/lib/python3.6/site-packages/snowflake/connector/errors.py", line 97, in errorhandler_wrapper
    cursor.errorhandler(connection, cursor, errorclass, errorvalue)
  File "/Users/edmund.yan/.pyenv/versions/3.6.2/envs/central-data/lib/python3.6/site-packages/snowflake/connector/errors.py", line 73, in default_errorhandler
    done_format_msg=errorvalue.get(u'done_format_msg'))
snowflake.connector.errors.ProgrammingError: 003001 (42501): 018dec4d-0307-9857-0000-0379146884aa: SQL access control error:
Insufficient privileges to operate on database 'CENTRAL_DATA'
```


### Results

https://github.com/fishtown-analytics/dbt/blob/399b33822a85493a8790df787eed59e951aecf6c/core/dbt/task/runnable.py#L306-L324

I believe there is a bug in `create_schemas()` and case sensitivity.  In Snowflake, `list_schemas()` will return schema names in upper-case.   However, if you define your schemas in your dbt project or profiles in lowercase, the set subtraction of `required_schemas - existing_schemas` will not identify that the schema already exists and will re-create it.

When I change my dbt_project.yml and profiles.yml to use only upper-case schema names, it works as expected.


### System information
The output of `dbt --version`:
```
0.14.0
```

The python version you're using (probably the output of `python --version`)
```
Python 3.6.2
```

### Proposed Solutions
1. dbt does not support case-sensitive schemas
This looks like the easiest route.  If you look at https://github.com/fishtown-analytics/dbt/issues/1411, we already consider databases to be case insensitive.

2. dbt supports case-sensitive schemas
If we go this route, I think the default should be case insensitive.  However, if you double quote your schema name, it will evaluate as case-sensitive.  This seems like a better solution, but I'm unsure of all the downstream call points that would need to be updated to support this.


Happy to help submit a patch if you give me some direction on what the correct fix should be :)
custom schema feature unintentionally creates schema stub 
## Issue

### Issue description
The custom schema feature unintentionally creates schema stub. 
For example, when we specify "email_platform_out" and "email_platform_in" with  custom schema feature, dbt will create "email_platform" as a dummy schema, which is not expected.

### Results

For example, I want to use custom schema to specify two schemas, "email_platform_out" and "email_platform_in".  

In profiles.yml, I set the `scheme=email_platform`
In dbt_project.yml, I set all models in these two schemas
```
models:
    email_dw_data_project:
        email_platform_out:
            schema: out

        email_platform_in:
            schema: in 
```
when I ran `dbt run`, receive the following error msg because dbt tried to create schema “email_platform” but failed, which is not expected, because I don't have nor want to create "email_platorm" schema.

### System information
The output of `dbt --version`:
```
2018-08-03 18:45:19,017 (MainThread): Parsing materializations/incremental/incremental.sql
2018-08-03 18:45:19,037 (MainThread): Parsing materializations/seed/bigquery.sql
2018-08-03 18:45:19,042 (MainThread): Parsing materializations/seed/seed.sql
2018-08-03 18:45:19,072 (MainThread): Parsing materializations/table/bigquery_table.sql
2018-08-03 18:45:19,093 (MainThread): Parsing materializations/table/table.sql
2018-08-03 18:45:19,111 (MainThread): Parsing materializations/view/bigquery_view.sql
2018-08-03 18:45:19,119 (MainThread): Parsing materializations/view/view.sql
2018-08-03 18:45:19,136 (MainThread): Parsing operations/catalog/get_catalog.sql
2018-08-03 18:45:19,138 (MainThread): Parsing schema_tests/accepted_values.sql
2018-08-03 18:45:19,140 (MainThread): Parsing schema_tests/not_null.sql
2018-08-03 18:45:19,143 (MainThread): Parsing schema_tests/relationships.sql
2018-08-03 18:45:19,145 (MainThread): Parsing schema_tests/unique.sql
2018-08-03 18:45:19,149 (MainThread): Parsing model.o2e_email_dw_data_project.account_core_base
2018-08-03 18:45:19,152 (MainThread): Parsing model.o2e_email_dw_data_project.account_emails_base
2018-08-03 18:45:19,156 (MainThread): Parsing model.o2e_email_dw_data_project.account_opps_base
2018-08-03 18:45:19,162 (MainThread): Parsing model.o2e_email_dw_data_project.mcr_frame_base
2018-08-03 18:45:19,165 (MainThread): Parsing model.o2e_email_dw_data_project.franchise_partner_daily_out
2018-08-03 18:45:19,168 (MainThread): Parsing model.o2e_email_dw_data_project.master_account_with_emails_daily_out
2018-08-03 18:45:19,172 (MainThread): Parsing model.o2e_email_dw_data_project.sfdc_opt_out_email_daily_out
2018-08-03 18:45:19,179 (MainThread): * WARNING: No constraints found for model 'daily_out' in file models/email_platform_out/daily_out/schema.yml

2018-08-03 18:45:19,186 (MainThread): Found 7 models, 0 tests, 0 archives, 0 analyses, 76 macros, 1 operations, 0 seed files
2018-08-03 18:45:19,188 (MainThread):
2018-08-03 18:45:19,192 (MainThread): Acquiring new redshift connection "master".
2018-08-03 18:45:19,192 (MainThread): Opening a new connection (0 currently allocated)
2018-08-03 18:45:19,192 (MainThread): Connecting to Redshift using 'database' credentials
2018-08-03 18:45:19,693 (MainThread): Using redshift connection "master".
2018-08-03 18:45:19,693 (MainThread): On master: select distinct nspname from pg_namespace
2018-08-03 18:45:19,834 (MainThread): SQL status: SELECT in 0.14 seconds
2018-08-03 18:45:19,835 (MainThread): Creating schema "email_platform".
2018-08-03 18:45:19,836 (MainThread): Using redshift connection "master".
2018-08-03 18:45:19,836 (MainThread): On master: BEGIN
2018-08-03 18:45:19,911 (MainThread): SQL status: BEGIN in 0.07 seconds
2018-08-03 18:45:19,911 (MainThread): Using redshift connection "master".
2018-08-03 18:45:19,912 (MainThread): On master: create schema if not exists "email_platform"
2018-08-03 18:45:19,984 (MainThread): Postgres error: permission denied for database o2enexus

2018-08-03 18:45:19,984 (MainThread): On master: ROLLBACK
2018-08-03 18:45:20,059 (MainThread): Flushing usage events
2018-08-03 18:45:20,059 (MainThread): Encountered an error:
2018-08-03 18:45:20,060 (MainThread): Database Error
  permission denied for database o2enexus
2018-08-03 18:45:20,068 (MainThread): Traceback (most recent call last):
  File "/usr/local/Cellar/dbt/0.10.2/libexec/lib/python3.7/site-packages/dbt/adapters/postgres/impl.py", line 25, in exception_handler
    yield
  File "/usr/local/Cellar/dbt/0.10.2/libexec/lib/python3.7/site-packages/dbt/adapters/default/impl.py", line 648, in add_query
    cursor.execute(sql, bindings)
```


### Steps to reproduce








Discussion:
Thanks for the report @edmundyan! Casing + quoting on Snowflake can be a bit of a nightmare!

I think it makes sense to make this schema lookup case-insensitive. One good approach could be to just upper-case (or lower-case) the schema names in each set (`required_schemas` and `existing_schemas`), then compute the set-difference.

dbt in general will do very undesirable things if users try to have two distinct schema names that are distinguished only in their casing. I think it's ok to continue to assume that users won't (or can't) do that here.

We'd love a PR for this one! Let me know if there's anything I can do to help out. Check out the [contributing guide](https://github.com/fishtown-analytics/dbt/blob/dev/0.14.1/CONTRIBUTING.md) for some info on how to get set up for development with dbt.
Thanks for the report @hui-zheng. As a workaround, I'd recommend either giving `create` permissions to the dbt user, or manually creating the schema specified in the profile.
It'd be nice to fix this behavior rather than asking us to create unwanted schemas. There are no tables in there; why should we create such a schema?

Edit:  I posted this same issue in Slack.  Here's my original message:

I have a custom `generate_schema_name` which pulls the schema name from the folder that the model is defined in. This is working great so far, except that the schema defined on the profile (required field, but completely unused in my project) is being re-created every time I call `dbt run`.

Workaround is to set it to the name of one of my subfolders, but it's odd to me that some internal DBT machinery thinks it should go create schemas for things that will never have tables in them. Any reason for this behavior?
I think we discussed this on Slack, but: the reason for this stub schema is because Snowflake requires a "default" schema when performing certain operations (like creating temporary tables). 

I think to fix this, we'll need to pass a schema name into the snowflake version of the `create_table_as` macro [here](https://github.com/fishtown-analytics/dbt/blob/dev/guion-bluford/dbt/include/global_project/macros/adapters/snowflake.sql#L3). Alternatively, we could write some code to only create the stub schema for Snowflake connections, but I think that implementation would be undesirable.

Resolving patch:
diff --git a/core/dbt/task/runnable.py b/core/dbt/task/runnable.py
--- a/core/dbt/task/runnable.py
+++ b/core/dbt/task/runnable.py
@@ -305,23 +305,16 @@ def get_model_schemas(self, selected_uids):
 
     def create_schemas(self, adapter, selected_uids):
         required_schemas = self.get_model_schemas(selected_uids)
-
-        # Snowflake needs to issue a "use {schema}" query, where schema
-        # is the one defined in the profile. Create this schema if it
-        # does not exist, otherwise subsequent queries will fail. Generally,
-        # dbt expects that this schema will exist anyway.
-        required_schemas.add(
-            (self.config.credentials.database, self.config.credentials.schema)
-        )
-
         required_databases = set(db for db, _ in required_schemas)
 
-        existing_schemas = set()
+        existing_schemas_lowered = set()
         for db in required_databases:
-            existing_schemas.update((db, s) for s in adapter.list_schemas(db))
+            existing_schemas_lowered.update(
+                (db.lower(), s.lower()) for s in adapter.list_schemas(db))
 
-        for database, schema in (required_schemas - existing_schemas):
-            adapter.create_schema(database, schema)
+        for db, schema in required_schemas:
+            if (db.lower(), schema.lower()) not in existing_schemas_lowered:
+                adapter.create_schema(db, schema)
 
     def get_result(self, results, elapsed_time, generated_at):
         return ExecutionResult(


Test patch: