Repository: dbt-labs/dbt-core

Issue:
EOL SQL comments remove entire line from execution
### Describe the bug
When using SQL line comment (`-- comment`) at the end of lines **and** running multiple SQL queries (separated by semicolons) within one [statement block](https://docs.getdbt.com/reference/dbt-jinja-functions/statement-blocks/), if any **one-line** query has a line comment after it (on the same line), that command won't be executed (i.e. not submitted to the database at all) - except if it's the first query.
 
- This affects all statement blocks, whether they come from a model, macro, materialization, etc.
- Block comments (`/* comment */`) do not have this effect (which provides a workaround)

This will work (all three queries will be executed):
```
{%- call statement('test') -%}
  select 1;
  select 2;
  select 3;
{%- endcall %}
```

This will work (all three queries will be executed):
```
{%- call statement('test') -%}
  select 1; /* comment */
  select 2; /* comment */
  select 3; /* comment */
{%- endcall %}
```

**This will not work** (only the first query will be executed):
```
{%- call statement('test') -%}
  select 1; -- comment
  select 2; -- comment
  select 3; -- comment
{%- endcall %}
```

This will work (all three queries will be executed):
```
{%- call statement('test') -%}
  select
    1; -- comment
  select
    2; -- comment
  select
    3; -- comment
{%- endcall %}
```

### Steps To Reproduce
Just put the examples above in a model or macro, and take a look at the queries executed on your database (query log, etc.)

### Expected behavior
All three queries should be executed in all examples.

### System information
**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [ ] bigquery
- [x] snowflake
- [ ] other (specify: ____________)


**The output of `dbt --version`:**
```
installed version: 0.18.0-rc1
   latest version: 0.17.2

Your version of dbt is ahead of the latest release!

Plugins:
  - snowflake: 0.18.0rc1
```

**The operating system you're using:**
```
macOS Catalina
Version 10.15.6 (19G2021)
```

**The output of `python --version`:**
```
Python 3.8.3
```
EOL SQL comments remove entire line from execution
### Describe the bug
When using SQL line comment (`-- comment`) at the end of lines **and** running multiple SQL queries (separated by semicolons) within one [statement block](https://docs.getdbt.com/reference/dbt-jinja-functions/statement-blocks/), if any **one-line** query has a line comment after it (on the same line), that command won't be executed (i.e. not submitted to the database at all) - except if it's the first query.
 
- This affects all statement blocks, whether they come from a model, macro, materialization, etc.
- Block comments (`/* comment */`) do not have this effect (which provides a workaround)

This will work (all three queries will be executed):
```
{%- call statement('test') -%}
  select 1;
  select 2;
  select 3;
{%- endcall %}
```

This will work (all three queries will be executed):
```
{%- call statement('test') -%}
  select 1; /* comment */
  select 2; /* comment */
  select 3; /* comment */
{%- endcall %}
```

**This will not work** (only the first query will be executed):
```
{%- call statement('test') -%}
  select 1; -- comment
  select 2; -- comment
  select 3; -- comment
{%- endcall %}
```

This will work (all three queries will be executed):
```
{%- call statement('test') -%}
  select
    1; -- comment
  select
    2; -- comment
  select
    3; -- comment
{%- endcall %}
```

### Steps To Reproduce
Just put the examples above in a model or macro, and take a look at the queries executed on your database (query log, etc.)

### Expected behavior
All three queries should be executed in all examples.

### System information
**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [ ] bigquery
- [x] snowflake
- [ ] other (specify: ____________)


**The output of `dbt --version`:**
```
installed version: 0.18.0-rc1
   latest version: 0.17.2

Your version of dbt is ahead of the latest release!

Plugins:
  - snowflake: 0.18.0rc1
```

**The operating system you're using:**
```
macOS Catalina
Version 10.15.6 (19G2021)
```

**The output of `python --version`:**
```
Python 3.8.3
```


Discussion:
Thanks for the detailed walkthrough, @mroy-seedbox.

I'm guessing that the offending code is our regex hack here:
https://github.com/fishtown-analytics/dbt/blob/2cc2d971c69ecf4143deb16f961f2c9ed8f22199/plugins/snowflake/dbt/adapters/snowflake/connections.py#L298-L304

This reflects the requirement of snowflake-connector-python to split a stream of semicolon-delimited statements into separate, individual queries. We rely on [their `split_statements` function](https://github.com/snowflakedb/snowflake-connector-python/blob/master/src/snowflake/connector/util_text.py#L30) to do this, too.

As you note, there are ways to work around this, so it's not a high-priority fix from our end. If you're interested in contributing the fix, I'd be happy to help!
Thanks for the detailed walkthrough, @mroy-seedbox.

I'm guessing that the offending code is our regex hack here:
https://github.com/fishtown-analytics/dbt/blob/2cc2d971c69ecf4143deb16f961f2c9ed8f22199/plugins/snowflake/dbt/adapters/snowflake/connections.py#L298-L304

This reflects the requirement of snowflake-connector-python to split a stream of semicolon-delimited statements into separate, individual queries. We rely on [their `split_statements` function](https://github.com/snowflakedb/snowflake-connector-python/blob/master/src/snowflake/connector/util_text.py#L30) to do this, too.

As you note, there are ways to work around this, so it's not a high-priority fix from our end. If you're interested in contributing the fix, I'd be happy to help!

Resolving patch:
diff --git a/plugins/snowflake/dbt/adapters/snowflake/connections.py b/plugins/snowflake/dbt/adapters/snowflake/connections.py
--- a/plugins/snowflake/dbt/adapters/snowflake/connections.py
+++ b/plugins/snowflake/dbt/adapters/snowflake/connections.py
@@ -305,7 +305,9 @@ def add_query(self, sql, auto_begin=True,
             # empty queries. this avoids using exceptions as flow control,
             # and also allows us to return the status of the last cursor
             without_comments = re.sub(
-                re.compile('^.*(--.*)$', re.MULTILINE),
+                re.compile(
+                    r'(\".*?\"|\'.*?\')|(/\*.*?\*/|--[^\r\n]*$)', re.MULTILINE
+                ),
                 '', individual_query).strip()
 
             if without_comments == "":


Test patch:
diff --git a/test/unit/test_snowflake_adapter.py b/test/unit/test_snowflake_adapter.py
--- a/test/unit/test_snowflake_adapter.py
+++ b/test/unit/test_snowflake_adapter.py
@@ -1,4 +1,5 @@
 import agate
+import re
 import unittest
 from contextlib import contextmanager
 from unittest import mock
@@ -523,3 +524,48 @@ def test_float_from_description(self):
         assert col.is_numeric() is False
         assert col.is_string() is False
         assert col.is_integer() is False
+
+
+class SnowflakeConnectionsTest(unittest.TestCase):
+    def test_comment_stripping_regex(self):
+        pattern = r'(\".*?\"|\'.*?\')|(/\*.*?\*/|--[^\r\n]*$)'
+        comment1 = '-- just comment'
+        comment2 = '/* just comment */'
+        query1 = 'select 1; -- comment'
+        query2 = 'select 1; /* comment */'
+        query3 = 'select 1; -- comment\nselect 2; /* comment */ '
+        query4 = 'select \n1; -- comment\nselect \n2; /* comment */ '
+        query5 = 'select 1; -- comment \nselect 2; -- comment \nselect 3; -- comment'
+
+        stripped_comment1 = re.sub(re.compile(pattern, re.MULTILINE),
+                                   '', comment1).strip()
+
+        stripped_comment2 = re.sub(re.compile(pattern, re.MULTILINE),
+                                   '', comment2).strip()
+
+        stripped_query1 = re.sub(re.compile(pattern, re.MULTILINE),
+                                 '', query1).strip()
+
+        stripped_query2 = re.sub(re.compile(pattern, re.MULTILINE),
+                                 '', query2).strip()
+
+        stripped_query3 = re.sub(re.compile(pattern, re.MULTILINE),
+                                 '', query3).strip()
+
+        stripped_query4 = re.sub(re.compile(pattern, re.MULTILINE),
+                                 '', query4).strip()
+
+        stripped_query5 = re.sub(re.compile(pattern, re.MULTILINE),
+                                 '', query5).strip()
+
+        expected_query_3 = 'select 1; \nselect 2;'
+        expected_query_4 = 'select \n1; \nselect \n2;'
+        expected_query_5 = 'select 1; \nselect 2; \nselect 3;'
+
+        self.assertEqual('', stripped_comment1)
+        self.assertEqual('', stripped_comment2)
+        self.assertEqual('select 1;', stripped_query1)
+        self.assertEqual('select 1;', stripped_query2)
+        self.assertEqual(expected_query_3, stripped_query3)
+        self.assertEqual(expected_query_4, stripped_query4)
+        self.assertEqual(expected_query_5, stripped_query5)