internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dbt-labs/dbt-core

Issue:
Support jinja expression statements
## Feature

### Feature description
http://jinja.pocoo.org/docs/2.10/extensions/#expression-statement

Presently, we hack expressions with:
```
{% set _ = my_dict.update({"a": 1, "b": 2}) %}
```

Instead, the jinja expression statement will make it possible to write:

```
{% do my_dict.update({"a": 1, "b": 2}) %}
```

This is a minor difference, but it will make jinja sql more readable and idiomatic.

### Who will this benefit?
jinja writers


Discussion:

Resolving patch:
diff --git a/dbt/clients/jinja.py b/dbt/clients/jinja.py
--- a/dbt/clients/jinja.py
+++ b/dbt/clients/jinja.py
@@ -246,7 +246,7 @@ def __call__(self, *args, **kwargs):
 
 def get_environment(node=None, capture_macros=False):
     args = {
-        'extensions': []
+        'extensions': ['jinja2.ext.do']
     }
 
     if capture_macros:


Test patch:
diff --git a/test/unit/test_jinja.py b/test/unit/test_jinja.py
new file mode 100644
--- /dev/null
+++ b/test/unit/test_jinja.py
@@ -0,0 +1,11 @@
+import unittest
+
+from dbt.clients.jinja import get_template
+
+class TestJinja(unittest.TestCase):
+    def test_do(self):
+        s = '{% set my_dict = {} %}\n{% do my_dict.update(a=1) %}'
+
+        template = get_template(s, {})
+        mod = template.make_module()
+        self.assertEqual(mod.my_dict, {'a': 1})
Repository: dbt-labs/dbt-core

Issue:
Create -m  flag for dbt run/test as an alias for dbt run --models
### Feature
Create `-m ` flag for `dbt run/test` as an alias for `dbt run --models`

### Feature Description
While `dbt run --m` will currently provide the same results, if another flag is created that begins with `--m` then the short form becomes ambiguous.

### Who will this benefit?
I have typed `dbt run --models` so many times that I found myself asking "why is `--models` not the default?" The entire dbt community will benefit from the newly created `-m` flag



Discussion:

Resolving patch:
diff --git a/dbt/main.py b/dbt/main.py
--- a/dbt/main.py
+++ b/dbt/main.py
@@ -477,6 +477,7 @@ def parse_args(args):
 
     for sub in [run_sub, compile_sub, generate_sub]:
         sub.add_argument(
+            '-m',
             '--models',
             required=False,
             nargs='+',


Test patch:
Repository: dbt-labs/dbt-core

Issue:
JSON Decoding error with Python2 + Anaconda: Decimal('0') is not JSON serializable
## Issue

### Issue description
The `dbt test` command can fail with an error like:
```
Decimal('0') is not JSON serializable
```

dbt explicitly implements a Decimal-aware JSON serializer, but it appears that there is more than one type of Decimal at play here. The `agate` library will optionally include the `cdecimal` version of Decimal if it is available (code [here](https://github.com/wireservice/agate/blob/master/agate/data_types/number.py#L5)).

dbt does a check [here](https://github.com/fishtown-analytics/dbt/blob/5be8c7f85f3bea2527ddff17e9a51f74b2d847a8/dbt/utils.py#L473) that looks like:
```
        if isinstance(obj, Decimal):
            return float(obj)
        return super(JSONEncoder, self).default(obj)
```

here, `Decimal` is `decimal.Decimal`, but it could be that `obj` is of type `cdecimal.Decimal`. This leads to a confusing error.

Instead, dbt should check if the `obj` is of type `decimal.Decimal` or `cdecimal.Decimal`.

I _think_ the Anaconda part of this is crucial, as I'm unsure how else someone might have `cdecimal` installed in their system. You can install it manually with `pip install m3-cdecimal`.

### Results
```
"/Users/xxx/anaconda2/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/xxx/anaconda2/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/Users/xxx/anaconda2/lib/python2.7/site-packages/dbt/utils.py", line 481, in default
    return super(JSONEncoder, self).default(obj)
  File "/Users/xxx/anaconda2/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: Decimal('0') is not JSON serializable
```

### System information
The output of `dbt --version`:
```
0.12.1
```

The operating system you're running on: OSX

The python version you're using (probably the output of `python --version`)
2.7.15 installed via Anaconda.


Discussion:

Resolving patch:
diff --git a/dbt/compat.py b/dbt/compat.py
--- a/dbt/compat.py
+++ b/dbt/compat.py
@@ -1,6 +1,14 @@
 import codecs
 import json
 import warnings
+import decimal
+
+try:
+    import cdecimal
+except ImportError:
+    DECIMALS = (decimal.Decimal,)
+else:
+    DECIMALS = (decimal.Decimal, cdecimal.Decimal)
 
 WHICH_PYTHON = None
 
diff --git a/dbt/utils.py b/dbt/utils.py
--- a/dbt/utils.py
+++ b/dbt/utils.py
@@ -14,7 +14,7 @@
 import dbt.flags
 
 from dbt.include import GLOBAL_DBT_MODULES_PATH
-from dbt.compat import basestring
+from dbt.compat import basestring, DECIMALS
 from dbt.logger import GLOBAL_LOGGER as logger
 from dbt.node_types import NodeType
 from dbt.clients import yaml_helper
@@ -476,6 +476,6 @@ class JSONEncoder(json.JSONEncoder):
     converted to floats.
     """
     def default(self, obj):
-        if isinstance(obj, Decimal):
+        if isinstance(obj, DECIMALS):
             return float(obj)
         return super(JSONEncoder, self).default(obj)


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Postgres character columns are not treated as string types
## Issue

### Issue description
dbt does not treat character types as string types, and therefore leaves out the length portion of the `character` column type when building columns. This happens during Archival (and likely incremental models when expanding types) on Postgres.

### Results
dbt should treat `character` as a string type, as with `varchar` and `text`.

### System information
The output of `dbt --version`:
```
0.12.x
```

### Steps to reproduce
Archive a table on Postgres with a column of type `character(36)`. dbt will recreate this column with type `character` with no size specified.


This issue was created from the description of https://github.com/fishtown-analytics/dbt/pull/1191


Discussion:

Resolving patch:
diff --git a/dbt/schema.py b/dbt/schema.py
--- a/dbt/schema.py
+++ b/dbt/schema.py
@@ -45,7 +45,7 @@ def data_type(self):
             return self.dtype
 
     def is_string(self):
-        return self.dtype.lower() in ['text', 'character varying']
+        return self.dtype.lower() in ['text', 'character varying', 'character']
 
     def is_numeric(self):
         return self.dtype.lower() in ['numeric', 'number']


Test patch:
diff --git a/test/unit/test_schema.py b/test/unit/test_schema.py
--- a/test/unit/test_schema.py
+++ b/test/unit/test_schema.py
@@ -2,6 +2,17 @@
 
 import dbt.schema
 
+class TestStringType(unittest.TestCase):
+
+    def test__character_type(self):
+        col = dbt.schema.Column(
+            'fieldname',
+            'character',
+            char_size=10
+        )
+
+        self.assertEqual(col.data_type, 'character varying(10)')
+
 
 class TestNumericType(unittest.TestCase):
Repository: dbt-labs/dbt-core

Issue:
Loosen restrictions of boto3 dependency

It can be tricky to run `dbt` alongside other programs which use `boto3`, particularly those which rely on more recent Amazon APIs (Comprehend, Sagemaker, etc.). Would be nice if we could loosen the restrictions a bit.


Discussion:

Resolving patch:
diff --git a/plugins/redshift/setup.py b/plugins/redshift/setup.py
--- a/plugins/redshift/setup.py
+++ b/plugins/redshift/setup.py
@@ -25,8 +25,8 @@
     install_requires=[
         'dbt-core=={}'.format(package_version),
         'dbt-postgres=={}'.format(package_version),
-        'boto3>=1.6.23,<1.8.0',
-        'botocore>=1.9.23,<1.11.0',
+        'boto3>=1.6.23,<1.10.0',
+        'botocore>=1.9.23,<1.13.0',
         'psycopg2>=2.7.5,<2.8',
     ]
 )
diff --git a/plugins/snowflake/setup.py b/plugins/snowflake/setup.py
--- a/plugins/snowflake/setup.py
+++ b/plugins/snowflake/setup.py
@@ -25,8 +25,6 @@
     },
     install_requires=[
         'dbt-core=={}'.format(package_version),
-        'snowflake-connector-python>=1.4.9',
-        'boto3>=1.6.23,<1.8.0',
-        'botocore>=1.9.23,<1.11.0',
+        'snowflake-connector-python>=1.6.12',
     ]
 )


Test patch:
Repository: dbt-labs/dbt-core

Issue:
improve debug logging when connecting to hub site
## Feature

### Feature description
The hubsite should basically always be available, so if it is for some reason unavailable, dbt should log all of this info it can about the failure. At present, it just shows:
```
2019-02-11 09:03:17,675 (MainThread): Unable to connect to registry hub
```

Include:
 - The requests
 - The response code
 - Any exception info if an exception occurred

### Who will this benefit?
Project maintainers, dbt users



Discussion:

Resolving patch:
diff --git a/core/dbt/clients/registry.py b/core/dbt/clients/registry.py
--- a/core/dbt/clients/registry.py
+++ b/core/dbt/clients/registry.py
@@ -3,6 +3,7 @@
 import requests
 from dbt.exceptions import RegistryException
 from dbt.utils import memoized
+from dbt.logger import GLOBAL_LOGGER as logger
 import os
 
 if os.getenv('DBT_PACKAGE_HUB_URL'):
@@ -32,7 +33,10 @@ def wrapper(*args, **kwargs):
 @_wrap_exceptions
 def _get(path, registry_base_url=None):
     url = _get_url(path, registry_base_url)
+    logger.debug('Making package registry request: GET {}'.format(url))
     resp = requests.get(url)
+    logger.debug('Response from registry: GET {} {}'.format(url,
+                                                            resp.status_code))
     resp.raise_for_status()
     return resp.json()
 
diff --git a/core/dbt/node_runners.py b/core/dbt/node_runners.py
--- a/core/dbt/node_runners.py
+++ b/core/dbt/node_runners.py
@@ -298,9 +298,9 @@ def print_start_line(self):
                                         self.num_nodes)
 
     def print_result_line(self, result):
-        schema_name = self.node.schema
+        description = self.describe_node()
         dbt.ui.printer.print_model_result_line(result,
-                                               schema_name,
+                                               description,
                                                self.node_index,
                                                self.num_nodes)
 
diff --git a/core/dbt/ui/printer.py b/core/dbt/ui/printer.py
--- a/core/dbt/ui/printer.py
+++ b/core/dbt/ui/printer.py
@@ -146,17 +146,11 @@ def print_test_result_line(result, schema_name, index, total):
         result.execution_time)
 
 
-def print_model_result_line(result, schema_name, index, total):
-    model = result.node
-
+def print_model_result_line(result, description, index, total):
     info, status = get_printable_result(result, 'created', 'creating')
 
     print_fancy_output_line(
-        "{info} {model_type} model {schema}.{relation}".format(
-            info=info,
-            model_type=get_materialization(model),
-            schema=schema_name,
-            relation=model.get('alias')),
+        "{info} {description}".format(info=info, description=description),
         status,
         index,
         total,


Test patch:
Repository: dbt-labs/dbt-core

Issue:
archive: don't create schema if it already exists
When `dbt archive` is invoked, dbt will _always_ try to create a schema for each archive using:

```sql
create schema if not exists {{ schema }};
```

Instead, dbt should check if the schema already exists, and skip this step if it does indeed exist. If users invoke `dbt archive` with a user that lacks `create schema` permissions, the above statement will fail even if the schema already exists! Relevant code is here: https://github.com/fishtown-analytics/dbt/blob/development/dbt/include/global_project/macros/materializations/archive/archive.sql#L113


Discussion:
Kicking this out of the 0.10.2 milestone. `create` permissions for the dbt user are generally expected by dbt. This would be a good enhancement if someone has the cycles to work on it!

Resolving patch:
diff --git a/plugins/bigquery/dbt/adapters/bigquery/impl.py b/plugins/bigquery/dbt/adapters/bigquery/impl.py
--- a/plugins/bigquery/dbt/adapters/bigquery/impl.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/impl.py
@@ -75,6 +75,7 @@ def rename_relation(self, from_relation, to_relation):
             '`rename_relation` is not implemented for this adapter!'
         )
 
+    @available
     def list_schemas(self, database):
         conn = self.connections.get_thread_connection()
         client = conn.handle
@@ -84,6 +85,11 @@ def list_schemas(self, database):
                                                 include_all=True)
             return [ds.dataset_id for ds in all_datasets]
 
+    @available
+    def check_schema_exists(self, database, schema):
+        superself = super(BigQueryAdapter, self)
+        return superself.check_schema_exists(database, schema)
+
     def get_columns_in_relation(self, relation):
         try:
             table = self.connections.get_bq_table(


Test patch:
Repository: dbt-labs/dbt-core

Issue:
dbt debug is broken
## Issue
dbt debug is broken in wilt/chamberlain. Currently it reports:

```
  Connection test: ERROR

dbt was unable to connect to the specified database.
The database returned the following error:

  >connection never acquired for thread (94480, 4745348544), have []

Check your database credentials and try again. For more information, visit:
https://docs.getdbt.com/docs/configure-your-profile
```

It should not, the connection will work fine. I have a PR for this.



Discussion:

Resolving patch:
diff --git a/core/dbt/task/debug.py b/core/dbt/task/debug.py
--- a/core/dbt/task/debug.py
+++ b/core/dbt/task/debug.py
@@ -268,7 +268,8 @@ def _log_profile_fail(self):
     def _connection_result(self):
         adapter = get_adapter(self.profile)
         try:
-            adapter.execute('select 1 as id')
+            with adapter.connection_named('debug'):
+                adapter.execute('select 1 as id')
         except Exception as exc:
             self.messages.append(COULD_NOT_CONNECT_MESSAGE.format(
                 err=str(exc),


Test patch:
Repository: dbt-labs/dbt-core

Issue:
anonymous event tracking slows down dbt runs when there is no internet connection
## Issue

### Issue description
dbt's anonymous event tracking will try to send events even when there is no internet connection. In these cases, each ping takes about a second to timeout, adding a meaningful amount of time to every run. This was actually the root cause of this issue: https://github.com/fishtown-analytics/dbt/issues/1002

dbt should either try to fire these events in a background thread, or stop trying to send pings if the first one fails. I think the former is preferable, as these shouldn't be blocking calls even if the pings are successful.

### Results
dbt blocked on sending anonymous events, whereas these events should not have measurably impacted the performance of a given dbt run.

### System information
0.11.x

The operating system you're running on:
This happened in a Docker container, though I imagine it would happen on any OS

### Steps to reproduce
Turn off your wifi and wait


Discussion:

Resolving patch:
diff --git a/core/dbt/tracking.py b/core/dbt/tracking.py
--- a/core/dbt/tracking.py
+++ b/core/dbt/tracking.py
@@ -7,6 +7,7 @@
 import pytz
 import platform
 import uuid
+import requests
 import yaml
 import os
 
@@ -25,7 +26,35 @@
 
 DBT_INVOCATION_ENV = 'DBT_INVOCATION_ENV'
 
-emitter = Emitter(COLLECTOR_URL, protocol=COLLECTOR_PROTOCOL, buffer_size=1)
+
+class TimeoutEmitter(Emitter):
+    def __init__(self):
+        super(TimeoutEmitter, self).__init__(COLLECTOR_URL,
+                                             protocol=COLLECTOR_PROTOCOL,
+                                             buffer_size=1,
+                                             on_failure=self.handle_failure)
+
+    @staticmethod
+    def handle_failure(num_ok, unsent):
+        # num_ok will always be 0, unsent will always be 1 entry long, because
+        # the buffer is length 1, so not much to talk about
+        logger.warning('Error sending message, disabling tracking')
+        do_not_track()
+
+    def http_get(self, payload):
+        sp_logger.info("Sending GET request to %s..." % self.endpoint)
+        sp_logger.debug("Payload: %s" % payload)
+        r = requests.get(self.endpoint, params=payload, timeout=2.0)
+
+        msg = "GET request finished with status code: " + str(r.status_code)
+        if self.is_good_status_code(r.status_code):
+            sp_logger.info(msg)
+        else:
+            sp_logger.warn(msg)
+        return r
+
+
+emitter = TimeoutEmitter()
 tracker = Tracker(emitter, namespace="cf", app_id="dbt")
 
 active_user = None


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Retry deps requests made to the hub site
## Feature

### Feature description
About once per month, we can see that many `dbt deps` invocations fail all at the same time. This happens because of some sort of intermittent error with the hub site host. While it may be worth taking action to understand and improve the uptime of the hub site, it's also a good idea to add retries to these requests.

In the [registry._get](https://github.com/fishtown-analytics/dbt/blob/dev/wilt-chamberlain/core/dbt/clients/registry.py#L34) method, dbt should retry any request that fails 1) without producing a response code or 2) that fails with a 5xx response code.

Since many dbt jobs run at specific wall clock times (like midnight UTC), we should randomize the timeout between retries to avoid a thundering herd scenario.

- After the first failure, dbt should wait 5-10 seconds before retrying.
- If the request fails again, dbt should wait 5-10 seconds again.
- If that request fails, then dbt should raise the resulting exception

@beckjake @cmcarthur you guys have more experience with this class of problem than I do -- is this a reasonable solution? Would you recommend a different approach for the timeouts?


Discussion:
That all sounds reasonable enough to me. I assume the fail counter is per-attempt?
Yeah - i think that's right. dbt makes a couple of types of requests to the hub site:
- get the index
- get the versions for a specific package
- get the contents for a specific version of a package

We should assume that any of these types of queries can fail, and the fail counter is indeed per-attempt/request.

Resolving patch:
diff --git a/core/dbt/clients/registry.py b/core/dbt/clients/registry.py
--- a/core/dbt/clients/registry.py
+++ b/core/dbt/clients/registry.py
@@ -5,6 +5,7 @@
 from dbt.utils import memoized
 from dbt.logger import GLOBAL_LOGGER as logger
 import os
+import time
 
 if os.getenv('DBT_PACKAGE_HUB_URL'):
     DEFAULT_REGISTRY_BASE_URL = os.getenv('DBT_PACKAGE_HUB_URL')
@@ -22,11 +23,20 @@ def _get_url(url, registry_base_url=None):
 def _wrap_exceptions(fn):
     @wraps(fn)
     def wrapper(*args, **kwargs):
-        try:
-            return fn(*args, **kwargs)
-        except requests.exceptions.ConnectionError as e:
-            six.raise_from(
-                RegistryException('Unable to connect to registry hub'), e)
+        max_attempts = 5
+        attempt = 0
+        while True:
+            attempt += 1
+            try:
+                return fn(*args, **kwargs)
+            except requests.exceptions.ConnectionError as exc:
+                if attempt < max_attempts:
+                    time.sleep(1)
+                    continue
+                six.raise_from(
+                    RegistryException('Unable to connect to registry hub'),
+                    exc
+                )
     return wrapper
 
 


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Tracking timeout change
## Issue
We should change the [tracking timeout](https://github.com/fishtown-analytics/dbt/blob/3cac2d3ab77626d021bba262c56f2ef077ae7a88/core/dbt/tracking.py#L47) from 2s to 5s. While most events fire in well under 2 seconds, a single failure will cause the rest of the events for the session to be suppressed. We should up this timeout to 5 seconds to reduce the incidence of false-positives. @cmcarthur and I looked at the numbers a couple of weeks ago, and I think this should provide a good balance between tracking accuracy and user experience.


Discussion:

Resolving patch:
diff --git a/core/dbt/tracking.py b/core/dbt/tracking.py
--- a/core/dbt/tracking.py
+++ b/core/dbt/tracking.py
@@ -44,7 +44,7 @@ def handle_failure(num_ok, unsent):
     def http_get(self, payload):
         sp_logger.info("Sending GET request to %s..." % self.endpoint)
         sp_logger.debug("Payload: %s" % payload)
-        r = requests.get(self.endpoint, params=payload, timeout=2.0)
+        r = requests.get(self.endpoint, params=payload, timeout=5.0)
 
         msg = "GET request finished with status code: " + str(r.status_code)
         if self.is_good_status_code(r.status_code):


Test patch:
Repository: dbt-labs/dbt-core

Issue:
[Wilt Chamberlain] add deprecation warning for python 2
We intend to remove support for Python 2 in a near-future dbt release. Let's add a deprecation warning to Wilt for users running Python 2.


Discussion:

Resolving patch:
diff --git a/core/dbt/main.py b/core/dbt/main.py
--- a/core/dbt/main.py
+++ b/core/dbt/main.py
@@ -106,6 +106,8 @@ def main(args=None):
             logger.error(traceback.format_exc())
         exit_code = ExitCodes.UnhandledError
 
+    _python2_compatibility_message()
+
     sys.exit(exit_code)
 
 
@@ -175,6 +177,22 @@ def track_run(task):
         dbt.tracking.flush()
 
 
+_PYTHON_27_WARNING = '''
+Python 2.7 will reach the end of its life on January 1st, 2020.
+Please upgrade your Python as Python 2.7 won't be maintained after that date.
+A future version of dbt will drop support for Python 2.7.
+'''.strip()
+
+
+def _python2_compatibility_message():
+    if dbt.compat.WHICH_PYTHON != 2:
+        return
+
+    logger.critical(
+        dbt.ui.printer.red('DEPRECATION: ') + _PYTHON_27_WARNING
+    )
+
+
 def run_from_args(parsed):
     log_cache_events(getattr(parsed, 'log_cache_events', False))
     flags.set_from_args(parsed)


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Bigquery list_schemas can 404
## Issue 
On bigquery, `list_schemas` can error with a 404 if anything else is adding/deleting schemas.

### Issue description
I ran the dbt integration tests, got this: https://circleci.com/gh/fishtown-analytics/dbt/13902

### Results
I got a 404 in `list_schemas`

### System information
In circleci, on wilt-chamberlain since I parallelized integration tests

### Steps to reproduce

Run the tests!



Discussion:

Resolving patch:
diff --git a/plugins/bigquery/dbt/adapters/bigquery/impl.py b/plugins/bigquery/dbt/adapters/bigquery/impl.py
--- a/plugins/bigquery/dbt/adapters/bigquery/impl.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/impl.py
@@ -91,8 +91,9 @@ def list_schemas(self, database):
         client = conn.handle
 
         with self.connections.exception_handler('list dataset'):
+            # this is similar to how we have to deal with listing tables
             all_datasets = client.list_datasets(project=database,
-                                                include_all=True)
+                                                max_results=10000)
             return [ds.dataset_id for ds in all_datasets]
 
     @available


Test patch: