Repository: dbt-labs/dbt-core

Issue:
Command line vars are parsed inconsistently in the dbt server
From @jtcohen6:

### Bug description

The dbt server handles command line `--var` arguments differently than `dbt run` invocations when partial parsing is enabled. It appears that at parse-time, the dbt server is using the variable values declared in `dbt_project.yml`, whereas at runtime, it uses the vars provided with `--vars`  in the rpc method.

### Results

I would expect values passed to `dbt run --vars` to be parsed, first and foremost, and override any default values, before any metadata queries are run against the database. That's true in dbt CLI and not true when using the dbt server.

### Steps to reproduce

Given a `dbt_project.yml` that includes
```yml
models:
  vars:
    environment_name: dev
```

and a model:

```sql

{{ config(database = var('environment_name')) }}

select 1 as fun
```

Run
```
dbt run -m my_test_model --vars 'environment_name: qa'
```

#### dbt CLI:
```
2020-03-27 21:33:04.743677 (MainThread): Found 6 models, 5 tests, 0 snapshots, 0 analyses, 131 macros, 1 operation, 0 seed files, 0 sources
2020-03-27 21:33:04.750941 (MainThread): 
2020-03-27 21:33:04.751394 (MainThread): Acquiring new snowflake connection "master".
2020-03-27 21:33:04.751511 (MainThread): Opening a new connection, currently in state init
2020-03-27 21:33:04.759351 (ThreadPoolExecutor-0_0): Acquiring new snowflake connection "list_analytics_qa".
2020-03-27 21:33:04.759620 (ThreadPoolExecutor-0_0): Opening a new connection, currently in state init
2020-03-27 21:33:04.884238 (ThreadPoolExecutor-0_0): Using snowflake connection "list_analytics_qa".
2020-03-27 21:33:04.884403 (ThreadPoolExecutor-0_0): On list_analytics_qa: /* {"app": "dbt", "dbt_version": "0.16.0", "profile_name": "jaffle_shop", "target_name": "dev", "connection_name": "list_analytics_qa"} */

    show terse schemas in database analytics_qa
    limit 10000
```

#### dbt Server:

```
2020-03-27 21:41:30.004433Z: Found 6 models, 5 tests, 0 snapshots, 0 analyses, 131 macros, 1 operation, 0 seed files, 0 sources
2020-03-27 21:41:30.010170Z: Acquiring new snowflake connection "master".
2020-03-27 21:41:30.010319Z: Opening a new connection, currently in state init
2020-03-27 21:41:30.018723Z: Acquiring new snowflake connection "list_analytics_dev".
2020-03-27 21:41:30.018850Z: Opening a new connection, currently in state init
2020-03-27 21:41:30.107756Z: Using snowflake connection "list_analytics_dev".
2020-03-27 21:41:30.107908Z: On list_analytics_dev: /* {"app": "dbt", "dbt_version": "0.16.0", "profile_name": "user", "target_name": "dev", "connection_name": "list_analytics_dev"} */

    show terse schemas in database analytics_dev
    limit 10000
```

In both cases I'm running `dbt==0.16.0`

-------

I would expect some weirdness around the `--vars` flag when partial parsing is enabled. By definition, we're using the previously-parsed variables to skip re-parsing every model with our updated variables, so in some ways, this is working as intended.

I believe we _should_ be storing a hash of CLI `--vars` and using that hash to determine if the partial parse cache is valid for the run or not. In this case, I suspect that something funny is happening w/r/t the dbt server's `cli_args` method.


Discussion:
@beckjake to advise on what kinds of tradeoffs we'd need to make to get something like this working more consistently.
I think the trick here is that it's not partial parsing that's involved (that does correctly invalidate the cache!) Instead it's the "API calls inherit the manifest from the server" behavior.

To fix this, I'll add a check and if vars changed, the task will re-generate the manifest at call-time. This will result in `cli_args` calls having to rebuild the manifest each time, which could be rough on performance.

In the future, we might want to make a cache that stores one manifest per `--vars` the server has seen, or something like that.

Resolving patch:
diff --git a/core/dbt/task/rpc/cli.py b/core/dbt/task/rpc/cli.py
--- a/core/dbt/task/rpc/cli.py
+++ b/core/dbt/task/rpc/cli.py
@@ -13,6 +13,7 @@
     Result,
 )
 from dbt.exceptions import InternalException
+from dbt.perf_utils import get_full_manifest
 from dbt.utils import parse_cli_vars
 
 from .base import RPCTask
@@ -39,14 +40,6 @@ def __init__(self, args, config, manifest):
     def set_config(self, config):
         super().set_config(config)
 
-        # read any cli vars we got and use it to update cli_vars
-        self.config.cli_vars.update(
-            parse_cli_vars(getattr(self.args, 'vars', '{}'))
-        )
-        # rewrite args.vars to reflect our merged vars
-        self.args.vars = yaml.safe_dump(self.config.cli_vars)
-        self.config.args = self.args
-
         if self.task_type is None:
             raise InternalException('task type not set for set_config')
         if issubclass(self.task_type, RemoteManifestMethod):
@@ -96,6 +89,23 @@ def handle_request(self) -> Result:
                 'CLI task is in a bad state: handle_request called with no '
                 'real_task set!'
             )
+
+        # It's important to update cli_vars here, because set_config()'s
+        # `self.config` is before the fork(), so it would alter the behavior of
+        # future calls.
+
+        # read any cli vars we got and use it to update cli_vars
+        self.config.cli_vars.update(
+            parse_cli_vars(getattr(self.args, 'vars', '{}'))
+        )
+        # If this changed the vars, rewrite args.vars to reflect our merged
+        # vars and reload the manifest.
+        dumped = yaml.safe_dump(self.config.cli_vars)
+        if dumped != self.args.vars:
+            self.real_task.args.vars = dumped
+            if isinstance(self.real_task, RemoteManifestMethod):
+                self.real_task.manifest = get_full_manifest(self.config)
+
         # we parsed args from the cli, so we're set on that front
         return self.real_task.handle_request()
 


Test patch:
diff --git a/test/rpc/test_base.py b/test/rpc/test_base.py
deleted file mode 100644
--- a/test/rpc/test_base.py
+++ /dev/null
@@ -1,916 +0,0 @@
-# flake8: disable=redefined-outer-name
-from datetime import datetime, timedelta
-import time
-import yaml
-from .util import (
-    ProjectDefinition, get_querier,
-)
-
-
-def test_rpc_basics(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        models={'my_model.sql': 'select 1 as id'}
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        querier.async_wait_for_result(querier.run_sql('select 1 as id'))
-
-        querier.async_wait_for_result(querier.run())
-
-        querier.async_wait_for_result(
-            querier.run_sql('select * from {{ ref("my_model") }}')
-        )
-
-        querier.async_wait_for_error(
-            querier.run_sql('select * from {{ reff("my_model") }}')
-        )
-
-
-def deps_with_packages(packages, bad_packages, project_dir, profiles_dir, schema):
-    project = ProjectDefinition(
-        models={
-            'my_model.sql': 'select 1 as id',
-        },
-        packages={'packages': packages},
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_dir,
-        profiles_dir=profiles_dir,
-        schema=schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        # we should be able to run sql queries at startup
-        querier.async_wait_for_result(querier.run_sql('select 1 as id'))
-
-        # the status should be something positive
-        querier.is_result(querier.status())
-
-        # deps should pass
-        querier.async_wait_for_result(querier.deps())
-
-        # queries should work after deps
-        tok1 = querier.is_async_result(querier.run())
-        tok2 = querier.is_async_result(querier.run_sql('select 1 as id'))
-
-        querier.is_result(querier.async_wait(tok2))
-        querier.is_result(querier.async_wait(tok1))
-
-        # now break the project
-        project.packages['packages'] = bad_packages
-        project.write_packages(project_dir, remove=True)
-
-        # queries should still work because we haven't reloaded
-        tok1 = querier.is_async_result(querier.run())
-        tok2 = querier.is_async_result(querier.run_sql('select 1 as id'))
-
-        querier.is_result(querier.async_wait(tok2))
-        querier.is_result(querier.async_wait(tok1))
-
-        # now run deps again, it should be sad
-        querier.async_wait_for_error(querier.deps())
-        # it should also not be running.
-        result = querier.is_result(querier.ps(active=True, completed=False))
-        assert result['rows'] == []
-
-        # fix packages again
-        project.packages['packages'] = packages
-        project.write_packages(project_dir, remove=True)
-        # keep queries broken, we haven't run deps yet
-        querier.is_error(querier.run())
-
-        # deps should pass now
-        querier.async_wait_for_result(querier.deps())
-        querier.is_result(querier.status())
-
-        tok1 = querier.is_async_result(querier.run())
-        tok2 = querier.is_async_result(querier.run_sql('select 1 as id'))
-
-        querier.is_result(querier.async_wait(tok2))
-        querier.is_result(querier.async_wait(tok1))
-
-
-def test_rpc_deps_packages(project_root, profiles_root, postgres_profile, unique_schema):
-    packages = [{
-        'package': 'fishtown-analytics/dbt_utils',
-        'version': '0.2.1',
-    }]
-    bad_packages = [{
-        'package': 'fishtown-analytics/dbt_util',
-        'version': '0.2.1',
-    }]
-    deps_with_packages(packages, bad_packages, project_root, profiles_root, unique_schema)
-
-
-def test_rpc_deps_git(project_root, profiles_root, postgres_profile, unique_schema):
-    packages = [{
-        'git': 'https://github.com/fishtown-analytics/dbt-utils.git',
-        'revision': '0.2.1'
-    }]
-    # if you use a bad URL, git thinks it's a private repo and prompts for auth
-    bad_packages = [{
-        'git': 'https://github.com/fishtown-analytics/dbt-utils.git',
-        'revision': 'not-a-real-revision'
-    }]
-    deps_with_packages(packages, bad_packages, project_root, profiles_root, unique_schema)
-
-
-bad_schema_yml = '''
-version: 2
-sources:
-  - name: test_source
-    loader: custom
-    schema: "{{ var('test_run_schema') }}"
-    tables:
-      - name: test_table
-        identifier: source
-        tests:
-          - relationships:
-            # this is invalid
-              - column_name: favorite_color
-              - to: ref('descendant_model')
-              - field: favorite_color
-'''
-
-fixed_schema_yml = '''
-version: 2
-sources:
-  - name: test_source
-    loader: custom
-    schema: "{{ var('test_run_schema') }}"
-    tables:
-      - name: test_table
-        identifier: source
-'''
-
-
-def test_rpc_status_error(project_root, profiles_root, postgres_profile, unique_schema):
-    project = ProjectDefinition(
-        models={
-            'descendant_model.sql': 'select * from {{ source("test_source", "test_table") }}',
-            'schema.yml': bad_schema_yml,
-        }
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-        criteria='error',
-    )
-    with querier_ctx as querier:
-
-        # the status should be an error result
-        result = querier.is_result(querier.status())
-        assert 'error' in result
-        assert 'message' in result['error']
-        assert 'Invalid test config' in result['error']['message']
-        assert 'state' in result
-        assert result['state'] == 'error'
-        assert 'logs' in result
-        logs = result['logs']
-        assert len(logs) > 0
-        for key in ('message', 'timestamp', 'levelname', 'level'):
-            assert key in logs[0]
-        assert 'pid' in result
-        assert querier.server.pid == result['pid']
-
-        error = querier.is_error(querier.compile_sql('select 1 as id'))
-        assert 'code' in error
-        assert error['code'] == 10011
-        assert 'message' in error
-        assert error['message'] == 'RPC server failed to compile project, call the "status" method for compile status'
-        assert 'data' in error
-        assert 'message' in error['data']
-        assert 'Invalid test config' in error['data']['message']
-
-        # deps should fail because it still can't parse the manifest
-        querier.async_wait_for_error(querier.deps())
-
-        # and not resolve the issue
-        result = querier.is_result(querier.status())
-        assert 'error' in result
-        assert 'message' in result['error']
-        assert 'Invalid test config' in result['error']['message']
-
-        error = querier.is_error(querier.compile_sql('select 1 as id'))
-        assert 'code' in error
-        assert error['code'] == 10011
-
-        project.models['schema.yml'] = fixed_schema_yml
-        project.write_models(project_root, remove=True)
-
-        # deps should work
-        querier.async_wait_for_result(querier.deps())
-
-        result = querier.is_result(querier.status())
-        assert result.get('error') is None
-        assert 'state' in result
-        assert result['state'] == 'ready'
-
-        querier.is_result(querier.compile_sql('select 1 as id'))
-
-
-def test_gc_change_interval(project_root, profiles_root, postgres_profile, unique_schema):
-    project = ProjectDefinition(
-        models={'my_model.sql': 'select 1 as id'}
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-
-        for _ in range(10):
-            querier.async_wait_for_result(querier.run())
-
-        result = querier.is_result(querier.ps(True, True))
-        assert len(result['rows']) == 10
-
-        result = querier.is_result(querier.gc(settings=dict(maxsize=1000, reapsize=5, auto_reap_age=0.1)))
-
-        for k in ('deleted', 'missing', 'running'):
-            assert k in result
-            assert len(result[k]) == 0
-
-        time.sleep(0.5)
-
-        result = querier.is_result(querier.ps(True, True))
-        assert len(result['rows']) == 0
-
-        result = querier.is_result(querier.gc(settings=dict(maxsize=2, reapsize=5, auto_reap_age=100000)))
-        for k in ('deleted', 'missing', 'running'):
-            assert k in result
-            assert len(result[k]) == 0
-
-        time.sleep(0.5)
-
-        for _ in range(10):
-            querier.async_wait_for_result(querier.run())
-
-        time.sleep(0.5)
-        result = querier.is_result(querier.ps(True, True))
-        assert len(result['rows']) == 2
-
-
-def test_ps_poll_output_match(project_root, profiles_root, postgres_profile, unique_schema):
-    project = ProjectDefinition(
-        models={'my_model.sql': 'select 1 as id'}
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-
-        poll_result = querier.async_wait_for_result(querier.run())
-
-        result = querier.is_result(querier.ps(active=True, completed=True))
-        assert 'rows' in result
-        rows = result['rows']
-        assert len(rows) == 1
-        ps_result = rows[0]
-
-        for key in ('start', 'end', 'elapsed', 'state'):
-            assert ps_result[key] == poll_result[key]
-
-
-macros_data = '''
-{% macro foo() %}
-    {{ return(1) }}
-{% endmacro %}
-{% macro bar(value) %}
-    {{ return(value + 1) }}
-{% endmacro %}
-{% macro quux(value) %}
-    {{ return(asdf) }}
-{% endmacro %}
-'''
-
-
-def test_run_operation(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        models={'my_model.sql': 'select 1 as id'},
-        macros={
-            'my_macros.sql': macros_data,
-        }
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        poll_result = querier.async_wait_for_result(
-            querier.run_operation(macro='foo', args={})
-        )
-
-        assert 'success' in poll_result
-        assert poll_result['success'] is True
-
-        poll_result = querier.async_wait_for_result(
-            querier.run_operation(macro='bar', args={'value': 10})
-        )
-
-        assert 'success' in poll_result
-        assert poll_result['success'] is True
-
-        poll_result = querier.async_wait_for_result(
-            querier.run_operation(macro='baz', args={}),
-            state='failed',
-        )
-        assert 'state' in poll_result
-        assert poll_result['state'] == 'failed'
-
-        poll_result = querier.async_wait_for_result(
-            querier.run_operation(macro='quux', args={})
-        )
-        assert 'success' in poll_result
-        assert poll_result['success'] is True
-
-
-def test_run_operation_cli(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        models={'my_model.sql': 'select 1 as id'},
-        macros={
-            'my_macros.sql': macros_data,
-        }
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        poll_result = querier.async_wait_for_result(
-            querier.cli_args(cli='run-operation foo')
-        )
-
-        assert 'success' in poll_result
-        assert poll_result['success'] is True
-
-        bar_cmd = '''run-operation bar --args="{'value': 10}"'''
-        poll_result = querier.async_wait_for_result(
-            querier.cli_args(cli=bar_cmd)
-        )
-
-        assert 'success' in poll_result
-        assert poll_result['success'] is True
-
-        poll_result = querier.async_wait_for_result(
-            querier.cli_args(cli='run-operation baz'),
-            state='failed',
-        )
-        assert 'state' in poll_result
-        assert poll_result['state'] == 'failed'
-
-        poll_result = querier.async_wait_for_result(
-            querier.cli_args(cli='run-operation quux')
-        )
-        assert 'success' in poll_result
-        assert poll_result['success'] is True
-
-
-snapshot_data = '''
-{% snapshot snapshot_actual %}
-
-    {{
-        config(
-            target_database=database,
-            target_schema=schema,
-            unique_key='id',
-            strategy='timestamp',
-            updated_at='updated_at',
-        )
-    }}
-    select 1 as id, '2019-10-31 23:59:40' as updated_at
-
-{% endsnapshot %}
-'''
-
-
-def test_snapshots(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        snapshots={'my_snapshots.sql': snapshot_data},
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        results = querier.async_wait_for_result(querier.snapshot())
-        assert len(results['results']) == 1
-
-        results = querier.async_wait_for_result(querier.snapshot(
-            exclude=['snapshot_actual'])
-        )
-
-        results = querier.async_wait_for_result(
-            querier.snapshot(select=['snapshot_actual'])
-        )
-        assert len(results['results']) == 1
-
-
-def test_snapshots_cli(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        snapshots={'my_snapshots.sql': snapshot_data},
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        results = querier.async_wait_for_result(
-            querier.cli_args(cli='snapshot')
-        )
-        assert len(results['results']) == 1
-
-        results = querier.async_wait_for_result(
-            querier.cli_args(cli='snapshot --exclude=snapshot_actual')
-        )
-        assert len(results['results']) == 0
-
-        results = querier.async_wait_for_result(
-            querier.cli_args(cli='snapshot --select=snapshot_actual')
-        )
-        assert len(results['results']) == 1
-
-
-def assert_has_threads(results, num_threads):
-    assert 'logs' in results
-    c_logs = [l for l in results['logs'] if 'Concurrency' in l['message']]
-    assert len(c_logs) == 1, \
-        f'Got invalid number of concurrency logs ({len(c_logs)})'
-    assert 'message' in c_logs[0]
-    assert f'Concurrency: {num_threads} threads' in c_logs[0]['message']
-
-
-def test_rpc_run_threads(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        models={'my_model.sql': 'select 1 as id'}
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-    with querier_ctx as querier:
-        results = querier.async_wait_for_result(querier.run(threads=5))
-        assert_has_threads(results, 5)
-
-        results = querier.async_wait_for_result(
-            querier.cli_args('run --threads=7')
-        )
-        assert_has_threads(results, 7)
-
-
-def test_rpc_compile_threads(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        models={'my_model.sql': 'select 1 as id'}
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-    with querier_ctx as querier:
-        results = querier.async_wait_for_result(querier.compile(threads=5))
-        assert_has_threads(results, 5)
-
-        results = querier.async_wait_for_result(
-            querier.cli_args('compile --threads=7')
-        )
-        assert_has_threads(results, 7)
-
-
-def test_rpc_test_threads(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    schema_yaml = {
-        'version': 2,
-        'models': [{
-            'name': 'my_model',
-            'columns': [
-                {
-                    'name': 'id',
-                    'tests': ['not_null', 'unique'],
-                },
-            ],
-        }],
-    }
-    project = ProjectDefinition(
-        models={
-            'my_model.sql': 'select 1 as id',
-            'schema.yml': yaml.safe_dump(schema_yaml)}
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-    with querier_ctx as querier:
-        # first run dbt to get the model built
-        querier.async_wait_for_result(querier.run())
-
-        results = querier.async_wait_for_result(querier.test(threads=5))
-        assert_has_threads(results, 5)
-
-        results = querier.async_wait_for_result(
-            querier.cli_args('test --threads=7')
-        )
-        assert_has_threads(results, 7)
-
-
-def test_rpc_snapshot_threads(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        snapshots={'my_snapshots.sql': snapshot_data},
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        results = querier.async_wait_for_result(querier.snapshot(threads=5))
-        assert_has_threads(results, 5)
-
-        results = querier.async_wait_for_result(
-            querier.cli_args('snapshot --threads=7')
-        )
-        assert_has_threads(results, 7)
-
-
-def test_rpc_seed_threads(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        project_data={'seeds': {'config': {'quote_columns': False}}},
-        seeds={'data.csv': 'a,b\n1,hello\n2,goodbye'},
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        results = querier.async_wait_for_result(querier.seed(threads=5))
-        assert_has_threads(results, 5)
-
-        results = querier.async_wait_for_result(
-            querier.cli_args('seed --threads=7')
-        )
-        assert_has_threads(results, 7)
-
-
-def test_rpc_seed_include_exclude(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        project_data={'seeds': {'config': {'quote_columns': False}}},
-        seeds={
-            'data_1.csv': 'a,b\n1,hello\n2,goodbye',
-            'data_2.csv': 'a,b\n1,data',
-        },
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        results = querier.async_wait_for_result(querier.seed(select=['data_1']))
-        assert len(results['results']) == 1
-        results = querier.async_wait_for_result(querier.seed(select='data_1'))
-        assert len(results['results']) == 1
-        results = querier.async_wait_for_result(querier.cli_args('seed --select=data_1'))
-        assert len(results['results']) == 1
-
-        results = querier.async_wait_for_result(querier.seed(exclude=['data_2']))
-        assert len(results['results']) == 1
-        results = querier.async_wait_for_result(querier.seed(exclude='data_2'))
-        assert len(results['results']) == 1
-        results = querier.async_wait_for_result(querier.cli_args('seed --exclude=data_2'))
-        assert len(results['results']) == 1
-
-
-sleeper_sql = '''
-{{ log('test output', info=True) }}
-{{ run_query('select * from pg_sleep(20)') }}
-select 1 as id
-'''
-
-logger_sql = '''
-{{ log('test output', info=True) }}
-select 1 as id
-'''
-
-
-def find_log_ordering(logs, *messages) -> bool:
-    log_iter = iter(logs)
-    found = 0
-
-    while found < len(messages):
-        try:
-            log = next(log_iter)
-        except StopIteration:
-            return False
-        if messages[found] in log['message']:
-            found += 1
-    return True
-
-
-def poll_logs(querier, token):
-    has_log = querier.is_result(querier.poll(token))
-    assert 'logs' in has_log
-    return has_log['logs']
-
-
-def wait_for_log_ordering(querier, token, attempts, *messages) -> int:
-    for _ in range(attempts):
-        time.sleep(1)
-        logs = poll_logs(querier, token)
-        if find_log_ordering(logs, *messages):
-            return len(logs)
-
-    msg = 'Never got expected messages {} in {}'.format(
-        messages,
-        [log['message'] for log in logs],
-    )
-    assert False, msg
-
-
-def test_get_status(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        models={'my_model.sql': 'select 1 as id'},
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        # make sure that logs_start/logs are honored during a task
-        token = querier.is_async_result(querier.run_sql(sleeper_sql))
-
-        no_log = querier.is_result(querier.poll(token, logs=False))
-        assert 'logs' in no_log
-        assert len(no_log['logs']) == 0
-
-        num_logs = wait_for_log_ordering(querier, token, 10)
-
-        trunc_log = querier.is_result(querier.poll(token, logs_start=num_logs))
-        assert 'logs' in trunc_log
-        assert len(trunc_log['logs']) == 0
-
-        querier.kill(token)
-
-        # make sure that logs_start/logs are honored after a task has finished
-        token = querier.is_async_result(querier.run_sql(logger_sql))
-        result = querier.is_result(querier.async_wait(token))
-        assert 'logs' in result
-        num_logs = len(result['logs'])
-        assert num_logs > 0
-
-        result = querier.is_result(querier.poll(token, logs_start=num_logs))
-        assert 'logs' in result
-        assert len(result['logs']) == 0
-
-        result = querier.is_result(querier.poll(token, logs=False))
-        assert 'logs' in result
-        assert len(result['logs']) == 0
-
-
-source_freshness_schema_yml = '''
-version: 2
-sources:
-  - name: test_source
-    loaded_at_field: b
-    schema: {schema}
-    freshness:
-      warn_after: {{count: 10, period: hour}}
-      error_after: {{count: 1, period: day}}
-    tables:
-      - name: test_table
-        identifier: source
-      - name: failure_table
-        identifier: other_source
-'''
-
-
-def test_source_freshness(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    start_time = datetime.utcnow()
-    warn_me = start_time - timedelta(hours=18)
-    error_me = start_time - timedelta(days=2)
-    # this should trigger a 'warn'
-    project = ProjectDefinition(
-        project_data={'seeds': {'config': {'quote_columns': False}}},
-        seeds={
-            'source.csv': 'a,b\n1,{}\n'.format(error_me.strftime('%Y-%m-%d %H:%M:%S')),
-            'other_source.csv': 'a,b\n1,{}\n'.format(error_me.strftime('%Y-%m-%d %H:%M:%S'))
-        },
-        models={
-            'sources.yml': source_freshness_schema_yml.format(schema=unique_schema),
-        },
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        seeds = querier.async_wait_for_result(querier.seed())
-        assert len(seeds['results']) == 2
-        # should error
-        error_results = querier.async_wait_for_result(querier.snapshot_freshness(), state='failed')
-        assert len(error_results['results']) == 2
-        for result in error_results['results']:
-            assert result['status'] == 'error'
-        error_results = querier.async_wait_for_result(querier.cli_args('source snapshot-freshness'), state='failed')
-        assert len(error_results['results']) == 2
-        for result in error_results['results']:
-            assert result['status'] == 'error'
-
-        project.seeds['source.csv'] += '2,{}\n'.format(warn_me.strftime('%Y-%m-%d %H:%M:%S'))
-        project.write_seeds(project_root, remove=True)
-        querier.async_wait_for_result(querier.seed())
-        # should warn
-        warn_results = querier.async_wait_for_result(querier.snapshot_freshness(select='test_source.test_table'))
-        assert len(warn_results['results']) == 1
-        assert warn_results['results'][0]['status'] == 'warn'
-        warn_results = querier.async_wait_for_result(querier.cli_args('source snapshot-freshness -s test_source.test_table'))
-        assert len(warn_results['results']) == 1
-        assert warn_results['results'][0]['status'] == 'warn'
-
-        project.seeds['source.csv'] += '3,{}\n'.format(start_time.strftime('%Y-%m-%d %H:%M:%S'))
-        project.write_seeds(project_root, remove=True)
-        querier.async_wait_for_result(querier.seed())
-        # should pass!
-        pass_results = querier.async_wait_for_result(querier.snapshot_freshness(select=['test_source.test_table']))
-        assert len(pass_results['results']) == 1
-        assert pass_results['results'][0]['status'] == 'pass'
-        pass_results = querier.async_wait_for_result(querier.cli_args('source snapshot-freshness --select test_source.test_table'))
-        assert len(pass_results['results']) == 1
-        assert pass_results['results'][0]['status'] == 'pass'
-
-
-def test_missing_tag_sighup(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        models={
-            'my_docs.md': '{% docs asdf %}have a close tag{% enddocs %}',
-        },
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-    with querier_ctx as querier:
-        # everything is fine
-        assert querier.wait_for_status('ready') is True
-
-        # write a junk docs file
-        project.models['my_docs.md'] = '{% docs asdf %}do not have a close tag'
-        project.write_models(project_root, remove=True)
-
-        querier.sighup()
-
-        assert querier.wait_for_status('error') is True
-        result = querier.is_result(querier.status())
-        assert 'error' in result
-        assert 'message' in result['error']
-        assert 'without finding a close tag for docs' in result['error']['message']
-
-        project.models['my_docs.md'] = '{% docs asdf %}have a close tag again{% enddocs %}'
-        project.write_models(project_root, remove=True)
-
-        querier.sighup()
-
-        assert querier.wait_for_status('ready') is True
-
-
-def test_rpc_vars(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        models={
-            'my_model.sql': 'select {{ var("param") }} as id',
-        },
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        results = querier.async_wait_for_result(querier.cli_args('run --vars "{param: 100}"'))
-        assert len(results['results']) == 1
-        assert results['results'][0]['node']['compiled_sql'] == 'select 100 as id'
-
-
-def test_get_manifest(
-    project_root, profiles_root, postgres_profile, unique_schema
-):
-    project = ProjectDefinition(
-        models={
-            'my_model.sql': 'select 1 as id',
-        },
-    )
-    querier_ctx = get_querier(
-        project_def=project,
-        project_dir=project_root,
-        profiles_dir=profiles_root,
-        schema=unique_schema,
-        test_kwargs={},
-    )
-
-    with querier_ctx as querier:
-        results = querier.async_wait_for_result(querier.cli_args('run'))
-        assert len(results['results']) == 1
-        assert results['results'][0]['node']['compiled_sql'] == 'select 1 as id'
-        result = querier.async_wait_for_result(querier.get_manifest())
-        assert 'manifest' in result
-        manifest = result['manifest']
-        assert manifest['nodes']['model.test.my_model']['raw_sql'] == 'select 1 as id'
-        assert 'manifest' in result
-        manifest = result['manifest']
-        assert manifest['nodes']['model.test.my_model']['compiled_sql'] == 'select 1 as id'
diff --git a/test/rpc/test_compile.py b/test/rpc/test_compile.py
new file mode 100644
--- /dev/null
+++ b/test/rpc/test_compile.py
@@ -0,0 +1,28 @@
+from .util import (
+    assert_has_threads,
+    get_querier,
+    ProjectDefinition,
+)
+
+
+def test_rpc_compile_threads(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        models={'my_model.sql': 'select 1 as id'}
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+    with querier_ctx as querier:
+        results = querier.async_wait_for_result(querier.compile(threads=5))
+        assert_has_threads(results, 5)
+
+        results = querier.async_wait_for_result(
+            querier.cli_args('compile --threads=7')
+        )
+        assert_has_threads(results, 7)
diff --git a/test/rpc/test_deps.py b/test/rpc/test_deps.py
new file mode 100644
--- /dev/null
+++ b/test/rpc/test_deps.py
@@ -0,0 +1,96 @@
+from .util import (
+    get_querier,
+    ProjectDefinition,
+)
+
+
+def deps_with_packages(packages, bad_packages, project_dir, profiles_dir, schema):
+    project = ProjectDefinition(
+        models={
+            'my_model.sql': 'select 1 as id',
+        },
+        packages={'packages': packages},
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_dir,
+        profiles_dir=profiles_dir,
+        schema=schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        # we should be able to run sql queries at startup
+        querier.async_wait_for_result(querier.run_sql('select 1 as id'))
+
+        # the status should be something positive
+        querier.is_result(querier.status())
+
+        # deps should pass
+        querier.async_wait_for_result(querier.deps())
+
+        # queries should work after deps
+        tok1 = querier.is_async_result(querier.run())
+        tok2 = querier.is_async_result(querier.run_sql('select 1 as id'))
+
+        querier.is_result(querier.async_wait(tok2))
+        querier.is_result(querier.async_wait(tok1))
+
+        # now break the project
+        project.packages['packages'] = bad_packages
+        project.write_packages(project_dir, remove=True)
+
+        # queries should still work because we haven't reloaded
+        tok1 = querier.is_async_result(querier.run())
+        tok2 = querier.is_async_result(querier.run_sql('select 1 as id'))
+
+        querier.is_result(querier.async_wait(tok2))
+        querier.is_result(querier.async_wait(tok1))
+
+        # now run deps again, it should be sad
+        querier.async_wait_for_error(querier.deps())
+        # it should also not be running.
+        result = querier.is_result(querier.ps(active=True, completed=False))
+        assert result['rows'] == []
+
+        # fix packages again
+        project.packages['packages'] = packages
+        project.write_packages(project_dir, remove=True)
+        # keep queries broken, we haven't run deps yet
+        querier.is_error(querier.run())
+
+        # deps should pass now
+        querier.async_wait_for_result(querier.deps())
+        querier.is_result(querier.status())
+
+        tok1 = querier.is_async_result(querier.run())
+        tok2 = querier.is_async_result(querier.run_sql('select 1 as id'))
+
+        querier.is_result(querier.async_wait(tok2))
+        querier.is_result(querier.async_wait(tok1))
+
+
+def test_rpc_deps_packages(project_root, profiles_root, postgres_profile, unique_schema):
+    packages = [{
+        'package': 'fishtown-analytics/dbt_utils',
+        'version': '0.2.1',
+    }]
+    bad_packages = [{
+        'package': 'fishtown-analytics/dbt_util',
+        'version': '0.2.1',
+    }]
+    deps_with_packages(packages, bad_packages, project_root, profiles_root, unique_schema)
+
+
+def test_rpc_deps_git(project_root, profiles_root, postgres_profile, unique_schema):
+    packages = [{
+        'git': 'https://github.com/fishtown-analytics/dbt-utils.git',
+        'revision': '0.2.1'
+    }]
+    # if you use a bad URL, git thinks it's a private repo and prompts for auth
+    bad_packages = [{
+        'git': 'https://github.com/fishtown-analytics/dbt-utils.git',
+        'revision': 'not-a-real-revision'
+    }]
+    deps_with_packages(packages, bad_packages, project_root, profiles_root, unique_schema)
+
diff --git a/test/rpc/test_management.py b/test/rpc/test_management.py
new file mode 100644
--- /dev/null
+++ b/test/rpc/test_management.py
@@ -0,0 +1,362 @@
+import time
+from .util import (
+    get_querier,
+    ProjectDefinition,
+)
+
+
+def test_rpc_basics(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        models={'my_model.sql': 'select 1 as id'}
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        querier.async_wait_for_result(querier.run_sql('select 1 as id'))
+
+        querier.async_wait_for_result(querier.run())
+
+        querier.async_wait_for_result(
+            querier.run_sql('select * from {{ ref("my_model") }}')
+        )
+
+        querier.async_wait_for_error(
+            querier.run_sql('select * from {{ reff("my_model") }}')
+        )
+
+
+bad_schema_yml = '''
+version: 2
+sources:
+  - name: test_source
+    loader: custom
+    schema: "{{ var('test_run_schema') }}"
+    tables:
+      - name: test_table
+        identifier: source
+        tests:
+          - relationships:
+            # this is invalid
+              - column_name: favorite_color
+              - to: ref('descendant_model')
+              - field: favorite_color
+'''
+
+fixed_schema_yml = '''
+version: 2
+sources:
+  - name: test_source
+    loader: custom
+    schema: "{{ var('test_run_schema') }}"
+    tables:
+      - name: test_table
+        identifier: source
+'''
+
+
+def test_rpc_status_error(project_root, profiles_root, postgres_profile, unique_schema):
+    project = ProjectDefinition(
+        models={
+            'descendant_model.sql': 'select * from {{ source("test_source", "test_table") }}',
+            'schema.yml': bad_schema_yml,
+        }
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+        criteria='error',
+    )
+    with querier_ctx as querier:
+
+        # the status should be an error result
+        result = querier.is_result(querier.status())
+        assert 'error' in result
+        assert 'message' in result['error']
+        assert 'Invalid test config' in result['error']['message']
+        assert 'state' in result
+        assert result['state'] == 'error'
+        assert 'logs' in result
+        logs = result['logs']
+        assert len(logs) > 0
+        for key in ('message', 'timestamp', 'levelname', 'level'):
+            assert key in logs[0]
+        assert 'pid' in result
+        assert querier.server.pid == result['pid']
+
+        error = querier.is_error(querier.compile_sql('select 1 as id'))
+        assert 'code' in error
+        assert error['code'] == 10011
+        assert 'message' in error
+        assert error['message'] == 'RPC server failed to compile project, call the "status" method for compile status'
+        assert 'data' in error
+        assert 'message' in error['data']
+        assert 'Invalid test config' in error['data']['message']
+
+        # deps should fail because it still can't parse the manifest
+        querier.async_wait_for_error(querier.deps())
+
+        # and not resolve the issue
+        result = querier.is_result(querier.status())
+        assert 'error' in result
+        assert 'message' in result['error']
+        assert 'Invalid test config' in result['error']['message']
+
+        error = querier.is_error(querier.compile_sql('select 1 as id'))
+        assert 'code' in error
+        assert error['code'] == 10011
+
+        project.models['schema.yml'] = fixed_schema_yml
+        project.write_models(project_root, remove=True)
+
+        # deps should work
+        querier.async_wait_for_result(querier.deps())
+
+        result = querier.is_result(querier.status())
+        assert result.get('error') is None
+        assert 'state' in result
+        assert result['state'] == 'ready'
+
+        querier.is_result(querier.compile_sql('select 1 as id'))
+
+
+def test_gc_change_interval(project_root, profiles_root, postgres_profile, unique_schema):
+    project = ProjectDefinition(
+        models={'my_model.sql': 'select 1 as id'}
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+
+        for _ in range(10):
+            querier.async_wait_for_result(querier.run())
+
+        result = querier.is_result(querier.ps(True, True))
+        assert len(result['rows']) == 10
+
+        result = querier.is_result(querier.gc(settings=dict(maxsize=1000, reapsize=5, auto_reap_age=0.1)))
+
+        for k in ('deleted', 'missing', 'running'):
+            assert k in result
+            assert len(result[k]) == 0
+
+        time.sleep(0.5)
+
+        result = querier.is_result(querier.ps(True, True))
+        assert len(result['rows']) == 0
+
+        result = querier.is_result(querier.gc(settings=dict(maxsize=2, reapsize=5, auto_reap_age=100000)))
+        for k in ('deleted', 'missing', 'running'):
+            assert k in result
+            assert len(result[k]) == 0
+
+        time.sleep(0.5)
+
+        for _ in range(10):
+            querier.async_wait_for_result(querier.run())
+
+        time.sleep(0.5)
+        result = querier.is_result(querier.ps(True, True))
+        assert len(result['rows']) == 2
+
+
+def test_ps_poll_output_match(project_root, profiles_root, postgres_profile, unique_schema):
+    project = ProjectDefinition(
+        models={'my_model.sql': 'select 1 as id'}
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+
+        poll_result = querier.async_wait_for_result(querier.run())
+
+        result = querier.is_result(querier.ps(active=True, completed=True))
+        assert 'rows' in result
+        rows = result['rows']
+        assert len(rows) == 1
+        ps_result = rows[0]
+
+        for key in ('start', 'end', 'elapsed', 'state'):
+            assert ps_result[key] == poll_result[key]
+
+
+sleeper_sql = '''
+{{ log('test output', info=True) }}
+{{ run_query('select * from pg_sleep(20)') }}
+select 1 as id
+'''
+
+logger_sql = '''
+{{ log('test output', info=True) }}
+select 1 as id
+'''
+
+
+def find_log_ordering(logs, *messages) -> bool:
+    log_iter = iter(logs)
+    found = 0
+
+    while found < len(messages):
+        try:
+            log = next(log_iter)
+        except StopIteration:
+            return False
+        if messages[found] in log['message']:
+            found += 1
+    return True
+
+
+def poll_logs(querier, token):
+    has_log = querier.is_result(querier.poll(token))
+    assert 'logs' in has_log
+    return has_log['logs']
+
+
+def wait_for_log_ordering(querier, token, attempts, *messages) -> int:
+    for _ in range(attempts):
+        time.sleep(1)
+        logs = poll_logs(querier, token)
+        if find_log_ordering(logs, *messages):
+            return len(logs)
+
+    msg = 'Never got expected messages {} in {}'.format(
+        messages,
+        [log['message'] for log in logs],
+    )
+    assert False, msg
+
+
+def test_get_status(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        models={'my_model.sql': 'select 1 as id'},
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        # make sure that logs_start/logs are honored during a task
+        token = querier.is_async_result(querier.run_sql(sleeper_sql))
+
+        no_log = querier.is_result(querier.poll(token, logs=False))
+        assert 'logs' in no_log
+        assert len(no_log['logs']) == 0
+
+        num_logs = wait_for_log_ordering(querier, token, 10)
+
+        trunc_log = querier.is_result(querier.poll(token, logs_start=num_logs))
+        assert 'logs' in trunc_log
+        assert len(trunc_log['logs']) == 0
+
+        querier.kill(token)
+
+        # make sure that logs_start/logs are honored after a task has finished
+        token = querier.is_async_result(querier.run_sql(logger_sql))
+        result = querier.is_result(querier.async_wait(token))
+        assert 'logs' in result
+        num_logs = len(result['logs'])
+        assert num_logs > 0
+
+        result = querier.is_result(querier.poll(token, logs_start=num_logs))
+        assert 'logs' in result
+        assert len(result['logs']) == 0
+
+        result = querier.is_result(querier.poll(token, logs=False))
+        assert 'logs' in result
+        assert len(result['logs']) == 0
+
+
+def test_missing_tag_sighup(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        models={
+            'my_docs.md': '{% docs asdf %}have a close tag{% enddocs %}',
+        },
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+    with querier_ctx as querier:
+        # everything is fine
+        assert querier.wait_for_status('ready') is True
+
+        # write a junk docs file
+        project.models['my_docs.md'] = '{% docs asdf %}do not have a close tag'
+        project.write_models(project_root, remove=True)
+
+        querier.sighup()
+
+        assert querier.wait_for_status('error') is True
+        result = querier.is_result(querier.status())
+        assert 'error' in result
+        assert 'message' in result['error']
+        assert 'without finding a close tag for docs' in result['error']['message']
+
+        project.models['my_docs.md'] = '{% docs asdf %}have a close tag again{% enddocs %}'
+        project.write_models(project_root, remove=True)
+
+        querier.sighup()
+
+        assert querier.wait_for_status('ready') is True
+
+
+def test_get_manifest(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        models={
+            'my_model.sql': 'select 1 as id',
+        },
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        results = querier.async_wait_for_result(querier.cli_args('run'))
+        assert len(results['results']) == 1
+        assert results['results'][0]['node']['compiled_sql'] == 'select 1 as id'
+        result = querier.async_wait_for_result(querier.get_manifest())
+        assert 'manifest' in result
+        manifest = result['manifest']
+        assert manifest['nodes']['model.test.my_model']['raw_sql'] == 'select 1 as id'
+        assert 'manifest' in result
+        manifest = result['manifest']
+        assert manifest['nodes']['model.test.my_model']['compiled_sql'] == 'select 1 as id'
diff --git a/test/rpc/test_run.py b/test/rpc/test_run.py
new file mode 100644
--- /dev/null
+++ b/test/rpc/test_run.py
@@ -0,0 +1,77 @@
+from .util import (
+    assert_has_threads,
+    get_querier,
+    ProjectDefinition,
+)
+
+
+def test_rpc_run_threads(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        models={'my_model.sql': 'select 1 as id'}
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+    with querier_ctx as querier:
+        results = querier.async_wait_for_result(querier.run(threads=5))
+        assert_has_threads(results, 5)
+
+        results = querier.async_wait_for_result(
+            querier.cli_args('run --threads=7')
+        )
+        assert_has_threads(results, 7)
+
+
+def test_rpc_run_vars(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        models={
+            'my_model.sql': 'select {{ var("param") }} as id',
+        },
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        results = querier.async_wait_for_result(querier.cli_args('run --vars "{param: 100}"'))
+        assert len(results['results']) == 1
+        assert results['results'][0]['node']['compiled_sql'] == 'select 100 as id'
+
+
+def test_rpc_run_vars_compiled(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        models={
+            'my_model.sql': '{{ config(materialized=var("materialized_var", "view")) }} select 1 as id',
+        },
+    )
+
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+    with querier_ctx as querier:
+        results = querier.async_wait_for_result(querier.cli_args('run --vars "{materialized_var: table}"'))
+        assert len(results['results']) == 1
+        assert results['results'][0]['node']['config']['materialized'] == 'table'
+        # make sure that `--vars` doesn't update global state - if it does,
+        # this run() will result in a view!
+        results = querier.async_wait_for_result(querier.cli_args('run'))
+        assert len(results['results']) == 1
+        assert results['results'][0]['node']['config']['materialized'] == 'view'
diff --git a/test/rpc/test_run_operation.py b/test/rpc/test_run_operation.py
new file mode 100644
--- /dev/null
+++ b/test/rpc/test_run_operation.py
@@ -0,0 +1,110 @@
+from .util import (
+    get_querier,
+    ProjectDefinition,
+)
+
+macros_data = '''
+{% macro foo() %}
+    {{ return(1) }}
+{% endmacro %}
+{% macro bar(value) %}
+    {{ return(value + 1) }}
+{% endmacro %}
+{% macro quux(value) %}
+    {{ return(asdf) }}
+{% endmacro %}
+'''
+
+
+def test_run_operation(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        models={'my_model.sql': 'select 1 as id'},
+        macros={
+            'my_macros.sql': macros_data,
+        }
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        poll_result = querier.async_wait_for_result(
+            querier.run_operation(macro='foo', args={})
+        )
+
+        assert 'success' in poll_result
+        assert poll_result['success'] is True
+
+        poll_result = querier.async_wait_for_result(
+            querier.run_operation(macro='bar', args={'value': 10})
+        )
+
+        assert 'success' in poll_result
+        assert poll_result['success'] is True
+
+        poll_result = querier.async_wait_for_result(
+            querier.run_operation(macro='baz', args={}),
+            state='failed',
+        )
+        assert 'state' in poll_result
+        assert poll_result['state'] == 'failed'
+
+        poll_result = querier.async_wait_for_result(
+            querier.run_operation(macro='quux', args={})
+        )
+        assert 'success' in poll_result
+        assert poll_result['success'] is True
+
+
+def test_run_operation_cli(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        models={'my_model.sql': 'select 1 as id'},
+        macros={
+            'my_macros.sql': macros_data,
+        }
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        poll_result = querier.async_wait_for_result(
+            querier.cli_args(cli='run-operation foo')
+        )
+
+        assert 'success' in poll_result
+        assert poll_result['success'] is True
+
+        bar_cmd = '''run-operation bar --args="{'value': 10}"'''
+        poll_result = querier.async_wait_for_result(
+            querier.cli_args(cli=bar_cmd)
+        )
+
+        assert 'success' in poll_result
+        assert poll_result['success'] is True
+
+        poll_result = querier.async_wait_for_result(
+            querier.cli_args(cli='run-operation baz'),
+            state='failed',
+        )
+        assert 'state' in poll_result
+        assert poll_result['state'] == 'failed'
+
+        poll_result = querier.async_wait_for_result(
+            querier.cli_args(cli='run-operation quux')
+        )
+        assert 'success' in poll_result
+        assert poll_result['success'] is True
+
diff --git a/test/rpc/test_seed.py b/test/rpc/test_seed.py
new file mode 100644
--- /dev/null
+++ b/test/rpc/test_seed.py
@@ -0,0 +1,64 @@
+from .util import (
+    assert_has_threads,
+    get_querier,
+    ProjectDefinition,
+)
+
+
+def test_rpc_seed_threads(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        project_data={'seeds': {'config': {'quote_columns': False}}},
+        seeds={'data.csv': 'a,b\n1,hello\n2,goodbye'},
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        results = querier.async_wait_for_result(querier.seed(threads=5))
+        assert_has_threads(results, 5)
+
+        results = querier.async_wait_for_result(
+            querier.cli_args('seed --threads=7')
+        )
+        assert_has_threads(results, 7)
+
+
+def test_rpc_seed_include_exclude(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        project_data={'seeds': {'config': {'quote_columns': False}}},
+        seeds={
+            'data_1.csv': 'a,b\n1,hello\n2,goodbye',
+            'data_2.csv': 'a,b\n1,data',
+        },
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        results = querier.async_wait_for_result(querier.seed(select=['data_1']))
+        assert len(results['results']) == 1
+        results = querier.async_wait_for_result(querier.seed(select='data_1'))
+        assert len(results['results']) == 1
+        results = querier.async_wait_for_result(querier.cli_args('seed --select=data_1'))
+        assert len(results['results']) == 1
+
+        results = querier.async_wait_for_result(querier.seed(exclude=['data_2']))
+        assert len(results['results']) == 1
+        results = querier.async_wait_for_result(querier.seed(exclude='data_2'))
+        assert len(results['results']) == 1
+        results = querier.async_wait_for_result(querier.cli_args('seed --exclude=data_2'))
+        assert len(results['results']) == 1
diff --git a/test/rpc/test_snapshots.py b/test/rpc/test_snapshots.py
new file mode 100644
--- /dev/null
+++ b/test/rpc/test_snapshots.py
@@ -0,0 +1,106 @@
+from .util import (
+    assert_has_threads,
+    get_querier,
+    ProjectDefinition,
+)
+
+snapshot_data = '''
+{% snapshot snapshot_actual %}
+
+    {{
+        config(
+            target_database=database,
+            target_schema=schema,
+            unique_key='id',
+            strategy='timestamp',
+            updated_at='updated_at',
+        )
+    }}
+    select 1 as id, '2019-10-31 23:59:40' as updated_at
+
+{% endsnapshot %}
+'''
+
+
+def test_snapshots(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        snapshots={'my_snapshots.sql': snapshot_data},
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        results = querier.async_wait_for_result(querier.snapshot())
+        assert len(results['results']) == 1
+
+        results = querier.async_wait_for_result(querier.snapshot(
+            exclude=['snapshot_actual'])
+        )
+
+        results = querier.async_wait_for_result(
+            querier.snapshot(select=['snapshot_actual'])
+        )
+        assert len(results['results']) == 1
+
+
+def test_snapshots_cli(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        snapshots={'my_snapshots.sql': snapshot_data},
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        results = querier.async_wait_for_result(
+            querier.cli_args(cli='snapshot')
+        )
+        assert len(results['results']) == 1
+
+        results = querier.async_wait_for_result(
+            querier.cli_args(cli='snapshot --exclude=snapshot_actual')
+        )
+        assert len(results['results']) == 0
+
+        results = querier.async_wait_for_result(
+            querier.cli_args(cli='snapshot --select=snapshot_actual')
+        )
+        assert len(results['results']) == 1
+
+
+def test_rpc_snapshot_threads(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        snapshots={'my_snapshots.sql': snapshot_data},
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        results = querier.async_wait_for_result(querier.snapshot(threads=5))
+        assert_has_threads(results, 5)
+
+        results = querier.async_wait_for_result(
+            querier.cli_args('snapshot --threads=7')
+        )
+        assert_has_threads(results, 7)
+
diff --git a/test/rpc/test_source_freshness.py b/test/rpc/test_source_freshness.py
new file mode 100644
--- /dev/null
+++ b/test/rpc/test_source_freshness.py
@@ -0,0 +1,82 @@
+from datetime import datetime, timedelta
+from .util import (
+    get_querier,
+    ProjectDefinition,
+)
+
+source_freshness_schema_yml = '''
+version: 2
+sources:
+  - name: test_source
+    loaded_at_field: b
+    schema: {schema}
+    freshness:
+      warn_after: {{count: 10, period: hour}}
+      error_after: {{count: 1, period: day}}
+    tables:
+      - name: test_table
+        identifier: source
+      - name: failure_table
+        identifier: other_source
+'''
+
+
+def test_source_freshness(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    start_time = datetime.utcnow()
+    warn_me = start_time - timedelta(hours=18)
+    error_me = start_time - timedelta(days=2)
+    # this should trigger a 'warn'
+    project = ProjectDefinition(
+        project_data={'seeds': {'config': {'quote_columns': False}}},
+        seeds={
+            'source.csv': 'a,b\n1,{}\n'.format(error_me.strftime('%Y-%m-%d %H:%M:%S')),
+            'other_source.csv': 'a,b\n1,{}\n'.format(error_me.strftime('%Y-%m-%d %H:%M:%S'))
+        },
+        models={
+            'sources.yml': source_freshness_schema_yml.format(schema=unique_schema),
+        },
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        seeds = querier.async_wait_for_result(querier.seed())
+        assert len(seeds['results']) == 2
+        # should error
+        error_results = querier.async_wait_for_result(querier.snapshot_freshness(), state='failed')
+        assert len(error_results['results']) == 2
+        for result in error_results['results']:
+            assert result['status'] == 'error'
+        error_results = querier.async_wait_for_result(querier.cli_args('source snapshot-freshness'), state='failed')
+        assert len(error_results['results']) == 2
+        for result in error_results['results']:
+            assert result['status'] == 'error'
+
+        project.seeds['source.csv'] += '2,{}\n'.format(warn_me.strftime('%Y-%m-%d %H:%M:%S'))
+        project.write_seeds(project_root, remove=True)
+        querier.async_wait_for_result(querier.seed())
+        # should warn
+        warn_results = querier.async_wait_for_result(querier.snapshot_freshness(select='test_source.test_table'))
+        assert len(warn_results['results']) == 1
+        assert warn_results['results'][0]['status'] == 'warn'
+        warn_results = querier.async_wait_for_result(querier.cli_args('source snapshot-freshness -s test_source.test_table'))
+        assert len(warn_results['results']) == 1
+        assert warn_results['results'][0]['status'] == 'warn'
+
+        project.seeds['source.csv'] += '3,{}\n'.format(start_time.strftime('%Y-%m-%d %H:%M:%S'))
+        project.write_seeds(project_root, remove=True)
+        querier.async_wait_for_result(querier.seed())
+        # should pass!
+        pass_results = querier.async_wait_for_result(querier.snapshot_freshness(select=['test_source.test_table']))
+        assert len(pass_results['results']) == 1
+        assert pass_results['results'][0]['status'] == 'pass'
+        pass_results = querier.async_wait_for_result(querier.cli_args('source snapshot-freshness --select test_source.test_table'))
+        assert len(pass_results['results']) == 1
+        assert pass_results['results'][0]['status'] == 'pass'
diff --git a/test/rpc/test_test.py b/test/rpc/test_test.py
new file mode 100644
--- /dev/null
+++ b/test/rpc/test_test.py
@@ -0,0 +1,46 @@
+import yaml
+from .util import (
+    assert_has_threads,
+    get_querier,
+    ProjectDefinition,
+)
+
+
+def test_rpc_test_threads(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    schema_yaml = {
+        'version': 2,
+        'models': [{
+            'name': 'my_model',
+            'columns': [
+                {
+                    'name': 'id',
+                    'tests': ['not_null', 'unique'],
+                },
+            ],
+        }],
+    }
+    project = ProjectDefinition(
+        models={
+            'my_model.sql': 'select 1 as id',
+            'schema.yml': yaml.safe_dump(schema_yaml)}
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+    with querier_ctx as querier:
+        # first run dbt to get the model built
+        querier.async_wait_for_result(querier.run())
+
+        results = querier.async_wait_for_result(querier.test(threads=5))
+        assert_has_threads(results, 5)
+
+        results = querier.async_wait_for_result(
+            querier.cli_args('test --threads=7')
+        )
+        assert_has_threads(results, 7)
diff --git a/test/rpc/util.py b/test/rpc/util.py
--- a/test/rpc/util.py
+++ b/test/rpc/util.py
@@ -616,3 +616,12 @@ def get_querier(
     )
     with schema_ctx, server_ctx as server:
         yield Querier(server)
+
+
+def assert_has_threads(results, num_threads):
+    assert 'logs' in results
+    c_logs = [l for l in results['logs'] if 'Concurrency' in l['message']]
+    assert len(c_logs) == 1, \
+        f'Got invalid number of concurrency logs ({len(c_logs)})'
+    assert 'message' in c_logs[0]
+    assert f'Concurrency: {num_threads} threads' in c_logs[0]['message']