Repository: dbt-labs/dbt-core

Issue:
Support reading both `yml` and `yaml` file extensions for schema configuration files.
**Issue**: Currently `dbt` checks checks for configuration files inside the `models` folder with the `yml` file extension on `dbt run`.

**Problem**: This may cause confusion for users that are used to using the [official](https://yaml.org/faq.html) `yaml` file extension for YAML files.

**Impact**: configurations fail to load on `dbt run`, [source block definitions aren't parsed](https://getdbt.slack.com/archives/C2JRRQDTL/p1565668068461100?thread_ts=1565647114.452700&cid=C2JRRQDTL), etc.

**Proposed solution**: support reading configuration files with both `yml` and `yaml` file extensions, or throw  informative exceptions as desired in one case or the other.

cc @clrcrl 


Discussion:
Thanks @soumyadsanyal! I like the idea of including .yaml in addition to .yml.

I think the operative lines of code to change are [around here](https://github.com/fishtown-analytics/dbt/blob/dev/0.14.1/core/dbt/parser/schemas.py#L550). We may ultimately need to change the [find_matching](https://github.com/fishtown-analytics/dbt/blob/dev/0.14.1/core/dbt/clients/system.py) method to accept a list of file patterns instead of just accepting a single string.

Last: while this feels like a small change, it's probably something we'll want to do in a minor release (like 0.15.0). This could be a breaking change as dbt might start evaluating `.yaml` files that it had previously ignored (which could lead to compilation errors).
@drewbanin Interested to pick this up if this is still a priority, as you have mentioned that this could be a breaking change, in order to maintain backward compatibility one of the ways we could proceed is as:

1. User's can specify a flag in project settings on whether to enable all patterns for yml, sql or not, if flag is not present, the code will work the same way as its working now. Or they can specify the file extension themselves.
2. In case flag is mentioned, we will include all forms of yaml, sql etc.. (caps or small).
3. find_matching will take list as input instead of string. 

hey @sumanau7 - I'm not so concerned about making this a "breaking" change - it's just something I'd want to communicate at release-time.

I think we should do the following things:
 - make file name lookups case-insensitive (.sql, .SQL, .yml, .YML) all work
 - add .yaml as a recognized extension for schema files

via
https://github.com/fishtown-analytics/dbt/blob/b77b3a45661e68a647a6be91e197c1cd13545982/core/dbt/parser/schemas.py#L129-L132

The annoying part: I don't really want to support both `.yml` and `.yaml` for the core config files (`dbt_project.yml`, `profiles.yml`, `packages.yml`) as I think that would lead to a ton of user-space errors. So, I'm ok with adding as a historical footnote that fact that I picked the wrong extension (`.yml` instead of `.yaml`) back in 2016 and just live with the consequences :)

Let me know what you think!
@drewbanin You are right, supporting both `.yml` and `.yaml` for the core config files may lead to user space errors. 
IMO lets make file name lookups case-insensitive and throw a warning in case core config files are found with YAML extension explaining in order to support new standards, .YAML file will be read in future versions of dbt.

Let's say we receive a positive response and there are no new issues from user asking us to revert the change we can then make it live in the next release. I would say 3 months of warning period and then make the the new change live. (Supporting both .yml and .yaml)
thoughts ?
Hey @sumanau7 - that sounds like a really good approach!

Let's go ahead and:
1. make a PR to make file extensions case-insensitive
2. search for `.yaml` files and raise a warning indicating that these files are not currently parsed by dbt, but will be in a future release

Really good thinking here! Are you interested in sending through a PR for this change? We'd be very helpful to assist you however we can if so :)
@drewbanin I will start work on this should be able to share a PR with TODO's and basic implementation over the weekend to get early feedback and then can take it forward.
@drewbanin Does the first draft looks good ?

Resolving patch:
diff --git a/core/dbt/clients/system.py b/core/dbt/clients/system.py
--- a/core/dbt/clients/system.py
+++ b/core/dbt/clients/system.py
@@ -3,6 +3,7 @@
 import json
 import os
 import os.path
+import re
 import shutil
 import subprocess
 import sys
@@ -40,6 +41,8 @@ def find_matching(
     """
     matching = []
     root_path = os.path.normpath(root_path)
+    regex = fnmatch.translate(file_pattern)
+    reobj = re.compile(regex, re.IGNORECASE)
 
     for relative_path_to_search in relative_paths_to_search:
         absolute_path_to_search = os.path.join(
@@ -52,8 +55,7 @@ def find_matching(
                 relative_path = os.path.relpath(
                     absolute_path, absolute_path_to_search
                 )
-
-                if fnmatch.fnmatch(local_file, file_pattern):
+                if reobj.match(local_file):
                     matching.append({
                         'searched_path': relative_path_to_search,
                         'absolute_path': absolute_path,
diff --git a/core/dbt/parser/schemas.py b/core/dbt/parser/schemas.py
--- a/core/dbt/parser/schemas.py
+++ b/core/dbt/parser/schemas.py
@@ -29,7 +29,8 @@
 )
 from dbt.exceptions import (
     validator_error_message, JSONValidationException,
-    raise_invalid_schema_yml_version, ValidationException, CompilationException
+    raise_invalid_schema_yml_version, ValidationException,
+    CompilationException, warn_or_error
 )
 from dbt.node_types import NodeType
 from dbt.parser.base import SimpleParser
@@ -130,6 +131,20 @@ def resource_type(self) -> NodeType:
         return NodeType.Test
 
     def get_paths(self):
+        # TODO: In order to support this, make FilesystemSearcher accept a list
+        # of file patterns. eg: ['.yml', '.yaml']
+        yaml_files = list(FilesystemSearcher(
+            self.project, self.project.all_source_paths, '.yaml'
+        ))
+        if yaml_files:
+            warn_or_error(
+                'A future version of dbt will parse files with both'
+                ' .yml and .yaml file extensions. dbt found'
+                f' {len(yaml_files)} files with .yaml extensions in'
+                ' your dbt project. To avoid errors when upgrading'
+                ' to a future release, either remove these files from'
+                ' your dbt project, or change their extensions.'
+            )
         return FilesystemSearcher(
             self.project, self.project.all_source_paths, '.yml'
         )


Test patch:
diff --git a/test/integration/008_schema_tests_test/case-sensitive-models/lowercase.sql b/test/integration/008_schema_tests_test/case-sensitive-models/lowercase.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/008_schema_tests_test/case-sensitive-models/lowercase.sql
@@ -0,0 +1 @@
+select 1 as id
diff --git a/test/integration/008_schema_tests_test/case-sensitive-models/schema.yaml b/test/integration/008_schema_tests_test/case-sensitive-models/schema.yaml
new file mode 100644
diff --git a/test/integration/008_schema_tests_test/case-sensitive-models/schema.yml b/test/integration/008_schema_tests_test/case-sensitive-models/schema.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/008_schema_tests_test/case-sensitive-models/schema.yml
@@ -0,0 +1,15 @@
+version: 2
+
+models:
+  - name: lowercase
+    columns:
+      - name: id
+        quote: true
+        tests:
+          - unique
+  - name: uppercase
+    columns:
+      - name: id
+        quote: true
+        tests:
+          - unique
diff --git a/test/integration/008_schema_tests_test/case-sensitive-models/uppercase.SQL b/test/integration/008_schema_tests_test/case-sensitive-models/uppercase.SQL
new file mode 100644
--- /dev/null
+++ b/test/integration/008_schema_tests_test/case-sensitive-models/uppercase.SQL
@@ -0,0 +1 @@
+select 1 as id
diff --git a/test/integration/008_schema_tests_test/test_schema_v2_tests.py b/test/integration/008_schema_tests_test/test_schema_v2_tests.py
--- a/test/integration/008_schema_tests_test/test_schema_v2_tests.py
+++ b/test/integration/008_schema_tests_test/test_schema_v2_tests.py
@@ -361,3 +361,43 @@ def test_postgres_argument_rendering(self):
         results = self.run_dbt(['test', '--vars', '{myvar: foo}'])
         self.assertEqual(len(results), 1)
         self.run_dbt(['test'], expect_pass=False)
+
+
+class TestSchemaCaseInsensitive(DBTIntegrationTest):
+    @property
+    def schema(self):
+        return "schema_tests_008"
+
+    @property
+    def models(self):
+        return "case-sensitive-models"
+
+    @use_profile('postgres')
+    def test_postgres_schema_lowercase_sql(self):
+        results = self.run_dbt(strict=False)
+        self.assertEqual(len(results), 2)
+        results = self.run_dbt(['test', '-m', 'lowercase'], strict=False)
+        self.assertEqual(len(results), 1)
+
+    @use_profile('postgres')
+    def test_postgres_schema_uppercase_sql(self):
+        results = self.run_dbt(strict=False)
+        self.assertEqual(len(results), 2)
+        results = self.run_dbt(['test', '-m', 'uppercase'], strict=False)
+        self.assertEqual(len(results), 1)
+
+
+class TestSchemaYAMLExtension(DBTIntegrationTest):
+    @property
+    def schema(self):
+        return "schema_tests_008"
+
+    @property
+    def models(self):
+        return "case-sensitive-models"
+
+    @use_profile('postgres')
+    def test_postgres_yaml_extension(self):
+        with self.assertRaises(Exception) as exc:
+            self.run_dbt(["run"])
+        self.assertIn('yaml', str(exc.exception))
diff --git a/test/unit/test_system_client.py b/test/unit/test_system_client.py
--- a/test/unit/test_system_client.py
+++ b/test/unit/test_system_client.py
@@ -2,7 +2,7 @@
 import shutil
 import stat
 import unittest
-from tempfile import mkdtemp
+from tempfile import mkdtemp, NamedTemporaryFile
 
 from dbt.exceptions import ExecutableError, WorkingDirectoryError
 import dbt.clients.system
@@ -133,3 +133,53 @@ def test__ok(self):
         out, err = dbt.clients.system.run_cmd(self.run_dir, self.exists_cmd)
         self.assertEqual(out.strip(), b'hello')
         self.assertEqual(err.strip(), b'')
+
+
+class TestFindMatching(unittest.TestCase):
+
+    def setUp(self):
+        self.base_dir = mkdtemp()
+        self.tempdir = mkdtemp(dir=self.base_dir)
+
+    def test_find_matching_lowercase_file_pattern(self):
+        with NamedTemporaryFile(
+            prefix='sql-files', suffix='.sql', dir=self.tempdir
+        ) as named_file:
+            file_path = os.path.dirname(named_file.name)
+            relative_path = os.path.basename(file_path)
+            out = dbt.clients.system.find_matching(
+                self.base_dir, [relative_path], '*.sql'
+            )
+            expected_output = [{
+                'searched_path': relative_path,
+                'absolute_path': named_file.name,
+                'relative_path': os.path.basename(named_file.name)
+            }]
+            self.assertEqual(out, expected_output)
+
+    def test_find_matching_uppercase_file_pattern(self):
+        with NamedTemporaryFile(prefix='sql-files', suffix='.SQL', dir=self.tempdir) as named_file:
+            file_path = os.path.dirname(named_file.name)
+            relative_path = os.path.basename(file_path)
+            out = dbt.clients.system.find_matching(
+                self.base_dir, [relative_path], '*.sql'
+            )
+            expected_output = [{
+                'searched_path': relative_path,
+                'absolute_path': named_file.name,
+                'relative_path': os.path.basename(named_file.name)
+            }]
+            self.assertEqual(out, expected_output)
+
+    def test_find_matching_file_pattern_not_found(self):
+        with NamedTemporaryFile(
+            prefix='sql-files', suffix='.SQLT', dir=self.tempdir
+        ):
+            out = dbt.clients.system.find_matching(self.tempdir, [''], '*.sql')
+            self.assertEqual(out, [])
+
+    def tearDown(self):
+        try:
+            shutil.rmtree(self.base_dir)
+        except:
+            pass