Repository: nautobot/nautobot

Issue:
Git repository sync should warn if files appear to be out of place
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for proposing specific new features or enhancements.
    If you have a general idea or question, please start a discussion instead:
    https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot.
-->
### Environment
* Python version: any
* Nautobot version: 1.0.1

<!--
    Describe in detail the new functionality you are proposing. Include any
    specific changes to work flows, data models, or the user interface.
-->
### Proposed Functionality

If a Git repository is mis-structured in an easily detectable way, it would be great if the Git sync process would detect such cases and log appropriate warning/error messages, for example if:

- a Job class is defined in the project root rather than in `/jobs/`
- YAML or JSON files identifiable as config context data are located outside of `/config_contexts/`
- a directory exists under `/export_templates/` that doesn't match a known `app_label` or `modelname` (e.g. `/export_templates/device/` instead of `/export_templates/dcim/device/`)
- etc.

<!--
    Convey an example use case for your proposed feature. Write from the
    perspective of a Nautobot user who would benefit from the proposed
    functionality and describe how.
--->
### Use Case

Improved ease of troubleshooting, especially when first integrating a Git repository with Nautobot. Currently the logging during a sync is mostly success-path-focused and doesn't make it obvious why data might not be detected or imported.

<!--
    Note any changes to the database schema necessary to support the new
    feature. For example, does the proposal require adding a new model or
    field? (Not all new features require database changes.)
--->
### Database Changes

None
<!--
    List any new dependencies on external libraries or services that this new
    feature would introduce. For example, does the proposal require the
    installation of a new Python package? (Not all new features introduce new
    dependencies.)
-->
### External Dependencies

None



Discussion:

Resolving patch:
diff --git a/nautobot/extras/datasources/git.py b/nautobot/extras/datasources/git.py
--- a/nautobot/extras/datasources/git.py
+++ b/nautobot/extras/datasources/git.py
@@ -371,10 +371,7 @@ def update_git_config_contexts(repository_record, job_result):
         try:
             with open(os.path.join(config_context_path, file_name), "r") as fd:
                 # The data file can be either JSON or YAML; since YAML is a superset of JSON, we can load it regardless
-                try:
-                    context_data = yaml.safe_load(fd)
-                except Exception as exc:
-                    raise RuntimeError(f"Error in loading config context data from `{file_name}`: {exc}")
+                context_data = yaml.safe_load(fd)
 
             # A file can contain one config context dict or a list thereof
             if isinstance(context_data, dict):
@@ -385,13 +382,11 @@ def update_git_config_contexts(repository_record, job_result):
                     context_name = import_config_context(context_data_entry, repository_record, job_result, logger)
                     managed_config_contexts.add(context_name)
             else:
-                raise RuntimeError(
-                    f"Error in loading config context data from `{file_name}`: data must be a dict or list of dicts"
-                )
+                raise RuntimeError("data must be a dict or list of dicts")
 
         except Exception as exc:
             job_result.log(
-                str(exc),
+                f"Error in loading config context data from `{file_name}`: {exc}",
                 level_choice=LogLevelChoices.LOG_FAILURE,
                 grouping="config contexts",
                 logger=logger,
@@ -411,6 +406,15 @@ def update_git_config_contexts(repository_record, job_result):
         "tenants",
         "tags",
     ):
+        if os.path.isdir(os.path.join(repository_record.filesystem_path, filter_type)):
+            job_result.log(
+                f'Found "{filter_type}" directory in the repository root. If this is meant to contain config contexts, '
+                "it should be moved into a `config_contexts/` subdirectory.",
+                level_choice=LogLevelChoices.LOG_WARNING,
+                grouping="config contexts",
+                logger=logger,
+            )
+
         dir_path = os.path.join(config_context_path, filter_type)
         if not os.path.isdir(dir_path):
             continue
@@ -425,10 +429,7 @@ def update_git_config_contexts(repository_record, job_result):
             try:
                 with open(os.path.join(dir_path, file_name), "r") as fd:
                     # Data file can be either JSON or YAML; since YAML is a superset of JSON, we can load it regardless
-                    try:
-                        context_data = yaml.safe_load(fd)
-                    except Exception as exc:
-                        raise RuntimeError(f"Error in loading config context data from `{file_name}`: {exc}")
+                    context_data = yaml.safe_load(fd)
 
                 # Unlike the above case, these files always contain just a single config context record
 
@@ -439,7 +440,7 @@ def update_git_config_contexts(repository_record, job_result):
                 managed_config_contexts.add(context_name)
             except Exception as exc:
                 job_result.log(
-                    str(exc),
+                    f"Error in loading config context data from `{file_name}`: {exc}",
                     level_choice=LogLevelChoices.LOG_FAILURE,
                     grouping="config contexts",
                     logger=logger,
@@ -448,6 +449,15 @@ def update_git_config_contexts(repository_record, job_result):
 
     # Finally, handle device- and virtual-machine-specific "local" context in (devices|virtual_machines)/<name>.(json|yaml)
     for local_type in ("devices", "virtual_machines"):
+        if os.path.isdir(os.path.join(repository_record.filesystem_path, local_type)):
+            job_result.log(
+                f'Found "{local_type}" directory in the repository root. If this is meant to contain config contexts, '
+                "it should be moved into a `config_contexts/` subdirectory.",
+                level_choice=LogLevelChoices.LOG_WARNING,
+                grouping="config contexts",
+                logger=logger,
+            )
+
         dir_path = os.path.join(config_context_path, local_type)
         if not os.path.isdir(dir_path):
             continue
@@ -461,10 +471,7 @@ def update_git_config_contexts(repository_record, job_result):
             )
             try:
                 with open(os.path.join(dir_path, file_name), "r") as fd:
-                    try:
-                        context_data = yaml.safe_load(fd)
-                    except Exception as exc:
-                        raise RuntimeError(f"Error in loading local config context from `{file_name}`: {exc}")
+                    context_data = yaml.safe_load(fd)
 
                 import_local_config_context(
                     local_type,
@@ -477,7 +484,7 @@ def update_git_config_contexts(repository_record, job_result):
                 managed_local_config_contexts[local_type].add(device_name)
             except Exception as exc:
                 job_result.log(
-                    str(exc),
+                    f"Error in loading local config context from `{local_type}/{file_name}`: {exc}",
                     level_choice=LogLevelChoices.LOG_FAILURE,
                     grouping="local config contexts",
                     logger=logger,
@@ -509,8 +516,13 @@ def import_config_context(context_data, repository_record, job_result, logger):
     context_record = None
     # TODO: check context_data against a schema of some sort?
 
+    if "_metadata" not in context_data:
+        raise RuntimeError("data is missing the required `_metadata` key.")
+    if "name" not in context_data["_metadata"]:
+        raise RuntimeError("data `_metadata` is missing the required `name` key.")
+
     # Set defaults for optional fields
-    context_metadata = context_data.setdefault("_metadata", {})
+    context_metadata = context_data["_metadata"]
     context_metadata.setdefault("weight", 1000)
     context_metadata.setdefault("description", "")
     context_metadata.setdefault("is_active", True)
@@ -658,21 +670,9 @@ def import_local_config_context(local_type, device_name, context_data, repositor
     except MultipleObjectsReturned:
         # Possible for Device as name is not guaranteed globally unique
         # TODO: come up with a design that accounts for non-unique names, as well as un-named Devices.
-        job_result.log(
-            "Multiple records with the same name found; unable to determine which one to apply to!",
-            level_choice=LogLevelChoices.LOG_FAILURE,
-            grouping="local config contexts",
-            logger=logger,
-        )
-        return
+        raise RuntimeError("multiple records with the same name found; unable to determine which one to apply to!")
     except ObjectDoesNotExist:
-        job_result.log(
-            "Record not found!",
-            level_choice=LogLevelChoices.LOG_FAILURE,
-            grouping="local config contexts",
-            logger=logger,
-        )
-        return
+        raise RuntimeError("record not found!")
 
     if record.local_context_data_owner is not None and record.local_context_data_owner != repository_record:
         job_result.log(
@@ -780,10 +780,7 @@ def update_git_config_context_schemas(repository_record, job_result):
         try:
             with open(os.path.join(config_context_schema_path, file_name), "r") as fd:
                 # The data file can be either JSON or YAML; since YAML is a superset of JSON, we can load it regardless
-                try:
-                    context_schema_data = yaml.safe_load(fd)
-                except Exception as exc:
-                    raise RuntimeError(f"Error in loading config context schema data from `{file_name}`: {exc}")
+                context_schema_data = yaml.safe_load(fd)
 
             # A file can contain one config context dict or a list thereof
             if isinstance(context_schema_data, dict):
@@ -797,16 +794,12 @@ def update_git_config_context_schemas(repository_record, job_result):
                         )
                         managed_config_context_schemas.add(context_name)
                     else:
-                        raise RuntimeError(
-                            f"Error in loading config context schema data from `{file_name}`: data must be a dict or list of dicts"
-                        )
+                        raise RuntimeError("each item in list data must be a dict")
             else:
-                raise RuntimeError(
-                    f"Error in loading config context schema data from `{file_name}`: data must be a dict or list of dicts"
-                )
+                raise RuntimeError("data must be a dict or a list of dicts")
         except Exception as exc:
             job_result.log(
-                str(exc),
+                f"Error in loading config context schema data from `{file_name}`: {exc}",
                 level_choice=LogLevelChoices.LOG_FAILURE,
                 grouping="config context schemas",
                 logger=logger,
@@ -828,10 +821,12 @@ def import_config_context_schema(context_schema_data, repository_record, job_res
     created = False
     modified = False
 
-    schema_metadata = context_schema_data.setdefault("_metadata", {})
+    if "_metadata" not in context_schema_data:
+        raise RuntimeError("data is missing the required `_metadata` key.")
+    if "name" not in context_schema_data["_metadata"]:
+        raise RuntimeError("data `_metadata` is missing the required `name` key.")
 
-    if not schema_metadata.get("name"):
-        raise RuntimeError("File has no name set.")
+    schema_metadata = context_schema_data["_metadata"]
 
     try:
         schema_record = ConfigContextSchema.objects.get(
@@ -983,6 +978,18 @@ def update_git_export_templates(repository_record, job_result):
 
     Templates are located in GIT_ROOT/<repo>/export_templates/<app_label>/<model>/<template name>.
     """
+    # Error checking - did the user put directories in the repository root instead of under /export_templates/?
+    for app_label in ["circuits", "dcim", "extras", "ipam", "tenancy", "users", "virtualization"]:
+        unexpected_path = os.path.join(repository_record.filesystem_path, app_label)
+        if os.path.isdir(unexpected_path):
+            job_result.log(
+                f'Found "{app_label}" directory in the repository root. If this is meant to contain export templates, '
+                "it should be moved into an `export_templates/` subdirectory.",
+                level_choice=LogLevelChoices.LOG_WARNING,
+                grouping="export templates",
+                logger=logger,
+            )
+
     export_template_path = os.path.join(repository_record.filesystem_path, "export_templates")
     if not os.path.isdir(export_template_path):
         return
diff --git a/nautobot/extras/datasources/utils.py b/nautobot/extras/datasources/utils.py
--- a/nautobot/extras/datasources/utils.py
+++ b/nautobot/extras/datasources/utils.py
@@ -31,7 +31,7 @@ def files_from_contenttype_directories(base_path, job_result, log_grouping):
             except ContentType.DoesNotExist:
                 job_result.log(
                     f"Skipping `{app_label}.{modelname}` as it isn't a known content type",
-                    level_choice=LogLevelChoices.LOG_FAILURE,
+                    level_choice=LogLevelChoices.LOG_WARNING,
                     grouping=log_grouping,
                     logger=logger,
                 )


Test patch:
diff --git a/nautobot/extras/tests/test_datasources.py b/nautobot/extras/tests/test_datasources.py
--- a/nautobot/extras/tests/test_datasources.py
+++ b/nautobot/extras/tests/test_datasources.py
@@ -17,6 +17,7 @@
     SecretsGroupAccessTypeChoices,
     SecretsGroupSecretTypeChoices,
 )
+from nautobot.extras.choices import LogLevelChoices
 from nautobot.extras.datasources.git import pull_git_repository_and_refresh_data, git_repository_diff_origin_and_local
 from nautobot.extras.datasources.registry import get_datasource_contents
 from nautobot.extras.management import create_custom_statuses
@@ -25,6 +26,7 @@
     ConfigContextSchema,
     ExportTemplate,
     GitRepository,
+    JobLogEntry,
     JobResult,
     Secret,
     SecretsGroup,
@@ -510,45 +512,30 @@ def populate_repo(path, url):
                     os.makedirs(os.path.join(path, "config_context_schemas"))
                     os.makedirs(os.path.join(path, "export_templates", "nosuchapp", "device"))
                     os.makedirs(os.path.join(path, "export_templates", "dcim", "nosuchmodel"))
+                    # Incorrect directories
+                    os.makedirs(os.path.join(path, "devices"))
+                    os.makedirs(os.path.join(path, "dcim"))
                     # Malformed JSON
                     with open(os.path.join(path, "config_contexts", "context.json"), "w") as fd:
                         fd.write('{"data": ')
                     # Valid JSON but missing required keys
                     with open(os.path.join(path, "config_contexts", "context2.json"), "w") as fd:
                         fd.write("{}")
+                    with open(os.path.join(path, "config_contexts", "context3.json"), "w") as fd:
+                        fd.write('{"_metadata": {}}')
+                    # Malformed JSON
                     with open(os.path.join(path, "config_context_schemas", "schema-1.yaml"), "w") as fd:
                         fd.write('{"data": ')
                     # Valid JSON but missing required keys
                     with open(os.path.join(path, "config_context_schemas", "schema-2.yaml"), "w") as fd:
                         fd.write("{}")
                     # No such device
-                    with open(
-                        os.path.join(path, "config_contexts", "devices", "nosuchdevice.json"),
-                        "w",
-                    ) as fd:
+                    with open(os.path.join(path, "config_contexts", "devices", "nosuchdevice.json"), "w") as fd:
                         fd.write("{}")
                     # Invalid paths
-                    with open(
-                        os.path.join(
-                            path,
-                            "export_templates",
-                            "nosuchapp",
-                            "device",
-                            "template.j2",
-                        ),
-                        "w",
-                    ) as fd:
+                    with open(os.path.join(path, "export_templates", "nosuchapp", "device", "template.j2"), "w") as fd:
                         fd.write("{% for device in queryset %}\n{{ device.name }}\n{% endfor %}")
-                    with open(
-                        os.path.join(
-                            path,
-                            "export_templates",
-                            "dcim",
-                            "nosuchmodel",
-                            "template.j2",
-                        ),
-                        "w",
-                    ) as fd:
+                    with open(os.path.join(path, "export_templates", "dcim", "nosuchmodel", "template.j2"), "w") as fd:
                         fd.write("{% for device in queryset %}\n{{ device.name }}\n{% endfor %}")
                     return mock.DEFAULT
 
@@ -565,6 +552,56 @@ def populate_repo(path, url):
                     self.job_result.data,
                 )
 
+                # Check for specific log messages
+                log_entries = JobLogEntry.objects.filter(job_result=self.job_result)
+                warning_logs = log_entries.filter(log_level=LogLevelChoices.LOG_WARNING)
+                failure_logs = log_entries.filter(log_level=LogLevelChoices.LOG_FAILURE)
+
+                warning_logs.get(
+                    grouping="config contexts", message__contains='Found "devices" directory in the repository root'
+                )
+                warning_logs.get(grouping="jobs", message__contains="No `jobs` subdirectory found")
+                warning_logs.get(
+                    grouping="export templates", message__contains='Found "dcim" directory in the repository root'
+                )
+                warning_logs.get(
+                    grouping="export templates",
+                    message__contains="Skipping `dcim.nosuchmodel` as it isn't a known content type",
+                )
+                warning_logs.get(
+                    grouping="export templates",
+                    message__contains="Skipping `nosuchapp.device` as it isn't a known content type",
+                )
+
+                failure_logs.get(
+                    grouping="config context schemas",
+                    message__contains="Error in loading config context schema data from `schema-1.yaml`",
+                )
+                failure_logs.get(
+                    grouping="config context schemas",
+                    message__contains="Error in loading config context schema data from `schema-2.yaml`: "
+                    "data is missing the required `_metadata` key",
+                )
+                failure_logs.get(
+                    grouping="config contexts",
+                    message__contains="Error in loading config context data from `context.json`",
+                )
+                failure_logs.get(
+                    grouping="config contexts",
+                    message__contains="Error in loading config context data from `context2.json`: "
+                    "data is missing the required `_metadata` key",
+                )
+                failure_logs.get(
+                    grouping="config contexts",
+                    message__contains="Error in loading config context data from `context3.json`: "
+                    "data `_metadata` is missing the required `name` key",
+                )
+                failure_logs.get(
+                    grouping="local config contexts",
+                    message__contains="Error in loading local config context from `devices/nosuchdevice.json`: "
+                    "record not found",
+                )
+
     def test_delete_git_repository_cleanup(self, MockGitRepo):
         """
         When deleting a GitRepository record, the data that it owned should also be deleted.