Repository: nautobot/nautobot

Issue:
Signal error to the GUI if a syntactically incorrect job is loaded from a git repository
### Environment
Using [nautobot-lab](https://github.com/nautobot/nautobot-lab) at 1.0.0b1
* Python version: 3.8.5
* Nautobot version: 1385ef2621ef (v1.0.0b2)

### Proposed Functionality

If there is a syntactical error in a job file in a git repository there should be a warning or an error message in the sync job logs that indicates this.

### Use Case

If someone was to write a job in a git repository and for example miss an import they wouldn't know if the error lies
- in the communication between the repository and the Nautobot instance
- in the job file itself
- in the layout of the repository
- some other place

### Database Changes

None

### External Dependencies

None


Discussion:
Hello @Kircheneer, thanks for the submission.

What is the current behavior you're experiencing? It sounds like if a job is attempting to be loaded, it is failing silently or in a non-obvious way? Can you please add some context on your experience? Thanks in advance!
I just did the following things:
1. Reset the lab container
2. Copy the example job (the new branch one) from the documentation in a git repository into a `jobs` folder under the repository root (that folder also contains an `__init__.py` file
3. Run the job (working as expected)
4. Change the `description` field in the `Meta` class to `descriptio`
5. Refresh the data source in Nautobot
6. Watch as the job silently disappears without an error message
7. Change it back to `description`
8. Refresh the data soruce in Nautobot
9. The job is still not visible

Is there maybe some caching at play here which I am missing?
@glennmatthews Can you please help me confirm this one? :)
I am unable to reproduce this on Nautobot v1.0.0b2. The job appears to still load but the `description` field disappears. 

Can you please [build and install `nautobot-lab` from GitHub ](https://github.com/nautobot/nautobot-lab#building-the-container)which will build b2 and see if this issue persists? 

Thanks!
Tagging @Kircheneer to see @jathanism's last comment! 
That leads to a new error (when copying the default job from the docs again):
```python
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/nautobot/extras/jobs.py", line 826, in _run_job
    output = job.run(data=data, commit=commit)
  File "/root/.nautobot/git/test/jobs/example.py", line 37, in run
    site = Site(
  File "/usr/local/lib/python3.8/dist-packages/django/db/models/base.py", line 483, in __init__
    _setattr(self, field.name, rel_obj)
  File "/usr/local/lib/python3.8/dist-packages/django/db/models/fields/related_descriptors.py", line 215, in __set__
    raise ValueError(
ValueError: Cannot assign "'planned'": "Site.status" must be a "Status" instance.
```

Leaving that aside because it is not relevant to this issue (EDIT: #186), I have created [this](https://github.com/Kircheneer/nautobot_data) public repository. Try the "master" and the "test" branch in there. For me, the job from the "master" branch exhibits the behavior mentioned above, while the (faulty, runs with errors) script I copied from the docs in the "test" branch is recognized without issues. This is using `nautobot-lab` at the following commit:

```
commit 45a7bad52609263fca0c4c71e90a0ac0ea6986a3 (HEAD -> main, origin/main, origin/HEAD)
Merge: 0ed1147 978f9f9
Author: James Williams <amascuba@gmail.com>
Date:   Mon Mar 15 12:16:15 2021 -0500

    Merge pull request #19 from jtdub/superuser

    Populate Nautobot Lab with Sandbox Data
```
@Kircheneer That's progress! It does look like the `NewBranch` job you linked was based off the [sample job of the same name](https://nautobot.readthedocs.io/en/latest/additional-features/jobs/#creating-objects-for-a-planned-site).

In any case your issue is on line 41:

https://github.com/Kircheneer/nautobot_data/blob/master/jobs/example.py#L41

You want that to be an instance of a `Status` object now and not a choice enum. So you need these changes

```python
# at the top
from nautobot.extras.models import Status

# one line 41
                status=Status.objects.get(slug="planned"),
```

Can you please see if updating your job fixes your issue?
I can verify the reported behavior using the "master" branch of the linked repo - the job fails (from the user's perspective) silently to load when accessing the Jobs UI, although the error *is* logged to the console at least:

```
21:00:27.643 ERROR   nautobot.jobs        jobs.py                               get_jobs() :
  Unable to load job example: f-string: expecting '}' (example.py, line 40)
```

It would be reasonable for us to add some handling logic to report errors as a banner on the Jobs page, along the lines of "The following job files could not be loaded...". Thanks for the report!
Needs to be retested for 1.3/1.4 as symptoms may have changed with concrete Job model.

Resolving patch:
diff --git a/nautobot/extras/datasources/git.py b/nautobot/extras/datasources/git.py
--- a/nautobot/extras/datasources/git.py
+++ b/nautobot/extras/datasources/git.py
@@ -916,7 +916,16 @@ def refresh_git_jobs(repository_record, job_result, delete=False):
     if not delete:
         jobs_path = os.path.join(repository_record.filesystem_path, "jobs")
         if os.path.isdir(jobs_path):
-            for job_info in jobs_in_directory(jobs_path):
+            for job_info in jobs_in_directory(jobs_path, report_errors=True):
+                if job_info.error is not None:
+                    job_result.log(
+                        message=f"Error in loading Jobs from `{job_info.module_name}`: `{job_info.error}`",
+                        grouping="jobs",
+                        level_choice=LogLevelChoices.LOG_FAILURE,
+                        logger=logger,
+                    )
+                    continue
+
                 job_model, created = refresh_job_model_from_job_class(
                     Job,
                     JobSourceChoices.SOURCE_GIT,
diff --git a/nautobot/extras/utils.py b/nautobot/extras/utils.py
--- a/nautobot/extras/utils.py
+++ b/nautobot/extras/utils.py
@@ -196,11 +196,16 @@ def get_worker_count(request=None):
 
 
 # namedtuple class yielded by the jobs_in_directory generator function, below
-# Example: ("devices", <module "devices">, "Hostname", <class "devices.Hostname">)
-JobClassInfo = collections.namedtuple("JobClassInfo", ["module_name", "module", "job_class_name", "job_class"])
+# Example: ("devices", <module "devices">, "Hostname", <class "devices.Hostname">, None)
+# Example: ("devices", None, None, None, "error at line 40")
+JobClassInfo = collections.namedtuple(
+    "JobClassInfo",
+    ["module_name", "module", "job_class_name", "job_class", "error"],
+    defaults=(None, None, None, None),  # all parameters except `module_name` are optional and default to None.
+)
 
 
-def jobs_in_directory(path, module_name=None, reload_modules=True):
+def jobs_in_directory(path, module_name=None, reload_modules=True, report_errors=False):
     """
     Walk the available Python modules in the given directory, and for each module, walk its Job class members.
 
@@ -208,9 +213,11 @@ def jobs_in_directory(path, module_name=None, reload_modules=True):
         path (str): Directory to import modules from, outside of sys.path
         module_name (str): Specific module name to select; if unspecified, all modules will be inspected
         reload_modules (bool): Whether to force reloading of modules even if previously loaded into Python.
+        report_errors (bool): If True, when an error is encountered, yield a JobClassInfo with the given error.
+                              If False (default), log the error but do not yield anything.
 
     Yields:
-        JobClassInfo: (module_name, module, job_class_name, job_class)
+        JobClassInfo: (module_name, module, job_class_name, job_class, error)
     """
     from .jobs import is_job  # avoid circular import
 
@@ -221,13 +228,13 @@ def jobs_in_directory(path, module_name=None, reload_modules=True):
             del sys.modules[discovered_module_name]
         try:
             module = importer.find_module(discovered_module_name).load_module(discovered_module_name)
+            # Get all members of the module that are Job subclasses
+            for job_class_name, job_class in inspect.getmembers(module, is_job):
+                yield JobClassInfo(discovered_module_name, module, job_class_name, job_class)
         except Exception as exc:
-            logger.error(f"Unable to load module {module_name} from {path}: {exc}")
-            # TODO: we want to be able to report these errors to the UI in some fashion?
-            continue
-        # Get all members of the module that are Job subclasses
-        for job_class_name, job_class in inspect.getmembers(module, is_job):
-            yield JobClassInfo(discovered_module_name, module, job_class_name, job_class)
+            logger.error(f"Unable to load module {discovered_module_name} from {path}: {exc}")
+            if report_errors:
+                yield JobClassInfo(module_name=discovered_module_name, error=exc)
 
 
 def refresh_job_model_from_job_class(job_model_class, job_source, job_class, *, git_repository=None):


Test patch:
diff --git a/nautobot/extras/tests/test_datasources.py b/nautobot/extras/tests/test_datasources.py
--- a/nautobot/extras/tests/test_datasources.py
+++ b/nautobot/extras/tests/test_datasources.py
@@ -117,7 +117,7 @@ def setUp(self):
 
     def populate_repo(self, path, url, *args, **kwargs):
         os.makedirs(path)
-        # Just make config_contexts and export_templates directories as we don't load jobs
+        # TODO: populate Jobs as well?
         os.makedirs(os.path.join(path, "config_contexts"))
         os.makedirs(os.path.join(path, "config_contexts", "devices"))
         os.makedirs(os.path.join(path, "config_context_schemas"))
@@ -266,7 +266,10 @@ def create_empty_repo(path, url):
                 self.repo.refresh_from_db()
                 self.assertEqual(self.repo.current_head, self.COMMIT_HEXSHA, self.job_result.data)
                 MockGitRepo.assert_called_with(os.path.join(tempdir, self.repo.slug), "http://localhost/git.git")
-                # TODO: inspect the logs in job_result.data?
+
+                log_entries = JobLogEntry.objects.filter(job_result=self.job_result)
+                warning_logs = log_entries.filter(log_level=LogLevelChoices.LOG_WARNING)
+                warning_logs.get(grouping="jobs", message__contains="No `jobs` subdirectory found")
 
     def test_pull_git_repository_and_refresh_data_with_token(self, MockGitRepo):
         """
@@ -506,12 +509,12 @@ def test_pull_git_repository_and_refresh_data_with_bad_data(self, MockGitRepo):
 
                 def populate_repo(path, url):
                     os.makedirs(path)
-                    # Just make config_contexts and export_templates directories as we don't load jobs
                     os.makedirs(os.path.join(path, "config_contexts"))
                     os.makedirs(os.path.join(path, "config_contexts", "devices"))
                     os.makedirs(os.path.join(path, "config_context_schemas"))
                     os.makedirs(os.path.join(path, "export_templates", "nosuchapp", "device"))
                     os.makedirs(os.path.join(path, "export_templates", "dcim", "nosuchmodel"))
+                    os.makedirs(os.path.join(path, "jobs"))
                     # Incorrect directories
                     os.makedirs(os.path.join(path, "devices"))
                     os.makedirs(os.path.join(path, "dcim"))
@@ -537,6 +540,11 @@ def populate_repo(path, url):
                         fd.write("{% for device in queryset %}\n{{ device.name }}\n{% endfor %}")
                     with open(os.path.join(path, "export_templates", "dcim", "nosuchmodel", "template.j2"), "w") as fd:
                         fd.write("{% for device in queryset %}\n{{ device.name }}\n{% endfor %}")
+                    # Malformed Python
+                    with open(os.path.join(path, "jobs", "syntaxerror.py"), "w") as fd:
+                        fd.write("print(")
+                    with open(os.path.join(path, "jobs", "importerror.py"), "w") as fd:
+                        fd.write("import nosuchmodule")
                     return mock.DEFAULT
 
                 MockGitRepo.side_effect = populate_repo
@@ -560,7 +568,6 @@ def populate_repo(path, url):
                 warning_logs.get(
                     grouping="config contexts", message__contains='Found "devices" directory in the repository root'
                 )
-                warning_logs.get(grouping="jobs", message__contains="No `jobs` subdirectory found")
                 warning_logs.get(
                     grouping="export templates", message__contains='Found "dcim" directory in the repository root'
                 )
@@ -601,6 +608,15 @@ def populate_repo(path, url):
                     message__contains="Error in loading local config context from `devices/nosuchdevice.json`: "
                     "record not found",
                 )
+                failure_logs.get(
+                    grouping="jobs",
+                    # The specific exception message differs between Python versions
+                    message__contains="Error in loading Jobs from `syntaxerror`: ",
+                )
+                failure_logs.get(
+                    grouping="jobs",
+                    message__contains="Error in loading Jobs from `importerror`: `No module named 'nosuchmodule'`",
+                )
 
     def test_delete_git_repository_cleanup(self, MockGitRepo):
         """