Repository: nautobot/nautobot

Issue:
`invoke markdownlint` does not run in the docker environment
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Nautobot version (Docker tag too if applicable): 2.0.0-b1
* Python version: 3.8.17
* Database platform, version:
* Middleware(s):

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. Run `invoke markdownlint` in the docker dev environment

<!-- What did you expect to happen? -->
### Expected Behavior
Markdownlint runs

<!-- What happened instead? -->
### Observed Behavior
The command errors out with:

```
npm ERR! could not determine executable to run

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-07-05T21_28_35_823Z-debug-0.log
```

The contents of the log:

```
0 verbose cli /usr/bin/node /usr/bin/npm
1 info using npm@9.5.1
2 info using node@v18.16.1
3 timing npm:load:whichnode Completed in 2ms
4 timing config:load:defaults Completed in 2ms
5 timing config:load:file:/usr/lib/node_modules/npm/npmrc Completed in 0ms
6 timing config:load:builtin Completed in 0ms
7 timing config:load:cli Completed in 4ms
8 timing config:load:env Completed in 0ms
9 timing config:load:file:/source/.npmrc Completed in 0ms
10 timing config:load:project Completed in 1ms
11 timing config:load:file:/root/.npmrc Completed in 0ms
12 timing config:load:user Completed in 0ms
13 timing config:load:file:/usr/etc/npmrc Completed in 0ms
14 timing config:load:global Completed in 0ms
15 timing config:load:setEnvs Completed in 1ms
16 timing config:load Completed in 9ms
17 timing npm:load:configload Completed in 9ms
18 timing npm:load:mkdirpcache Completed in 0ms
19 timing npm:load:mkdirplogs Completed in 1ms
20 verbose title npm exec markdownlint --ignore nautobot/project-static --ignore nautobot/ui/node_modules --config .markdownlint.yml --rules scripts/use-relative-md-links.js nautobot examples CHANGELOG.md CODE_OF_CONDUCT.md CONTRIBUTING.md README.md SECURITY.md
21 verbose argv "exec" "--" "markdownlint" "--ignore" "nautobot/project-static" "--ignore" "nautobot/ui/node_modules" "--config" ".markdownlint.yml" "--rules" "scripts/use-relative-md-links.js" "nautobot" "examples" "CHANGELOG.md" "CODE_OF_CONDUCT.md" "CONTRIBUTING.md" "README.md" "SECURITY.md"
22 timing npm:load:setTitle Completed in 1ms
23 timing config:load:flatten Completed in 3ms
24 timing npm:load:display Completed in 3ms
25 verbose logfile logs-max:10 dir:/root/.npm/_logs/2023-07-05T21_28_35_823Z-
26 verbose logfile /root/.npm/_logs/2023-07-05T21_28_35_823Z-debug-0.log
27 timing npm:load:logFile Completed in 4ms
28 timing npm:load:timers Completed in 0ms
29 timing npm:load:configScope Completed in 0ms
30 timing npm:load Completed in 21ms
31 silly logfile done cleaning log files
32 timing arborist:ctor Completed in 1ms
33 http fetch GET 200 https://registry.npmjs.org/markdownlint 222ms (cache miss)
34 timing command:exec Completed in 243ms
35 verbose stack Error: could not determine executable to run
35 verbose stack     at getBinFromManifest (/usr/lib/node_modules/npm/node_modules/libnpmexec/lib/get-bin-from-manifest.js:17:23)
35 verbose stack     at exec (/usr/lib/node_modules/npm/node_modules/libnpmexec/lib/index.js:185:15)
35 verbose stack     at async module.exports (/usr/lib/node_modules/npm/lib/cli.js:134:5)
36 verbose pkgid markdownlint@0.29.0
37 verbose cwd /source
38 verbose Linux 6.1.0-9-amd64
39 verbose node v18.16.1
40 verbose npm  v9.5.1
41 error could not determine executable to run
42 verbose exit 1
43 timing npm Completed in 275ms
44 verbose code 1
45 error A complete log of this run can be found in:
45 error     /root/.npm/_logs/2023-07-05T21_28_35_823Z-debug-0.log
```

The npm dependencies are installed in `/source/` in the dockerfile `dependencies-dev` stage but:
1. Immediately after they're installed `/source/` is overwritten
2. `/source/` gets bound from the host filesystem so anything that was in that directory in the container image is lost

Workaround until the bug is fixed:

`inv cli`
`npm install` from within the container will install the dependencies to the locally bound filesystem
`invoke markdownlint` does not run in the docker environment
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Nautobot version (Docker tag too if applicable): 2.0.0-b1
* Python version: 3.8.17
* Database platform, version:
* Middleware(s):

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. Run `invoke markdownlint` in the docker dev environment

<!-- What did you expect to happen? -->
### Expected Behavior
Markdownlint runs

<!-- What happened instead? -->
### Observed Behavior
The command errors out with:

```
npm ERR! could not determine executable to run

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-07-05T21_28_35_823Z-debug-0.log
```

The contents of the log:

```
0 verbose cli /usr/bin/node /usr/bin/npm
1 info using npm@9.5.1
2 info using node@v18.16.1
3 timing npm:load:whichnode Completed in 2ms
4 timing config:load:defaults Completed in 2ms
5 timing config:load:file:/usr/lib/node_modules/npm/npmrc Completed in 0ms
6 timing config:load:builtin Completed in 0ms
7 timing config:load:cli Completed in 4ms
8 timing config:load:env Completed in 0ms
9 timing config:load:file:/source/.npmrc Completed in 0ms
10 timing config:load:project Completed in 1ms
11 timing config:load:file:/root/.npmrc Completed in 0ms
12 timing config:load:user Completed in 0ms
13 timing config:load:file:/usr/etc/npmrc Completed in 0ms
14 timing config:load:global Completed in 0ms
15 timing config:load:setEnvs Completed in 1ms
16 timing config:load Completed in 9ms
17 timing npm:load:configload Completed in 9ms
18 timing npm:load:mkdirpcache Completed in 0ms
19 timing npm:load:mkdirplogs Completed in 1ms
20 verbose title npm exec markdownlint --ignore nautobot/project-static --ignore nautobot/ui/node_modules --config .markdownlint.yml --rules scripts/use-relative-md-links.js nautobot examples CHANGELOG.md CODE_OF_CONDUCT.md CONTRIBUTING.md README.md SECURITY.md
21 verbose argv "exec" "--" "markdownlint" "--ignore" "nautobot/project-static" "--ignore" "nautobot/ui/node_modules" "--config" ".markdownlint.yml" "--rules" "scripts/use-relative-md-links.js" "nautobot" "examples" "CHANGELOG.md" "CODE_OF_CONDUCT.md" "CONTRIBUTING.md" "README.md" "SECURITY.md"
22 timing npm:load:setTitle Completed in 1ms
23 timing config:load:flatten Completed in 3ms
24 timing npm:load:display Completed in 3ms
25 verbose logfile logs-max:10 dir:/root/.npm/_logs/2023-07-05T21_28_35_823Z-
26 verbose logfile /root/.npm/_logs/2023-07-05T21_28_35_823Z-debug-0.log
27 timing npm:load:logFile Completed in 4ms
28 timing npm:load:timers Completed in 0ms
29 timing npm:load:configScope Completed in 0ms
30 timing npm:load Completed in 21ms
31 silly logfile done cleaning log files
32 timing arborist:ctor Completed in 1ms
33 http fetch GET 200 https://registry.npmjs.org/markdownlint 222ms (cache miss)
34 timing command:exec Completed in 243ms
35 verbose stack Error: could not determine executable to run
35 verbose stack     at getBinFromManifest (/usr/lib/node_modules/npm/node_modules/libnpmexec/lib/get-bin-from-manifest.js:17:23)
35 verbose stack     at exec (/usr/lib/node_modules/npm/node_modules/libnpmexec/lib/index.js:185:15)
35 verbose stack     at async module.exports (/usr/lib/node_modules/npm/lib/cli.js:134:5)
36 verbose pkgid markdownlint@0.29.0
37 verbose cwd /source
38 verbose Linux 6.1.0-9-amd64
39 verbose node v18.16.1
40 verbose npm  v9.5.1
41 error could not determine executable to run
42 verbose exit 1
43 timing npm Completed in 275ms
44 verbose code 1
45 error A complete log of this run can be found in:
45 error     /root/.npm/_logs/2023-07-05T21_28_35_823Z-debug-0.log
```

The npm dependencies are installed in `/source/` in the dockerfile `dependencies-dev` stage but:
1. Immediately after they're installed `/source/` is overwritten
2. `/source/` gets bound from the host filesystem so anything that was in that directory in the container image is lost

Workaround until the bug is fixed:

`inv cli`
`npm install` from within the container will install the dependencies to the locally bound filesystem
`invoke markdownlint` does not run in the docker environment
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Nautobot version (Docker tag too if applicable): 2.0.0-b1
* Python version: 3.8.17
* Database platform, version:
* Middleware(s):

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. Run `invoke markdownlint` in the docker dev environment

<!-- What did you expect to happen? -->
### Expected Behavior
Markdownlint runs

<!-- What happened instead? -->
### Observed Behavior
The command errors out with:

```
npm ERR! could not determine executable to run

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-07-05T21_28_35_823Z-debug-0.log
```

The contents of the log:

```
0 verbose cli /usr/bin/node /usr/bin/npm
1 info using npm@9.5.1
2 info using node@v18.16.1
3 timing npm:load:whichnode Completed in 2ms
4 timing config:load:defaults Completed in 2ms
5 timing config:load:file:/usr/lib/node_modules/npm/npmrc Completed in 0ms
6 timing config:load:builtin Completed in 0ms
7 timing config:load:cli Completed in 4ms
8 timing config:load:env Completed in 0ms
9 timing config:load:file:/source/.npmrc Completed in 0ms
10 timing config:load:project Completed in 1ms
11 timing config:load:file:/root/.npmrc Completed in 0ms
12 timing config:load:user Completed in 0ms
13 timing config:load:file:/usr/etc/npmrc Completed in 0ms
14 timing config:load:global Completed in 0ms
15 timing config:load:setEnvs Completed in 1ms
16 timing config:load Completed in 9ms
17 timing npm:load:configload Completed in 9ms
18 timing npm:load:mkdirpcache Completed in 0ms
19 timing npm:load:mkdirplogs Completed in 1ms
20 verbose title npm exec markdownlint --ignore nautobot/project-static --ignore nautobot/ui/node_modules --config .markdownlint.yml --rules scripts/use-relative-md-links.js nautobot examples CHANGELOG.md CODE_OF_CONDUCT.md CONTRIBUTING.md README.md SECURITY.md
21 verbose argv "exec" "--" "markdownlint" "--ignore" "nautobot/project-static" "--ignore" "nautobot/ui/node_modules" "--config" ".markdownlint.yml" "--rules" "scripts/use-relative-md-links.js" "nautobot" "examples" "CHANGELOG.md" "CODE_OF_CONDUCT.md" "CONTRIBUTING.md" "README.md" "SECURITY.md"
22 timing npm:load:setTitle Completed in 1ms
23 timing config:load:flatten Completed in 3ms
24 timing npm:load:display Completed in 3ms
25 verbose logfile logs-max:10 dir:/root/.npm/_logs/2023-07-05T21_28_35_823Z-
26 verbose logfile /root/.npm/_logs/2023-07-05T21_28_35_823Z-debug-0.log
27 timing npm:load:logFile Completed in 4ms
28 timing npm:load:timers Completed in 0ms
29 timing npm:load:configScope Completed in 0ms
30 timing npm:load Completed in 21ms
31 silly logfile done cleaning log files
32 timing arborist:ctor Completed in 1ms
33 http fetch GET 200 https://registry.npmjs.org/markdownlint 222ms (cache miss)
34 timing command:exec Completed in 243ms
35 verbose stack Error: could not determine executable to run
35 verbose stack     at getBinFromManifest (/usr/lib/node_modules/npm/node_modules/libnpmexec/lib/get-bin-from-manifest.js:17:23)
35 verbose stack     at exec (/usr/lib/node_modules/npm/node_modules/libnpmexec/lib/index.js:185:15)
35 verbose stack     at async module.exports (/usr/lib/node_modules/npm/lib/cli.js:134:5)
36 verbose pkgid markdownlint@0.29.0
37 verbose cwd /source
38 verbose Linux 6.1.0-9-amd64
39 verbose node v18.16.1
40 verbose npm  v9.5.1
41 error could not determine executable to run
42 verbose exit 1
43 timing npm Completed in 275ms
44 verbose code 1
45 error A complete log of this run can be found in:
45 error     /root/.npm/_logs/2023-07-05T21_28_35_823Z-debug-0.log
```

The npm dependencies are installed in `/source/` in the dockerfile `dependencies-dev` stage but:
1. Immediately after they're installed `/source/` is overwritten
2. `/source/` gets bound from the host filesystem so anything that was in that directory in the container image is lost

Workaround until the bug is fixed:

`inv cli`
`npm install` from within the container will install the dependencies to the locally bound filesystem


Discussion:
Thanks for filing this Gary! I've noticed the same symptom.
Thanks for filing this Gary! I've noticed the same symptom.
Thanks for filing this Gary! I've noticed the same symptom.

Resolving patch:
diff --git a/development/nautobot_config.py b/development/nautobot_config.py
--- a/development/nautobot_config.py
+++ b/development/nautobot_config.py
@@ -4,11 +4,14 @@
 from nautobot.core.settings import *  # noqa: F403
 from nautobot.core.settings_funcs import is_truthy
 
+SECRET_KEY = os.getenv("NAUTOBOT_SECRET_KEY", "012345678901234567890123456789012345678901234567890123456789")
+
 #
 # Debugging defaults to True rather than False for the development environment
 #
 DEBUG = is_truthy(os.getenv("NAUTOBOT_DEBUG", "True"))
 
+
 # Django Debug Toolbar - enabled only when debugging
 if DEBUG:
     if "debug_toolbar" not in INSTALLED_APPS:  # noqa: F405
diff --git a/tasks.py b/tasks.py
--- a/tasks.py
+++ b/tasks.py
@@ -207,10 +207,9 @@ def run_command(context, command, service="nautobot", **kwargs):
         "cache": "Whether to use Docker's cache when building the image. (Default: enabled)",
         "poetry_parallel": "Enable/disable poetry to install packages in parallel. (Default: True)",
         "pull": "Whether to pull Docker images when building the image. (Default: disabled)",
-        "skip_docs_build": "Skip (re)build of documentation after building the image.",
     }
 )
-def build(context, force_rm=False, cache=True, poetry_parallel=True, pull=False, skip_docs_build=False):
+def build(context, force_rm=False, cache=True, poetry_parallel=True, pull=False):
     """Build Nautobot docker image."""
     command = f"build --build-arg PYTHON_VER={context.nautobot.python_ver}"
 
@@ -227,10 +226,6 @@ def build(context, force_rm=False, cache=True, poetry_parallel=True, pull=False,
 
     docker_compose(context, command, env={"DOCKER_BUILDKIT": "1", "COMPOSE_DOCKER_CLI_BUILD": "1"})
 
-    if not skip_docs_build:
-        # Build the docs so they are available. Skip if you're using a `final-dev` or `final` image instead of `dev`.
-        build_nautobot_docs(context)
-
 
 @task(
     help={
@@ -635,8 +630,23 @@ def hadolint(context):
 @task
 def markdownlint(context):
     """Lint Markdown files."""
-    command = "npm exec -- markdownlint --ignore nautobot/project-static --ignore nautobot/ui/node_modules --config .markdownlint.yml --rules scripts/use-relative-md-links.js nautobot examples *.md"
-    run_command(context, command)
+    if is_truthy(context.nautobot.local):
+        command = (
+            "cd nautobot/ui && npx -- markdownlint-cli "
+            "--ignore ../../nautobot/project-static --ignore ../../nautobot/ui/node_modules "
+            "--config ../../.markdownlint.yml --rules ../../scripts/use-relative-md-links.js "
+            "../../nautobot ../../examples ../../*.md"
+        )
+        run_command(context, command)
+    else:
+        command = (
+            "npx -- markdownlint-cli "
+            "--ignore /source/nautobot/project-static --ignore /source/nautobot/ui/node_modules "
+            "--config /source/.markdownlint.yml --rules /source/scripts/use-relative-md-links.js "
+            "/source/nautobot /source/examples /source/*.md"
+        )
+        docker_command = f"run --workdir='/opt/nautobot/ui' --entrypoint '{command}' nautobot"
+        docker_compose(context, docker_command, pty=True)
 
 
 @task
@@ -926,12 +936,12 @@ def prettier(context, autoformat=False):
         arg = "--check"
 
     if is_truthy(context.nautobot.local):
-        path = NAUTOBOT_UI_DIR
+        run_command(context, f"cd nautobot/ui && {prettier_command} {arg} .")
     else:
-        path = "."
-
-    command = f"{prettier_command} {arg} {path}"
-    run_command(context, command, service="nodejs")
+        docker_compose(
+            context,
+            f"run --workdir='/opt/nautobot/ui' --entrypoint '{prettier_command} {arg} /source/nautobot/ui' nautobot",
+        )
 
 
 @task(
@@ -947,12 +957,16 @@ def eslint(context, autoformat=False):
         eslint_command += " --fix"
 
     if is_truthy(context.nautobot.local):
-        path = NAUTOBOT_UI_DIR
+        run_command(context, f"cd nautobot/ui && {eslint_command} .")
     else:
-        path = "."
-
-    command = f"{eslint_command} {path}"
-    run_command(context, command, service="nodejs")
+        # TODO: we should really run against /source/nautobot/ui, not /opt/nautobot/ui, but eslint aborts if we do:
+        #   ESLint couldn't find the config "@react-app" to extend from.
+        #   Please check that the name of the config is correct.
+        # Probably this is because we don't install node_modules under /source/nautobot/ui normally...?
+        docker_compose(
+            context,
+            f"run --workdir='/opt/nautobot/ui' --entrypoint '{eslint_command} /opt/nautobot/ui' nautobot",
+        )
 
 
 @task(


Test patch: