Repository: nautobot/nautobot

Issue:
Path `./jsconfig.paths.json` does not exist preventing nodejs container from starting
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Nautobot version: 2.0.0 (next-[8413399](https://github.com/nautobot/nautobot/commit/84133994d6e2bd7566b71383142f0ae4e9458daf))
* Python version: 3.10.6
* Database platform, version: N/A
* Middleware(s): N/A

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. Clone the nautobot `next` branch (currently at commit [8413399](https://github.com/nautobot/nautobot/commit/84133994d6e2bd7566b71383142f0ae4e9458daf))
2. `invoke build debug`

<!-- What did you expect to happen? -->
### Expected Behavior
Nautobot development server / nodejs docker container starts successfully and can be reached at localhost:3000.

<!-- What happened instead? -->
### Observed Behavior
The Nautobot server on localhost:8080 starts successfully; however, the dev container on port 3000 does not. When trying to force start the nautobot-nodejs-1 container, this error is given: 
`react-app-alias:readConfig: '${confPath}' field 'extends' is '${extPath}' - not exists'`

Full debug log:
```
> nautobot@2.0.0-beta.2 start
nautobot-nodejs-1         | > craco start
nautobot-nodejs-1         | 
nautobot-nodejs-1         | /opt/nautobot/ui/node_modules/react-app-alias/src/index.js:147
nautobot-nodejs-1         |     throw Error(`react-app-alias:readConfig: '${confPath}' field 'extends' is '${extPath}' - not exists'`)
nautobot-nodejs-1         |           ^
nautobot-nodejs-1         | 
nautobot-nodejs-1         | Error: react-app-alias:readConfig: '/opt/nautobot/ui/jsconfig.json' field 'extends' is '/opt/nautobot/ui/jsconfig.paths.json' - not exists'
nautobot-nodejs-1         |     at readConfig (/opt/nautobot/ui/node_modules/react-app-alias/src/index.js:147:11)
nautobot-nodejs-1         |     at defaultOptions (/opt/nautobot/ui/node_modules/react-app-alias/src/index.js:189:16)
nautobot-nodejs-1         |     at aliasWebpack (/opt/nautobot/ui/node_modules/react-app-alias-ex/src/index.js:132:20)
nautobot-nodejs-1         |     at Object.overrideWebpackConfig (/opt/nautobot/ui/lib/nautobot-craco-alias-plugin.js:31:16)
nautobot-nodejs-1         |     at overrideWebpack (/opt/nautobot/ui/node_modules/@craco/craco/dist/lib/features/plugins.js:36:38)
nautobot-nodejs-1         |     at /opt/nautobot/ui/node_modules/@craco/craco/dist/lib/features/plugins.js:53:29
nautobot-nodejs-1         |     at Array.forEach (<anonymous>)
nautobot-nodejs-1         |     at applyWebpackConfigPlugins (/opt/nautobot/ui/node_modules/@craco/craco/dist/lib/features/plugins.js:52:29)
nautobot-nodejs-1         |     at mergeWebpackConfig (/opt/nautobot/ui/node_modules/@craco/craco/dist/lib/features/webpack/merge-webpack-config.js:119:70)
nautobot-nodejs-1         |     at overrideWebpackDev (/opt/nautobot/ui/node_modules/@craco/craco/dist/lib/features/webpack/override.js:8:80)
nautobot-nodejs-1         | 
nautobot-nodejs-1         | Node.js v18.16.1
nautobot-nodejs-1 exited with code 1
```

This path `./jsconfig.paths.json` within `/nautobot/ui/jsconfig.json` does not exist and should probably be something else. Full `jsconfig.json` file:
```
{
    "extends": "./jsconfig.paths.json",
    "compilerOptions": {},
    "exclude": ["node_modules", "build"]
}
```
Path `./jsconfig.paths.json` does not exist preventing nodejs container from starting
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Nautobot version: 2.0.0 (next-[8413399](https://github.com/nautobot/nautobot/commit/84133994d6e2bd7566b71383142f0ae4e9458daf))
* Python version: 3.10.6
* Database platform, version: N/A
* Middleware(s): N/A

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. Clone the nautobot `next` branch (currently at commit [8413399](https://github.com/nautobot/nautobot/commit/84133994d6e2bd7566b71383142f0ae4e9458daf))
2. `invoke build debug`

<!-- What did you expect to happen? -->
### Expected Behavior
Nautobot development server / nodejs docker container starts successfully and can be reached at localhost:3000.

<!-- What happened instead? -->
### Observed Behavior
The Nautobot server on localhost:8080 starts successfully; however, the dev container on port 3000 does not. When trying to force start the nautobot-nodejs-1 container, this error is given: 
`react-app-alias:readConfig: '${confPath}' field 'extends' is '${extPath}' - not exists'`

Full debug log:
```
> nautobot@2.0.0-beta.2 start
nautobot-nodejs-1         | > craco start
nautobot-nodejs-1         | 
nautobot-nodejs-1         | /opt/nautobot/ui/node_modules/react-app-alias/src/index.js:147
nautobot-nodejs-1         |     throw Error(`react-app-alias:readConfig: '${confPath}' field 'extends' is '${extPath}' - not exists'`)
nautobot-nodejs-1         |           ^
nautobot-nodejs-1         | 
nautobot-nodejs-1         | Error: react-app-alias:readConfig: '/opt/nautobot/ui/jsconfig.json' field 'extends' is '/opt/nautobot/ui/jsconfig.paths.json' - not exists'
nautobot-nodejs-1         |     at readConfig (/opt/nautobot/ui/node_modules/react-app-alias/src/index.js:147:11)
nautobot-nodejs-1         |     at defaultOptions (/opt/nautobot/ui/node_modules/react-app-alias/src/index.js:189:16)
nautobot-nodejs-1         |     at aliasWebpack (/opt/nautobot/ui/node_modules/react-app-alias-ex/src/index.js:132:20)
nautobot-nodejs-1         |     at Object.overrideWebpackConfig (/opt/nautobot/ui/lib/nautobot-craco-alias-plugin.js:31:16)
nautobot-nodejs-1         |     at overrideWebpack (/opt/nautobot/ui/node_modules/@craco/craco/dist/lib/features/plugins.js:36:38)
nautobot-nodejs-1         |     at /opt/nautobot/ui/node_modules/@craco/craco/dist/lib/features/plugins.js:53:29
nautobot-nodejs-1         |     at Array.forEach (<anonymous>)
nautobot-nodejs-1         |     at applyWebpackConfigPlugins (/opt/nautobot/ui/node_modules/@craco/craco/dist/lib/features/plugins.js:52:29)
nautobot-nodejs-1         |     at mergeWebpackConfig (/opt/nautobot/ui/node_modules/@craco/craco/dist/lib/features/webpack/merge-webpack-config.js:119:70)
nautobot-nodejs-1         |     at overrideWebpackDev (/opt/nautobot/ui/node_modules/@craco/craco/dist/lib/features/webpack/override.js:8:80)
nautobot-nodejs-1         | 
nautobot-nodejs-1         | Node.js v18.16.1
nautobot-nodejs-1 exited with code 1
```

This path `./jsconfig.paths.json` within `/nautobot/ui/jsconfig.json` does not exist and should probably be something else. Full `jsconfig.json` file:
```
{
    "extends": "./jsconfig.paths.json",
    "compilerOptions": {},
    "exclude": ["node_modules", "build"]
}
```


Discussion:
Should `/nautobot/ui/jsconfig.json` be:
```
{
    "extends": "./jsconfig-base.json",
    "compilerOptions": {},
    "exclude": ["node_modules", "build"]
}
```
instead of
```
{
    "extends": "./jsconfig.paths.json",
    "compilerOptions": {},
    "exclude": ["node_modules", "build"]
}
```
?
Darn! I could have sworn I fixed this in #4082. `jsconfig.paths.json` is supposed to be generated by the `nautobot-server build_ui` command that is run as part of the Docker build.
I missed this in #4082. The issue is that while the files are in `/opt/nautobot/ui/` in the base image, `docker-compose.dev.yml` mounts the local source code on top of this directory, clobbering those files. Additionally, since the image is running in `/opt/nautobot/ui/` rather than `/source/nautobot/ui/`, it won't automatically pick up changes to the source files and do live rebuilds now (which is the whole point of this container!). Mea culpa.

This is going to be tricky to fix since we can mount volumes for directories (e.g., preserving `/opt/nautobot/ui/node_modules/` within the image when we mount local `nautobot/ui/` onto `/opt/nautobot/ui/`, but not for files - we need some way to "merge" the container's `/opt/nautobot/ui/jsconfig.paths.json` into the volume mounted from localhost, ideally without having that file get exported back to the local filesystem since it should never be committed to git. Similar problem exists for the other two generated files `nautobot/ui/src/app_config.js` and `nautobot/ui/src/app_routes.json`.
Should `/nautobot/ui/jsconfig.json` be:
```
{
    "extends": "./jsconfig-base.json",
    "compilerOptions": {},
    "exclude": ["node_modules", "build"]
}
```
instead of
```
{
    "extends": "./jsconfig.paths.json",
    "compilerOptions": {},
    "exclude": ["node_modules", "build"]
}
```
?
Darn! I could have sworn I fixed this in #4082. `jsconfig.paths.json` is supposed to be generated by the `nautobot-server build_ui` command that is run as part of the Docker build.
I missed this in #4082. The issue is that while the files are in `/opt/nautobot/ui/` in the base image, `docker-compose.dev.yml` mounts the local source code on top of this directory, clobbering those files. Additionally, since the image is running in `/opt/nautobot/ui/` rather than `/source/nautobot/ui/`, it won't automatically pick up changes to the source files and do live rebuilds now (which is the whole point of this container!). Mea culpa.

This is going to be tricky to fix since we can mount volumes for directories (e.g., preserving `/opt/nautobot/ui/node_modules/` within the image when we mount local `nautobot/ui/` onto `/opt/nautobot/ui/`, but not for files - we need some way to "merge" the container's `/opt/nautobot/ui/jsconfig.paths.json` into the volume mounted from localhost, ideally without having that file get exported back to the local filesystem since it should never be committed to git. Similar problem exists for the other two generated files `nautobot/ui/src/app_config.js` and `nautobot/ui/src/app_routes.json`.

Resolving patch:
diff --git a/nautobot/core/management/commands/build_ui.py b/nautobot/core/management/commands/build_ui.py
--- a/nautobot/core/management/commands/build_ui.py
+++ b/nautobot/core/management/commands/build_ui.py
@@ -30,7 +30,7 @@ def add_arguments(self, parser):
             action="store_false",
             dest="render_apps",
             default=True,
-            help="Do not render Nautobot App imports.",
+            help="Do not (re)render Nautobot App imports.",
         )
         parser.add_argument(
             "--no-npm-build",
@@ -127,59 +127,86 @@ def render_routes_imports(self, app_base_path, app_name, app_config):
             # If an app does not include its url.py file or urls.py does not include a urlpatterns, skip.
             return data
 
-    def render_app_imports(self):
-        """Render `app_imports.js`, update `jsconfig.json` to map to the path for each and render `app_routes.json` to register all app routes."""
-        self.stdout.write(self.style.WARNING(">>> Rendering Nautobot App imports..."))
+    def render_app_imports(self, render_apps=False):
+        """
+        Render dynamically constructed app-related files - `app_imports.js`, `app_routes.json`, `jsconfig.paths.json`.
+
+        - If `render_apps` is True, these files will be regenerated according to the apps included in settings.PLUGINS.
+        - If `render_apps` is False but the files do not yet exist, they will be created but will not include any apps.
+        - If `render_apps` is False and the files already exist, they will be left untouched.
+        """
 
         ui_dir = settings.NAUTOBOT_UI_DIR
 
+        # Input files
         router_file_path = Path(ui_dir, "src", "router.js")
-        app_routes_file_path = Path(ui_dir, "src", "app_routes.json")
-        jsconfig_file_path = Path(ui_dir, "jsconfig.paths.json")
-        jsconfig_base_file_path = Path(ui_dir, "jsconfig-base.json")
-        app_routes = {}
+        jsconfig_base_file_path = Path(ui_dir, "src", "file_templates", "jsconfig-base.json")
+        environment = jinja2.sandbox.SandboxedEnvironment(
+            loader=jinja2.FileSystemLoader(Path(ui_dir, "src", "file_templates"))
+        )
+        template = environment.get_template("app_imports.js.j2")
+
+        # Output files
+        output_dir = Path(ui_dir, "generated")
+        os.makedirs(output_dir, exist_ok=True)
+        jsconfig_file_path = Path(output_dir, "jsconfig.paths.json")
+        app_routes_file_path = Path(output_dir, "app_routes.json")
+        app_imports_final_file_path = Path(output_dir, "app_imports.js")
 
+        app_routes = {}
         with open(jsconfig_base_file_path, "r", encoding="utf-8") as base_config_file:
             jsconfig = json.load(base_config_file)
 
-        # We're going to modify this list if apps don't have a `ui` directory.
-        enabled_apps = copy.copy(settings.PLUGINS)
-
-        for app_class_path in settings.PLUGINS:
-            app_name = app_class_path.split(".")[-1]
-            app_config = apps.get_app_config(app_name)
-            abs_app_path = Path(app_config.path).resolve()
-            abs_app_ui_path = abs_app_path / "ui"
-            app_path = Path(os.path.relpath(abs_app_path, ui_dir))
-            app_ui_path = app_path / "ui"
-            if app_routes_imports := self.render_routes_imports(abs_app_path, app_name, app_config):
-                app_routes[app_name] = app_routes_imports
-
-            # Assert that an App has a UI folder.
-            if not abs_app_ui_path.exists():
-                self.stdout.write(self.style.ERROR(f"- App {app_name!r} does not publish a UI; Skipping..."))
-                enabled_apps.remove(app_class_path)
-                continue
-            self.stdout.write(self.style.SUCCESS(f"- App {app_name!r} imported"))
-
-            jsconfig["compilerOptions"]["paths"][f"@{app_name}/*"] = [f"{app_ui_path}/*"]
-
-        with open(jsconfig_file_path, "w", encoding="utf-8") as generated_config_file:
-            json.dump(jsconfig, generated_config_file, indent=4)
-
-        with open(app_routes_file_path, "w", encoding="utf-8") as app_routes_file:
-            json.dump(app_routes, app_routes_file, indent=4)
-
-        app_imports_final_file_path = Path(ui_dir, "src", "app_imports.js")
-        environment = jinja2.sandbox.SandboxedEnvironment(loader=jinja2.FileSystemLoader(ui_dir))
-        template = environment.get_template("app_imports.js.j2")
-        content = template.render(apps=enabled_apps)
-
-        with open(app_imports_final_file_path, "w", encoding="utf-8") as generated_import_file:
-            generated_import_file.write(content)
-
-        # Touch the router to attempt to trigger a server reload.
-        Path(router_file_path).touch()
+        if render_apps:
+            self.stdout.write(self.style.WARNING(">>> Rendering Nautobot App imports..."))
+            # We're going to modify this list if apps don't have a `ui` directory.
+            enabled_apps = copy.copy(settings.PLUGINS)
+
+            for app_class_path in settings.PLUGINS:
+                app_name = app_class_path.split(".")[-1]
+                app_config = apps.get_app_config(app_name)
+                abs_app_path = Path(app_config.path).resolve()
+                abs_app_ui_path = abs_app_path / "ui"
+                app_path = Path(os.path.relpath(abs_app_path, ui_dir))
+                app_ui_path = app_path / "ui"
+                if app_routes_imports := self.render_routes_imports(abs_app_path, app_name, app_config):
+                    app_routes[app_name] = app_routes_imports
+
+                # Assert that an App has a UI folder.
+                if not abs_app_ui_path.exists():
+                    self.stdout.write(self.style.ERROR(f"    - App {app_name!r} does not publish a UI; Skipping..."))
+                    enabled_apps.remove(app_class_path)
+                    continue
+                self.stdout.write(self.style.SUCCESS(f"    - App {app_name!r} imported"))
+
+                jsconfig["compilerOptions"]["paths"][f"@{app_name}/*"] = [f"{app_ui_path}/*"]
+        else:
+            enabled_apps = []
+
+        files_generated = False
+
+        if render_apps or not os.path.exists(jsconfig_file_path):
+            with open(jsconfig_file_path, "w", encoding="utf-8") as generated_config_file:
+                json.dump(jsconfig, generated_config_file, indent=4)
+            self.stdout.write(self.style.SUCCESS(f"    - Rendered {jsconfig_file_path}"))
+            files_generated = True
+
+        if render_apps or not os.path.exists(app_routes_file_path):
+            with open(app_routes_file_path, "w", encoding="utf-8") as app_routes_file:
+                json.dump(app_routes, app_routes_file, indent=4)
+            self.stdout.write(self.style.SUCCESS(f"    - Rendered {app_routes_file_path}"))
+            files_generated = True
+
+        if render_apps or not os.path.exists(app_imports_final_file_path):
+            content = template.render(apps=enabled_apps)
+            with open(app_imports_final_file_path, "w", encoding="utf-8") as generated_import_file:
+                generated_import_file.write(content)
+            self.stdout.write(self.style.SUCCESS(f"    - Rendered {app_imports_final_file_path}"))
+            files_generated = True
+
+        if files_generated:
+            # Touch the router to attempt to trigger a server reload.
+            Path(router_file_path).touch()
 
     def run_command(self, command, message, cwd=settings.NAUTOBOT_UI_DIR):
         """
@@ -233,9 +260,8 @@ def handle(self, *args, **options):
             ">>> Copying UI source files...",
         )
 
-        # Generate `app_imports.js`
-        if options["render_apps"]:
-            self.render_app_imports()
+        # Generate `app_imports.js`, `app_routes.json`, and `jsconfig.paths.json` if necessary or if requested.
+        self.render_app_imports(options["render_apps"])
 
         # Run `npm install` and keep it silent by default.
         if options["npm_install"]:


Test patch:
diff --git a/nautobot/ui/src/components/__tests__/AppFullWidthComponents.test.js b/nautobot/ui/src/components/__tests__/AppFullWidthComponents.test.js
--- a/nautobot/ui/src/components/__tests__/AppFullWidthComponents.test.js
+++ b/nautobot/ui/src/components/__tests__/AppFullWidthComponents.test.js
@@ -2,7 +2,7 @@ import { render } from "@testing-library/react";
 import AppFullWidthComponentsWithProps from "../AppFullWidthComponents";
 
 // Set NautobotApps to {}; since we are not testing for plugin integration with nautobot
-jest.mock("../../app_imports", () => ({
+jest.mock("@generated/app_imports", () => ({
     __esModule: true,
     default: {},
 }));
diff --git a/nautobot/ui/src/components/__tests__/Apps.test.js b/nautobot/ui/src/components/__tests__/Apps.test.js
--- a/nautobot/ui/src/components/__tests__/Apps.test.js
+++ b/nautobot/ui/src/components/__tests__/Apps.test.js
@@ -1,7 +1,7 @@
 import Apps from "../Apps";
 
 // As of now we are not testing any plugin integration; so mock default value would be empty
-jest.mock("../../app_imports", () => ({
+jest.mock("@generated/app_imports", () => ({
     __esModule: true,
     default: {},
 }));