Repository: cookiecutter/cookiecutter-django

Issue:
Replace Gulp with Webpack
## Use Webpack as static build tool

Now this project is still using Gulp as frontend build tool, since Webpack is very popoular and the ecosystem is very mature. I think it is better to use Webpack as frontend build tool.

## Rationale

1. `Webpack` is the most popular bundle solution in the frontend community today, it has received 50k stars on Github.
1. It has a great ecosystem, many plugins, loaders. If we search `webpack` on npmjs.com, we can get 20k resulst.
1. If we do not need `React`, `Vue`, we can still use Webpack to help us compile `ES6`, `SCSS` and do many other things (Many people do not know that!)
1. With a proper config, Webpack can save time and let us build modern web application in quick way.

Actually, other community did quicker on this:

1. Laraval has [laravel-mix](https://github.com/JeffreyWay/laravel-mix)
1. Rails has [webpacker](https://github.com/rails/webpacker)

They all simplified the Webpack config job and make developer can quickly write ES6 and SCSS.

## How I solve the problem

I already created a project [python-webpack-boilerplate](https://github.com/AccordBox/python-webpack-boilerplate)

It contains:

1. A `Webpack project template` which has good structure, easy to use and customize. User can use it even they have no idea how to config Webpack.
1. Template tags which can help load Webpack bundle file in the templates, it is smart to `auto load` dependency files.

If some users want to use React or Vue, the `python-webpack-boilerplate` also supports them.

## Features

- **Supports Django and Flask** (will support more framework in the future)
- Automatic multiple entry points
- Automatic code splitting
- Hot Module Replacement (HMR)
- Easy to config and customize
- ES6 Support via [babel](https://babeljs.io/) (v7)
- JavaScript Linting via [eslint-loader](https://github.com/MoOx/eslint-loader)
- SCSS Support via [sass-loader](https://github.com/jtangelder/sass-loader)
- Autoprefixing of browserspecific CSS rules via [postcss](https://postcss.org/) and [autoprefixer](https://github.com/postcss/autoprefixer)
- Style Linting via [stylelint](https://stylelint.io/)

## Benefits

`python-webpack-boilerplate` is to only provide frontend solution, so it can be used with both `Django` and `Flask` projects. 

So Django developers and Flask developers can both contribute and help improve this project and make it better.

## Project link

https://github.com/AccordBox/python-webpack-boilerplate

If some people are interested in this, you can check my post [New Webpack Boilerplate Project for Django & Flask](https://www.accordbox.com/blog/new-webpack-boilerplate-project-django-flask/) to get more about this project.

And I will later create a PR.
Replace Gulp with Webpack
Please check #3091 to see why this PR is created.

## Notes to help code review

### Basic workflow

1. In `hooks/post_gen_project`, I added a function to cookiecutter `frontend` project from the [python-webpack-boilerplate
](https://github.com/AccordBox/python-webpack-boilerplate/)
1. If people have any questions about the frontend, Webpack, they can create issues there.
1. I will also create PR for `cookiecutter-flask`, no matter people use Django or Flask, the frontend project would have consistent style, which make the project easier to maintain.

### Gulp

I have removed the Gulp code and all work is done by [python-webpack-boilerplate](https://github.com/AccordBox/python-webpack-boilerplate/)

### Bootstrap

This project use Bootstrap theme by default.

The `python-webpack-boilerplate` also contains `bootstrap` by default (installed by using `npm install`), so the page theme can still work.

`custom_bootstrap_compilation` is removed from the `cookiecutter.json` because it is not useful anymore.

### Docker

I also updated some docker code, the workflow can work without problem.





Discussion:
Wow! @michael-yin,

Very good work. I am looking forward to it.
Very informative description @michael-yin . If possible, can you clarify the advantages over the current workflow with Gulp?
Same. Definitely looking forward to this as well. I think adding JS frontend frameworks will increase maintenance by a long shot here since something like React and Vue are updated often and work differently.

I think web pack is not only becoming more popular and its performance is better but also it combines certain functionalities together like a combo of Gulp and Grunt.
@Afrowave @demestav @Andrew-Chen-Wang 

I already pushed PR, could you please check and give me feedback if you have?

Thx.

@browniebroke @Andrew-Chen-Wang 

Hi, I have pushed some commits the PR and now it is ready to review. 

## Workflow

In the python-webpack-boilerplate `0.0.5`, I changed the workflow to generate the frontend app from the package in `site-package` instead of the remote Github repo.

So all the config files are `freezed` and this can help people to troubleshoot in some cases.

And I see a little problem in this project

1. In normal cases, user need to install `python-webpack-boilerplate`, and then run command to create frontend app
1. However, this can not work in the `local docker-compose` command, **because `frontend` does not exist when we run `docker-compose -f local.yml build`**, the `node` service will break

So we might still need to use `cookiecutter` to create frontend app from the Github tag in `hooks/post_gen_project.py `

```
    cookiecutter(
        f"https://github.com/AccordBox/python-webpack-boilerplate/",
        checkout=f"v{pkg_version}",
        directory="webpack_boilerplate/frontend_template",
        no_input=True,
    )
```

1.  It seems we can not cookiecutter from Pypi directly?
2. For better security, we can also pass Git commit to the `checkout`

I want to hear your thoughts about this. 

## Doc

I also write a doc to help people who are new to frontend stuff

![frontend_doc_screenshot](https://user-images.githubusercontent.com/4366781/141675135-23a2a0ae-a04e-4162-8f2d-f7d379b04af8.png)

## Bootstrap 5

Now `python-webpack-boilerplate` is using Bootstrap 5.1.1 by default, and everything works fine with the latest version `cookiecutter-django`

1. People can write SCSS to customize Bootstrap, so I think `custom_bootstrap_compilation` can be deprecated.
2. Modern ES bundle solution
3. SCSS, CSS, JS code linting.

I wish this can help people to write better frontend stuff with Django

Please let me know if you have any questions. 

Thx.



> However, this can not work in the local docker-compose command, because frontend does not exist when we run docker-compose -f local.yml build, the node service will break

how about putting it in the entry point file?
@Andrew-Chen-Wang 

I do not think creating `frontend` app in the `entry point file` is good option here.

I will think about it for a while.

Thx.

Resolving patch:
diff --git a/hooks/post_gen_project.py b/hooks/post_gen_project.py
--- a/hooks/post_gen_project.py
+++ b/hooks/post_gen_project.py
@@ -10,6 +10,7 @@
 """
 from __future__ import print_function
 
+import json
 import os
 import random
 import shutil
@@ -87,15 +88,30 @@ def remove_heroku_build_hooks():
     shutil.rmtree("bin")
 
 
+def remove_sass_files():
+    shutil.rmtree(os.path.join("{{cookiecutter.project_slug}}", "static", "sass"))
+
+
 def remove_gulp_files():
     file_names = ["gulpfile.js"]
     for file_name in file_names:
         os.remove(file_name)
-    remove_sass_files()
 
 
-def remove_sass_files():
-    shutil.rmtree(os.path.join("{{cookiecutter.project_slug}}", "static", "sass"))
+def remove_webpack_files():
+    shutil.rmtree("webpack")
+    remove_vendors_js()
+
+
+def remove_vendors_js():
+    vendors_js_path = os.path.join(
+        "{{ cookiecutter.project_slug }}",
+        "static",
+        "js",
+        "vendors.js",
+    )
+    if os.path.exists(vendors_js_path):
+        os.remove(vendors_js_path)
 
 
 def remove_packagejson_file():
@@ -104,6 +120,83 @@ def remove_packagejson_file():
         os.remove(file_name)
 
 
+def update_package_json(remove_dev_deps=None, remove_keys=None, scripts=None):
+    remove_dev_deps = remove_dev_deps or []
+    remove_keys = remove_keys or []
+    scripts = scripts or {}
+    with open("package.json", mode="r") as fd:
+        content = json.load(fd)
+    for package_name in remove_dev_deps:
+        content["devDependencies"].pop(package_name)
+    for key in remove_keys:
+        content.pop(key)
+    content["scripts"].update(scripts)
+    with open("package.json", mode="w") as fd:
+        json.dump(content, fd, ensure_ascii=False, indent=2)
+        fd.write("\n")
+
+
+def handle_js_runner(choice, use_docker, use_async):
+    if choice == "Gulp":
+        update_package_json(
+            remove_dev_deps=[
+                "@babel/core",
+                "@babel/preset-env",
+                "babel-loader",
+                "concurrently",
+                "css-loader",
+                "mini-css-extract-plugin",
+                "postcss-loader",
+                "postcss-preset-env",
+                "sass-loader",
+                "webpack",
+                "webpack-bundle-tracker",
+                "webpack-cli",
+                "webpack-dev-server",
+                "webpack-merge",
+            ],
+            remove_keys=["babel"],
+            scripts={
+                "dev": "gulp",
+                "build": "gulp generate-assets",
+            },
+        )
+        remove_webpack_files()
+    elif choice == "Webpack":
+        scripts = {
+            "dev": "webpack serve --config webpack/dev.config.js",
+            "build": "webpack --config webpack/prod.config.js",
+        }
+        remove_dev_deps = [
+            "browser-sync",
+            "cssnano",
+            "gulp",
+            "gulp-imagemin",
+            "gulp-plumber",
+            "gulp-postcss",
+            "gulp-rename",
+            "gulp-sass",
+            "gulp-uglify-es",
+        ]
+        if not use_docker:
+            dev_django_cmd = (
+                "uvicorn config.asgi:application --reload"
+                if use_async
+                else "python manage.py runserver_plus"
+            )
+            scripts.update(
+                {
+                    "dev": "concurrently npm:dev:*",
+                    "dev:webpack": "webpack serve --config webpack/dev.config.js",
+                    "dev:django": dev_django_cmd,
+                }
+            )
+        else:
+            remove_dev_deps.append("concurrently")
+        update_package_json(remove_dev_deps=remove_dev_deps, scripts=scripts)
+        remove_gulp_files()
+
+
 def remove_celery_files():
     file_names = [
         os.path.join("config", "celery_app.py"),
@@ -384,13 +477,21 @@ def main():
         if "{{ cookiecutter.keep_local_envs_in_vcs }}".lower() == "y":
             append_to_gitignore_file("!.envs/.local/")
 
-    if "{{ cookiecutter.frontend_pipeline }}" != "Gulp":
+    if "{{ cookiecutter.frontend_pipeline }}" in ["None", "Django Compressor"]:
         remove_gulp_files()
+        remove_webpack_files()
+        remove_sass_files()
         remove_packagejson_file()
         if "{{ cookiecutter.use_docker }}".lower() == "y":
             remove_node_dockerfile()
+    else:
+        handle_js_runner(
+            "{{ cookiecutter.frontend_pipeline }}",
+            use_docker=("{{ cookiecutter.use_docker }}".lower() == "y"),
+            use_async=("{{ cookiecutter.use_async }}".lower() == "y"),
+        )
 
-    if "{{ cookiecutter.cloud_provider}}" == "None":
+    if "{{ cookiecutter.cloud_provider }}" == "None":
         print(
             WARNING + "You chose not to use a cloud provider, "
             "media files won't be served in production." + TERMINATOR
diff --git a/{{cookiecutter.project_slug}}/config/settings/base.py b/{{cookiecutter.project_slug}}/config/settings/base.py
--- a/{{cookiecutter.project_slug}}/config/settings/base.py
+++ b/{{cookiecutter.project_slug}}/config/settings/base.py
@@ -87,6 +87,9 @@
     "corsheaders",
     "drf_spectacular",
 {%- endif %}
+{%- if cookiecutter.frontend_pipeline == 'Webpack' %}
+    "webpack_loader",
+{%- endif %}
 ]
 
 LOCAL_APPS = [
@@ -356,6 +359,19 @@
     "VERSION": "1.0.0",
     "SERVE_PERMISSIONS": ["rest_framework.permissions.IsAdminUser"],
 }
+{%- endif %}
+{%- if cookiecutter.frontend_pipeline == 'Webpack' %}
+# django-webpack-loader
+# ------------------------------------------------------------------------------
+WEBPACK_LOADER = {
+    "DEFAULT": {
+        "CACHE": not DEBUG,
+        "STATS_FILE": BASE_DIR / "webpack-stats.json",
+        "POLL_INTERVAL": 0.1,
+        "IGNORE": [r".+\.hot-update.js", r".+\.map"],
+    }
+}
+
 {%- endif %}
 # Your stuff...
 # ------------------------------------------------------------------------------
diff --git a/{{cookiecutter.project_slug}}/config/settings/local.py b/{{cookiecutter.project_slug}}/config/settings/local.py
--- a/{{cookiecutter.project_slug}}/config/settings/local.py
+++ b/{{cookiecutter.project_slug}}/config/settings/local.py
@@ -69,7 +69,7 @@
 
     hostname, _, ips = socket.gethostbyname_ex(socket.gethostname())
     INTERNAL_IPS += [".".join(ip.split(".")[:-1] + ["1"]) for ip in ips]
-    {%- if cookiecutter.frontend_pipeline == 'Gulp' %}
+    {%- if cookiecutter.frontend_pipeline in ['Gulp', 'Webpack'] %}
     try:
         _, _, ips = socket.gethostbyname_ex("node")
         INTERNAL_IPS.extend(ips)
@@ -94,6 +94,12 @@
 # https://docs.celeryq.dev/en/stable/userguide/configuration.html#task-eager-propagates
 CELERY_TASK_EAGER_PROPAGATES = True
 
+{%- endif %}
+{%- if cookiecutter.frontend_pipeline == 'Webpack' %}
+# django-webpack-loader
+# ------------------------------------------------------------------------------
+WEBPACK_LOADER["DEFAULT"]["CACHE"] = not DEBUG  # noqa F405
+
 {%- endif %}
 # Your stuff...
 # ------------------------------------------------------------------------------


Test patch:
diff --git a/tests/test_bare.sh b/tests/test_bare.sh
--- a/tests/test_bare.sh
+++ b/tests/test_bare.sh
@@ -32,13 +32,11 @@ pytest
 # Make sure the check doesn't raise any warnings
 python manage.py check --fail-level WARNING
 
+# Run npm build script if package.json is present
 if [ -f "package.json" ]
 then
     npm install
-    if [ -f "gulpfile.js" ]
-    then
-        npm run build
-    fi
+    npm run build
 fi
 
 # Generate the HTML for the documentation
diff --git a/tests/test_cookiecutter_generation.py b/tests/test_cookiecutter_generation.py
--- a/tests/test_cookiecutter_generation.py
+++ b/tests/test_cookiecutter_generation.py
@@ -101,6 +101,7 @@ def context():
     {"frontend_pipeline": "None"},
     {"frontend_pipeline": "Django Compressor"},
     {"frontend_pipeline": "Gulp"},
+    {"frontend_pipeline": "Webpack"},
     {"use_celery": "y"},
     {"use_celery": "n"},
     {"use_mailhog": "y"},
diff --git a/tests/test_docker.sh b/tests/test_docker.sh
--- a/tests/test_docker.sh
+++ b/tests/test_docker.sh
@@ -41,3 +41,9 @@ docker-compose -f local.yml run django python manage.py check --fail-level WARNI
 
 # Generate the HTML for the documentation
 docker-compose -f local.yml run docs make html
+
+# Run npm build script if package.json is present
+if [ -f "package.json" ]
+then
+    docker-compose -f local.yml run node npm run build
+fi