Repository: phoenixframework/phoenix
Pull request #4027 (2020-10-14 13:23:57)

Run tests from generated apps in CI
The goal of this PR is to have the integration tester ensure the test suites from generated apps pass. By default, tests that use the database are excluded, but can be included using standard ExUnit tag related arguments like `--include database:postgresql`.

I'm submitting this as a draft because I'd like to get your thoughts on this approach and still have some things to figure out.

* We need to have the databases accessible via local ports on the test runner's container, so we don't have to modify the configs of the generated apps. I'm currently setting up a proxy with `socat` but that doesn't appear to be working for `mssql`. I wonder if `iptables` would work better for this, but haven't been able to figure it out yet.
* I'm still trying to figure out how I should break these tests into different files so tests can run in parallel, but I don't want to have so many test files it gets confusing. I'm thinking I'll start out with one test module per database and break it up later. I know the postgresql module will be huge and the test suite won't achieve the parallelism it could, but maybe it will become clear how to split things up later. Also, the next step in the integration test work is to run the `mix phx.gen.*` generators on a project and ensure these tests pass as well.
* I still need migrate the rest of the tests from 
  * `phx_new_test.exs`
  * `phx_new_web_test.exs`
  * `phx_new_ecto_test.exs`
  * `phx_new_umbrella_test.exs`

@josevalim If you have any thoughts on the direction of this work, I'd greatly appreciate it. Thanks!

josevalim: Hi @aaronrenner! This looks exciting and I think the database setup will be necessary to test things like phx.gen.auth. In my opinion, changing the configuration files are fine, but I am fine with either approach.

My only suggestion is to avoid seeing this as a replacement for the unit tests. In particular, I don't think we need to migrate the assertions from the existing generators. The generators test a bunch of scenarios which we will never completely fully explore via the integration tests (I mean, we could but the integration tests would take so long to run that nobody would run them locally and we would get stuck in a very long feedback cycle).

Here are things I would test:

1. Make sure the code compiles and tests pass out of the box for two or three basic scenarios (this is what we already do)
2. Have a test for the development workflow (i.e. we can do a request, code is compiled dynamically, basically [the test here](https://github.com/phoenixframework/phoenix/blob/master/test/mix/tasks/phx.new_test.exs)) for the scenarios in `1.`
3. Make sure the tests for the gen commands pass (for each database)

And that's pretty much it. It is mean to catch obvious errors but not test every possible combinations our generators can emit. Does it make sense? :)
aaronrenner: @josevalim I just updated these integration tests to run the `mix phx.gen.{html,live,json}` commands. Once #4033 is merged and this is rebased on top, the tests should pass.

In investigated replacing the development workflow test in `test/mix/tasks/phx.new_test.exs`, but I wasn't able to easily shutdown the server after running `mix phx.server` because of the zombie process issue. I think this test would be best left as is.

Please let me know if there are any changes you'd like to see to these tests or more generators you'd like to have tested. Once this looks good, I'll start preparing `phx.gen.auth` to be merged in.
josevalim: @aaronrenner this looks great! I have added some comments about some tests we can skip for both mysql and mssql.

Something else we could do is to use setup_all in to setup the basic phoenix tests across the same suite.

Something like this:

```elixir
setup_all do
  with_installer_setup "original" do
    # this should compile all code, including deps for dev and test
    generate_app(...)
  end
end
```

And then on each test:

```elixir
test "compiles" do
  with_installer_run "original", "name of the copy" do
    assert_compiles()
  end
end
```

The `with_installer_setup` removes any existing dir and sets a basic one up while the `with_installer_run` copies the compiled one. Mixing this with the existing deps copying should make it overall very efficient.

Btw, when we copy the deps from integration_test itself, do we also copy the `_build` directory?
josevalim: > In investigated replacing the development workflow test in test/mix/tasks/phx.new_test.exs, but I wasn't able to easily shutdown the server after running mix phx.server because of the zombie process issue. I think this test would be best left as is.

You can do this when starting mix phx.server:

     System.cmd "elixir", ["--no-halt", "-e", "spawn fn -> IO.gets('') && System.halt(0) end", "-S", "mix", "phx.server"]

The idea is to setup a process that waits until `IO.gets('')` returns something (in this case, :eof) to shutdown the VM. You could even add a `--watch-stdin` flag to `mix phx.server` but I think we can skip it for now.

In any case, I think both `mix phx.server` and [the `with_installer_setup` suggestion I gave here](https://github.com/phoenixframework/phoenix/pull/4027) can be done in another PR. Let's get this one merged first. :)

Btw, when we copy the deps from integration_test itself, do we also copy the _build directory? Also, pls let me know when you think this is ready for merge. :) Thank you! :heart:
aaronrenner: Hi @josevalim! Thanks for the suggestion on how to shut down the phoenix server after the test halts. I'm looking forward to trying it!

We are copying the `_build`, `deps`, and `mix.lock` file from the integration test project into each of the generated apps to save significant build time:

https://github.com/phoenixframework/phoenix/blob/779259bc6e281bba2f1e71e4b642e97281399095/integration_test/test/support/code_generator_case.ex#L10-L26

I do like this approach where we actually generate the app during the test, because then we have the flexibility to generate it with any number of options even in the same test suite. For example, when testing `phx.gen.live`, this PR generates the app with `--live` where the rest of the generator tests don't use this option. But I agree, we can tackle that in a future PR if we want.

The last thing I'm trying to do is to get the 1.9.4/21.3.8.17 integration test to pass. I think when I copy `_build` from the integration test project into the test app, I need to expand the symlinks. I've been trying to put together some code to do that, it looks like I've run out of time this morning. I'll try to get that done in the next evening or two.

Thanks again for all your help on this!


josevalim: > The last thing I'm trying to do is to get the 1.9.4/21.3.8.17 integration test to pass.

Don't worry about it. There is a PR that soon will remove support for 1.9.4 and 1.10.0 on the new app generator, which means we no longer need to test those.

> I do like this approach where we actually generate the app during the test, because then we have the flexibility to generate it with any number of options even in the same test suite.

In my suggestion you would lose this ability. The goal is to share apps only within the same block. So for example, we don't need to generate the app twice, one for testing formatting and another for testing the DB. But that's a separate convo. :)
josevalim: I gone ahead and bumped the integration tests to only Elixir v1.11. Which is the same we are doing in PR #4040.

Patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 6eec7ec51..aa0c8d9d7 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -40,21 +40,33 @@ jobs:
       image: hexpm/elixir:${{ matrix.elixir }}-erlang-${{ matrix.otp }}-alpine-3.12.0
       env:
         ELIXIR_ASSERT_TIMEOUT: 2000
-
+    services:
+      postgres:
+        image: postgres
+        ports:
+          - 5432:5432
+        env:
+          POSTGRES_PASSWORD: postgres
+      mysql:
+        image: mysql
+        ports:
+          - 3306:3306
+        env:
+          MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
+      mssql:
+        image: mcr.microsoft.com/mssql/server:2019-latest
+        env:
+          ACCEPT_EULA: Y
+          SA_PASSWORD: some!Password
+        ports:
+          - 1433:1433
     strategy:
       fail-fast: false
       matrix:
         include:
-          # Can't run integration tests before OTP 21 due to incompatiblity with cowlib.
-          #
-          # See https://github.com/phoenixframework/phoenix/issues/3903
-          - elixir: 1.9.4
-            otp: 21.3.8.17
-          - elixir: 1.10.4
-            otp: 21.3.8.17
-          - elixir: 1.11.0
+          - elixir: 1.11.1
             otp: 21.3.8.18
-          - elixir: 1.11.0
+          - elixir: 1.11.1
             otp: 23.1.1
     steps:
       - uses: actions/checkout@v2.3.1
@@ -62,12 +74,17 @@ jobs:
         run: |
           mix local.rebar --force
           mix local.hex --force
-          apk add --no-progress --update git
+          apk add --no-progress --update git socat
+      - name: Set up local proxies
+        run: |
+          socat TCP-LISTEN:5432,fork TCP-CONNECT:postgres:5432&
+          socat TCP-LISTEN:3306,fork TCP-CONNECT:mysql:3306&
+          socat TCP-LISTEN:1433,fork TCP-CONNECT:mssql:1433&
       - name: Install dependencies
         run: mix deps.get
         working-directory: integration_test
       - name: Run integration tests
-        run: mix test
+        run: mix test --include database
         working-directory: integration_test
 
   npm_test:
diff --git a/installer/templates/phx_assets/package.json b/installer/templates/phx_assets/package.json
index 9ec7d1abf..56fd45120 100644
--- a/installer/templates/phx_assets/package.json
+++ b/installer/templates/phx_assets/package.json
@@ -19,7 +19,7 @@
     "copy-webpack-plugin": "^5.1.1",
     "css-loader": "^3.4.2",
     "sass-loader": "^8.0.2",
-    "node-sass": "^4.13.1",
+    "sass": "^1.27.0",
     "hard-source-webpack-plugin": "^0.13.1",
     "mini-css-extract-plugin": "^0.9.0",
     "optimize-css-assets-webpack-plugin": "^5.0.1",


Test patch:
diff --git a/integration_test/README.md b/integration_test/README.md
index c1f1b3059..d602f4964 100644
--- a/integration_test/README.md
+++ b/integration_test/README.md
@@ -8,10 +8,24 @@ To install dependencies, run:
 
     $ mix deps.get
 
-Then run the tests with:
+Then run the basic test suite (no dependencies on the database) with:
 
     $ mix test
 
+To run the test suite with tests that test a specific database, run:
+
+    $ mix test --include database:postgresql
+    $ mix test --include database:mysql
+    $ mix test --include database:mssql
+
+For convenience, there is also a `docker-compose.yml` file that allows for starting up all of the supported databases locally.
+
+    $ docker-compose up
+
+This allows all tests to be run with the following command
+
+    $ mix test --include database
+
 ## How tests are written
 
 In order to have consistent, repeatable builds, all dependencies for all phoenix
diff --git a/integration_test/docker-compose.yml b/integration_test/docker-compose.yml
new file mode 100644
index 000000000..c483f0583
--- /dev/null
+++ b/integration_test/docker-compose.yml
@@ -0,0 +1,21 @@
+version: '3'
+services:
+  postgres:
+    image: postgres
+    ports:
+      - "5432:5432"
+    environment:
+      POSTGRES_PASSWORD: postgres
+  mysql:
+    image: mysql
+    ports:
+      - "3306:3306"
+    environment:
+      MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
+  mssql:
+    image: mcr.microsoft.com/mssql/server:2019-latest
+    environment:
+      ACCEPT_EULA: Y
+      SA_PASSWORD: some!Password
+    ports:
+      - "1433:1433"
diff --git a/integration_test/mix.exs b/integration_test/mix.exs
index 5c8052c69..2992e40a7 100644
--- a/integration_test/mix.exs
+++ b/integration_test/mix.exs
@@ -31,6 +31,8 @@ defmodule Phoenix.Integration.MixProject do
       {:phoenix_ecto, "~> 4.1"},
       {:ecto_sql, "~> 3.4.4"},
       {:postgrex, ">= 0.0.0"},
+      {:myxql, ">= 0.0.0"},
+      {:tds, ">= 0.0.0"},
       {:phoenix_live_view, "~> 0.14.0"},
       {:floki, ">= 0.27.0"},
       {:phoenix_html, "~> 2.11"},
diff --git a/integration_test/mix.lock b/integration_test/mix.lock
index 2c3a552d9..c4c2121e2 100644
--- a/integration_test/mix.lock
+++ b/integration_test/mix.lock
@@ -3,7 +3,7 @@
   "cowboy": {:hex, :cowboy, "2.8.0", "f3dc62e35797ecd9ac1b50db74611193c29815401e53bac9a5c0577bd7bc667d", [:rebar3], [{:cowlib, "~> 2.9.1", [hex: :cowlib, repo: "hexpm", optional: false]}, {:ranch, "~> 1.7.1", [hex: :ranch, repo: "hexpm", optional: false]}], "hexpm", "4643e4fba74ac96d4d152c75803de6fad0b3fa5df354c71afdd6cbeeb15fac8a"},
   "cowlib": {:hex, :cowlib, "2.9.1", "61a6c7c50cf07fdd24b2f45b89500bb93b6686579b069a89f88cb211e1125c78", [:rebar3], [], "hexpm", "e4175dc240a70d996156160891e1c62238ede1729e45740bdd38064dad476170"},
   "db_connection": {:hex, :db_connection, "2.2.2", "3bbca41b199e1598245b716248964926303b5d4609ff065125ce98bcd368939e", [:mix], [{:connection, "~> 1.0.2", [hex: :connection, repo: "hexpm", optional: false]}], "hexpm", "642af240d8a8affb93b4ba5a6fcd2bbcbdc327e1a524b825d383711536f8070c"},
-  "decimal": {:hex, :decimal, "2.0.0", "a78296e617b0f5dd4c6caf57c714431347912ffb1d0842e998e9792b5642d697", [:mix], [], "hexpm", "34666e9c55dea81013e77d9d87370fe6cb6291d1ef32f46a1600230b1d44f577"},
+  "decimal": {:hex, :decimal, "1.9.0", "83e8daf59631d632b171faabafb4a9f4242c514b0a06ba3df493951c08f64d07", [:mix], [], "hexpm", "b1f2343568eed6928f3e751cf2dffde95bfaa19dd95d09e8a9ea92ccfd6f7d85"},
   "ecto": {:hex, :ecto, "3.4.6", "08f7afad3257d6eb8613309af31037e16c36808dfda5a3cd0cb4e9738db030e4", [:mix], [{:decimal, "~> 1.6 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "6f13a9e2a62e75c2dcfc7207bfc65645ab387af8360db4c89fee8b5a4bf3f70b"},
   "ecto_sql": {:hex, :ecto_sql, "3.4.5", "30161f81b167d561a9a2df4329c10ae05ff36eca7ccc84628f2c8b9fa1e43323", [:mix], [{:db_connection, "~> 2.2", [hex: :db_connection, repo: "hexpm", optional: false]}, {:ecto, "~> 3.4.3", [hex: :ecto, repo: "hexpm", optional: false]}, {:myxql, "~> 0.3.0 or ~> 0.4.0", [hex: :myxql, repo: "hexpm", optional: true]}, {:postgrex, "~> 0.15.0", [hex: :postgrex, repo: "hexpm", optional: true]}, {:tds, "~> 2.1.0", [hex: :tds, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "31990c6a3579b36a3c0841d34a94c275e727de8b84f58509da5f1b2032c98ac2"},
   "file_system": {:hex, :file_system, "0.2.9", "545b9c9d502e8bfa71a5315fac2a923bd060fd9acb797fe6595f54b0f975fd32", [:mix], [], "hexpm", "3cf87a377fe1d93043adeec4889feacf594957226b4f19d5897096d6f61345d8"},
@@ -12,6 +12,7 @@
   "html_entities": {:hex, :html_entities, "0.5.1", "1c9715058b42c35a2ab65edc5b36d0ea66dd083767bef6e3edb57870ef556549", [:mix], [], "hexpm", "30efab070904eb897ff05cd52fa61c1025d7f8ef3a9ca250bc4e6513d16c32de"},
   "jason": {:hex, :jason, "1.2.2", "ba43e3f2709fd1aa1dce90aaabfd039d000469c05c56f0b8e31978e03fa39052", [:mix], [{:decimal, "~> 1.0 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "18a228f5f0058ee183f29f9eae0805c6e59d61c3b006760668d8d18ff0d12179"},
   "mime": {:hex, :mime, "1.4.0", "5066f14944b470286146047d2f73518cf5cca82f8e4815cf35d196b58cf07c47", [:mix], [], "hexpm", "75fa42c4228ea9a23f70f123c74ba7cece6a03b1fd474fe13f6a7a85c6ea4ff6"},
+  "myxql": {:hex, :myxql, "0.4.3", "9ad738fcfdf91a807b968f8994ccd4701786e572a3ce3a939c8e0f9001e38486", [:mix], [{:db_connection, "~> 2.0", [hex: :db_connection, repo: "hexpm", optional: false]}, {:decimal, "~> 1.6 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:geo, "~> 3.3", [hex: :geo, repo: "hexpm", optional: true]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}], "hexpm", "744e4c314d9c9bcc55066c20997f11f91c7e458dc94990ab9ed45caf94ee01ed"},
   "phoenix_ecto": {:hex, :phoenix_ecto, "4.2.1", "13f124cf0a3ce0f1948cf24654c7b9f2347169ff75c1123f44674afee6af3b03", [:mix], [{:ecto, "~> 3.0", [hex: :ecto, repo: "hexpm", optional: false]}, {:phoenix_html, "~> 2.14.2 or ~> 2.15", [hex: :phoenix_html, repo: "hexpm", optional: true]}, {:plug, "~> 1.0", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "478a1bae899cac0a6e02be1deec7e2944b7754c04e7d4107fc5a517f877743c0"},
   "phoenix_html": {:hex, :phoenix_html, "2.14.2", "b8a3899a72050f3f48a36430da507dd99caf0ac2d06c77529b1646964f3d563e", [:mix], [{:plug, "~> 1.5", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "58061c8dfd25da5df1ea0ca47c972f161beb6c875cd293917045b92ffe1bf617"},
   "phoenix_live_dashboard": {:hex, :phoenix_live_dashboard, "0.3.0", "e5d56f19bbf71c9645651b946ff945e86a92064127a56aac09afc340aa40b2c2", [:mix], [{:ecto_psql_extras, "~> 0.2", [hex: :ecto_psql_extras, repo: "hexpm", optional: true]}, {:phoenix_html, "~> 2.14.1 or ~> 2.15", [hex: :phoenix_html, repo: "hexpm", optional: false]}, {:phoenix_live_view, "~> 0.14.3", [hex: :phoenix_live_view, repo: "hexpm", optional: false]}, {:telemetry_metrics, "~> 0.4.0 or ~> 0.5.0 or ~> 0.6.0", [hex: :telemetry_metrics, repo: "hexpm", optional: false]}], "hexpm", "9af8800efa40dcc82842fc55253b7b69c056cd3e985ea08ae9bae7b7f07e0bb3"},
@@ -23,6 +24,7 @@
   "plug_crypto": {:hex, :plug_crypto, "1.2.0", "1cb20793aa63a6c619dd18bb33d7a3aa94818e5fd39ad357051a67f26dfa2df6", [:mix], [], "hexpm", "a48b538ae8bf381ffac344520755f3007cc10bd8e90b240af98ea29b69683fc2"},
   "postgrex": {:hex, :postgrex, "0.15.6", "a464c72010a56e3214fe2b99c1a76faab4c2bb0255cabdef30dea763a3569aa2", [:mix], [{:connection, "~> 1.0", [hex: :connection, repo: "hexpm", optional: false]}, {:db_connection, "~> 2.1", [hex: :db_connection, repo: "hexpm", optional: false]}, {:decimal, "~> 1.5 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}], "hexpm", "f99268325ac8f66ffd6c4964faab9e70fbf721234ab2ad238c00f9530b8cdd55"},
   "ranch": {:hex, :ranch, "1.7.1", "6b1fab51b49196860b733a49c07604465a47bdb78aa10c1c16a3d199f7f8c881", [:rebar3], [], "hexpm", "451d8527787df716d99dc36162fca05934915db0b6141bbdac2ea8d3c7afc7d7"},
+  "tds": {:hex, :tds, "2.1.1", "b6163ea716d74ed90a3a83668db2e7c74c1e722fd3538ef5758e0a084fde8d60", [:mix], [{:db_connection, "~> 2.0", [hex: :db_connection, repo: "hexpm", optional: false]}, {:decimal, "~> 1.6", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}], "hexpm", "6b28e1f06a57867eb6b1a957ae5d872b09214ba771ef08cf5ca9d52d6d372876"},
   "telemetry": {:hex, :telemetry, "0.4.2", "2808c992455e08d6177322f14d3bdb6b625fbcfd233a73505870d8738a2f4599", [:rebar3], [], "hexpm", "2d1419bd9dda6a206d7b5852179511722e2b18812310d304620c7bd92a13fcef"},
   "telemetry_metrics": {:hex, :telemetry_metrics, "0.6.0", "da9d49ee7e6bb1c259d36ce6539cd45ae14d81247a2b0c90edf55e2b50507f7b", [:mix], [{:telemetry, "~> 0.4", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "5cfe67ad464b243835512aa44321cee91faed6ea868d7fb761d7016e02915c3d"},
   "telemetry_poller": {:hex, :telemetry_poller, "0.5.1", "21071cc2e536810bac5628b935521ff3e28f0303e770951158c73eaaa01e962a", [:rebar3], [{:telemetry, "~> 0.4", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "4cab72069210bc6e7a080cec9afffad1b33370149ed5d379b81c7c5f0c663fd4"},
diff --git a/integration_test/test/code_generation/app_with_defaults_test.exs b/integration_test/test/code_generation/app_with_defaults_test.exs
index f6a3a7995..da4a9085e 100644
--- a/integration_test/test/code_generation/app_with_defaults_test.exs
+++ b/integration_test/test/code_generation/app_with_defaults_test.exs
@@ -1,12 +1,164 @@
 defmodule Phoenix.Integration.CodeGeneration.AppWithDefaultsTest do
   use Phoenix.Integration.CodeGeneratorCase, async: true
 
-  test "newly generated app has no warnings or errors" do
-    with_installer_tmp("new with defaults", fn tmp_dir ->
-      app_root_path = generate_phoenix_app(tmp_dir, "phx_blog")
+  describe "new with defaults" do
+    test "has no compilation or formatter warnings" do
+      with_installer_tmp("new with defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "phx_blog")
 
-      assert_no_compilation_warnings(app_root_path)
-      assert_passes_formatter_check(app_root_path)
-    end)
+        assert_no_compilation_warnings(app_root_path)
+        assert_passes_formatter_check(app_root_path)
+      end)
+    end
+
+    @tag database: :postgresql
+    test "has a passing test suite" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "default_app")
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+
+  describe "phx.gen.html" do
+    test "has no compilation or formatter warnings" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "phx_blog")
+
+        mix_run!(~w(phx.gen.html Blog Post posts title:unique body:string), app_root_path)
+
+        modify_file(Path.join(app_root_path, "lib/phx_blog_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", PhxBlogWeb do
+              pipe_through [:browser]
+
+              resources "/posts", PostController
+            end
+          """)
+        end)
+
+        assert_no_compilation_warnings(app_root_path)
+        assert_passes_formatter_check(app_root_path)
+      end)
+    end
+
+    @tag database: :postgresql
+    test "has a passing test suite" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "phx_blog")
+
+        mix_run!(~w(phx.gen.html Blog Post posts title body:string), app_root_path)
+
+        modify_file(Path.join(app_root_path, "lib/phx_blog_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", PhxBlogWeb do
+              pipe_through [:browser]
+
+              resources "/posts", PostController
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+
+  describe "phx.gen.json" do
+    test "has no compilation or formatter warnings" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "phx_blog")
+
+        mix_run!(~w(phx.gen.json Blog Post posts title:unique body:string), app_root_path)
+
+        assert_no_compilation_warnings(app_root_path)
+        assert_passes_formatter_check(app_root_path)
+      end)
+    end
+
+    @tag database: :postgresql
+    test "has a passing test suite" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "phx_blog")
+
+        mix_run!(~w(phx.gen.json Blog Post posts title body:string), app_root_path)
+
+        modify_file(Path.join(app_root_path, "lib/phx_blog_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", PhxBlogWeb do
+              pipe_through [:api]
+
+              resources "/posts", PostController, except: [:new, :edit]
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+
+  describe "phx.gen.live" do
+    test "has no compilation or formatter warnings" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "phx_blog", ["--live"])
+
+        mix_run!(~w(phx.gen.live Blog Post posts title:unique body:string), app_root_path)
+
+        modify_file(Path.join(app_root_path, "lib/phx_blog_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", PhxBlogWeb do
+              pipe_through [:browser]
+
+              live "/posts", PostLive.Index, :index
+              live "/posts/new", PostLive.Index, :new
+              live "/posts/:id/edit", PostLive.Index, :edit
+
+              live "/posts/:id", PostLive.Show, :show
+              live "/posts/:id/show/edit", PostLive.Show, :edit
+            end
+          """)
+        end)
+
+        assert_no_compilation_warnings(app_root_path)
+        assert_passes_formatter_check(app_root_path)
+      end)
+    end
+
+    @tag database: :postgresql
+    test "has a passing test suite" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "phx_blog", ["--live"])
+
+        mix_run!(~w(phx.gen.live Blog Post posts title body:string), app_root_path)
+
+        modify_file(Path.join(app_root_path, "lib/phx_blog_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", PhxBlogWeb do
+              pipe_through [:browser]
+
+              live "/posts", PostLive.Index, :index
+              live "/posts/new", PostLive.Index, :new
+              live "/posts/:id/edit", PostLive.Index, :edit
+
+              live "/posts/:id", PostLive.Show, :show
+              live "/posts/:id/show/edit", PostLive.Show, :edit
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
   end
 end
diff --git a/integration_test/test/code_generation/app_with_mssql_adapter_test.exs b/integration_test/test/code_generation/app_with_mssql_adapter_test.exs
new file mode 100644
index 000000000..b6f6e5644
--- /dev/null
+++ b/integration_test/test/code_generation/app_with_mssql_adapter_test.exs
@@ -0,0 +1,86 @@
+defmodule Phoenix.Integration.CodeGeneration.AppWithMSSQLAdapterTest do
+  use Phoenix.Integration.CodeGeneratorCase, async: true
+
+  describe "phx.gen.html" do
+    @tag database: :mssql
+    test "has a passing test suite" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} =
+          generate_phoenix_app(tmp_dir, "default_mssql_app", ["--database", "mssql"])
+
+        mix_run!(~w(phx.gen.html Blog Post posts title body:string), app_root_path)
+
+        modify_file(Path.join(app_root_path, "lib/default_mssql_app_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", DefaultMssqlAppWeb do
+              pipe_through [:browser]
+
+              resources "/posts", PostController
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+
+  describe "phx.gen.json" do
+    @tag database: :mssql
+    test "has a passing test suite" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} =
+          generate_phoenix_app(tmp_dir, "default_mssql_app", ["--database", "mssql"])
+
+        mix_run!(~w(phx.gen.json Blog Post posts title body:string), app_root_path)
+
+        modify_file(Path.join(app_root_path, "lib/default_mssql_app_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", DefaultMssqlAppWeb do
+              pipe_through [:api]
+
+              resources "/posts", PostController, except: [:new, :edit]
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+
+  describe "phx.gen.live" do
+    @tag database: :mssql
+    test "has a passing test suite" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} =
+          generate_phoenix_app(tmp_dir, "default_mssql_app", ["--database", "mssql", "--live"])
+
+        mix_run!(~w(phx.gen.live Blog Post posts title body:string), app_root_path)
+
+        modify_file(Path.join(app_root_path, "lib/default_mssql_app_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", DefaultMssqlAppWeb do
+              pipe_through [:browser]
+
+              live "/posts", PostLive.Index, :index
+              live "/posts/new", PostLive.Index, :new
+              live "/posts/:id/edit", PostLive.Index, :edit
+
+              live "/posts/:id", PostLive.Show, :show
+              live "/posts/:id/show/edit", PostLive.Show, :edit
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+end
diff --git a/integration_test/test/code_generation/app_with_mysql_adapter_test.exs b/integration_test/test/code_generation/app_with_mysql_adapter_test.exs
new file mode 100644
index 000000000..11bafa20b
--- /dev/null
+++ b/integration_test/test/code_generation/app_with_mysql_adapter_test.exs
@@ -0,0 +1,86 @@
+defmodule Phoenix.Integration.CodeGeneration.AppWithMySqlAdapterTest do
+  use Phoenix.Integration.CodeGeneratorCase, async: true
+
+  describe "phx.gen.html" do
+    @tag database: :mysql
+    test "has a passing test suite" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} =
+          generate_phoenix_app(tmp_dir, "default_mysql_app", ["--database", "mysql"])
+
+        mix_run!(~w(phx.gen.html Blog Post posts title body:string), app_root_path)
+
+        modify_file(Path.join(app_root_path, "lib/default_mysql_app_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", DefaultMysqlAppWeb do
+              pipe_through [:browser]
+
+              resources "/posts", PostController
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+
+  describe "phx.gen.json" do
+    @tag database: :mysql
+    test "has a passing test suite" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} =
+          generate_phoenix_app(tmp_dir, "default_mysql_app", ["--database", "mysql"])
+
+        mix_run!(~w(phx.gen.json Blog Post posts title body:string), app_root_path)
+
+        modify_file(Path.join(app_root_path, "lib/default_mysql_app_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", DefaultMysqlAppWeb do
+              pipe_through [:api]
+
+              resources "/posts", PostController, except: [:new, :edit]
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+
+  describe "phx.gen.live" do
+    @tag database: :mysql
+    test "has a passing test suite" do
+      with_installer_tmp("app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} =
+          generate_phoenix_app(tmp_dir, "default_mysql_app", ["--database", "mysql", "--live"])
+
+        mix_run!(~w(phx.gen.live Blog Post posts title body:string), app_root_path)
+
+        modify_file(Path.join(app_root_path, "lib/default_mysql_app_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", DefaultMysqlAppWeb do
+              pipe_through [:browser]
+
+              live "/posts", PostLive.Index, :index
+              live "/posts/new", PostLive.Index, :new
+              live "/posts/:id/edit", PostLive.Index, :edit
+
+              live "/posts/:id", PostLive.Show, :show
+              live "/posts/:id/show/edit", PostLive.Show, :edit
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+end
diff --git a/integration_test/test/code_generation/app_with_no_options_test.exs b/integration_test/test/code_generation/app_with_no_options_test.exs
index 34094a02b..987ae5fa1 100644
--- a/integration_test/test/code_generation/app_with_no_options_test.exs
+++ b/integration_test/test/code_generation/app_with_no_options_test.exs
@@ -3,16 +3,18 @@ defmodule Phoenix.Integration.CodeGeneration.AppWithNoOptionsTest do
 
   test "newly generated app has no warnings or errors" do
     with_installer_tmp("app_with_no_options", fn tmp_dir ->
-      app_root_path = generate_phoenix_app(tmp_dir, "phx_blog", [
-            "--no-html",
-            "--no-webpack",
-            "--no-ecto",
-            "--no-gettext",
-            "--no-dashboard"
-          ])
+      {app_root_path, _} =
+        generate_phoenix_app(tmp_dir, "phx_blog", [
+          "--no-html",
+          "--no-webpack",
+          "--no-ecto",
+          "--no-gettext",
+          "--no-dashboard"
+        ])
 
       assert_no_compilation_warnings(app_root_path)
       assert_passes_formatter_check(app_root_path)
+      assert_tests_pass(app_root_path)
     end)
   end
 end
diff --git a/integration_test/test/code_generation/umbrella_app_with_defaults_test.exs b/integration_test/test/code_generation/umbrella_app_with_defaults_test.exs
index d1eafa4a5..a529bd3c8 100644
--- a/integration_test/test/code_generation/umbrella_app_with_defaults_test.exs
+++ b/integration_test/test/code_generation/umbrella_app_with_defaults_test.exs
@@ -1,12 +1,171 @@
 defmodule Phoenix.Integration.CodeGeneration.UmbrellaAppWithDefaultsTest do
   use Phoenix.Integration.CodeGeneratorCase, async: true
 
-  test "new umbrella app" do
-    with_installer_tmp("umbrella_app_with_defaults", fn tmp_dir ->
-      app_root_path = generate_phoenix_app(tmp_dir, "phx_blog", ["--umbrella"])
+  describe "new umbrella app" do
+    test "has no compilation or formatter warnings" do
+      with_installer_tmp("umbrella_app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "phx_blog", ["--umbrella"])
 
-      assert_no_compilation_warnings(app_root_path)
-      assert_passes_formatter_check(app_root_path)
-    end)
+        assert_no_compilation_warnings(app_root_path)
+        assert_passes_formatter_check(app_root_path)
+      end)
+    end
+
+    @tag database: :postgresql
+    test "has a passing test suite" do
+      with_installer_tmp("umbrella_app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} =
+          generate_phoenix_app(tmp_dir, "umbrella_with_defaults", ["--umbrella"])
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+
+  describe "phx.gen.html" do
+    test "has no compilation or formatter warnings" do
+      with_installer_tmp("umbrella_app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "rainy_day", ["--umbrella"])
+        web_root_path = Path.join(app_root_path, "apps/rainy_day_web")
+
+        mix_run!(~w(phx.gen.html Blog Post posts title:unique body:string), web_root_path)
+
+        modify_file(Path.join(web_root_path, "lib/rainy_day_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", RainyDayWeb do
+              pipe_through [:browser]
+
+              resources "/posts", PostController
+            end
+          """)
+        end)
+
+        assert_no_compilation_warnings(app_root_path)
+        assert_passes_formatter_check(app_root_path)
+      end)
+    end
+
+    @tag database: :postgresql
+    test "has a passing test suite" do
+      with_installer_tmp("umbrella_app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "rainy_day", ["--umbrella"])
+        web_root_path = Path.join(app_root_path, "apps/rainy_day_web")
+
+        mix_run!(~w(phx.gen.html Blog Post posts title body:string), web_root_path)
+
+        modify_file(Path.join(web_root_path, "lib/rainy_day_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", RainyDayWeb do
+              pipe_through [:browser]
+
+              resources "/posts", PostController
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+
+  describe "phx.gen.json" do
+    test "has no compilation or formatter warnings" do
+      with_installer_tmp("umbrella_app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "rainy_day", ["--umbrella"])
+        web_root_path = Path.join(app_root_path, "apps/rainy_day_web")
+
+        mix_run!(~w(phx.gen.json Blog Post posts title:unique body:string), web_root_path)
+
+        assert_no_compilation_warnings(app_root_path)
+        assert_passes_formatter_check(app_root_path)
+      end)
+    end
+
+    @tag database: :postgresql
+    test "has a passing test suite" do
+      with_installer_tmp("umbrella_app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "rainy_day", ["--umbrella"])
+        web_root_path = Path.join(app_root_path, "apps/rainy_day_web")
+
+        mix_run!(~w(phx.gen.json Blog Post posts title body:string), web_root_path)
+
+        modify_file(Path.join(web_root_path, "lib/rainy_day_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", RainyDayWeb do
+              pipe_through [:api]
+
+              resources "/posts", PostController
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
+  end
+
+  describe "phx.gen.live" do
+    test "has no compilation or formatter warnings" do
+      with_installer_tmp("umbrella_app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "rainy_day", ["--umbrella", "--live"])
+        web_root_path = Path.join(app_root_path, "apps/rainy_day_web")
+
+        mix_run!(~w(phx.gen.live Blog Post posts title:unique body:string), web_root_path)
+
+        modify_file(Path.join(web_root_path, "lib/rainy_day_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", RainyDayWeb do
+              pipe_through [:browser]
+
+              live "/posts", PostLive.Index, :index
+              live "/posts/new", PostLive.Index, :new
+              live "/posts/:id/edit", PostLive.Index, :edit
+
+              live "/posts/:id", PostLive.Show, :show
+              live "/posts/:id/show/edit", PostLive.Show, :edit
+            end
+          """)
+        end)
+
+        assert_no_compilation_warnings(app_root_path)
+        assert_passes_formatter_check(app_root_path)
+      end)
+    end
+
+    @tag database: :postgresql
+    test "has a passing test suite" do
+      with_installer_tmp("umbrella_app_with_defaults", fn tmp_dir ->
+        {app_root_path, _} = generate_phoenix_app(tmp_dir, "rainy_day", ["--umbrella", "--live"])
+        web_root_path = Path.join(app_root_path, "apps/rainy_day_web")
+
+        mix_run!(~w(phx.gen.live Blog Post posts title body:string), web_root_path)
+
+        modify_file(Path.join(web_root_path, "lib/rainy_day_web/router.ex"), fn file ->
+          inject_before_final_end(file, """
+
+            scope "/", RainyDayWeb do
+              pipe_through [:browser]
+
+              live "/posts", PostLive.Index, :index
+              live "/posts/new", PostLive.Index, :new
+              live "/posts/:id/edit", PostLive.Index, :edit
+
+              live "/posts/:id", PostLive.Show, :show
+              live "/posts/:id/show/edit", PostLive.Show, :edit
+            end
+          """)
+        end)
+
+        drop_test_database(app_root_path)
+        assert_tests_pass(app_root_path)
+      end)
+    end
   end
 end
diff --git a/integration_test/test/support/code_generator_case.ex b/integration_test/test/support/code_generator_case.ex
index d99443909..5df6c6007 100644
--- a/integration_test/test/support/code_generator_case.ex
+++ b/integration_test/test/support/code_generator_case.ex
@@ -13,7 +13,8 @@ defmodule Phoenix.Integration.CodeGeneratorCase do
     integration_test_root_path = Path.expand("../../", __DIR__)
     app_root_path = get_app_root_path(tmp_dir, app_name, opts)
 
-    mix_run!(["phx.new", app_path, "--dev", "--no-install"] ++ opts, integration_test_root_path)
+    output =
+      mix_run!(["phx.new", app_path, "--dev", "--no-install"] ++ opts, integration_test_root_path)
 
     for path <- ~w(mix.lock deps _build) do
       File.cp_r!(
@@ -22,7 +23,7 @@ defmodule Phoenix.Integration.CodeGeneratorCase do
       )
     end
 
-    app_root_path
+    {app_root_path, output}
   end
 
   def mix_run!(args, app_path, opts \\ [])
@@ -51,6 +52,39 @@ defmodule Phoenix.Integration.CodeGeneratorCase do
     System.cmd("mix", args, [stderr_to_stdout: true, cd: Path.expand(app_path)] ++ opts)
   end
 
+  def assert_dir(path) do
+    assert File.dir?(path), "Expected #{path} to be a directory, but is not"
+  end
+
+  def assert_file(file) do
+    assert File.regular?(file), "Expected #{file} to exist, but does not"
+  end
+
+  def refute_file(file) do
+    refute File.regular?(file), "Expected #{file} to not exist, but it does"
+  end
+
+  def assert_file(file, match) do
+    cond do
+      is_list(match) ->
+        assert_file(file, &Enum.each(match, fn m -> assert &1 =~ m end))
+
+      is_binary(match) or Regex.regex?(match) ->
+        assert_file(file, &assert(&1 =~ match))
+
+      is_function(match, 1) ->
+        assert_file(file)
+        match.(File.read!(file))
+
+      true ->
+        raise inspect({file, match})
+    end
+  end
+
+  def assert_tests_pass(app_path) do
+    mix_run!(~w(test), app_path)
+  end
+
   def assert_passes_formatter_check(app_path) do
     mix_run!(~w(format --check-formatted), app_path)
   end
@@ -59,6 +93,10 @@ defmodule Phoenix.Integration.CodeGeneratorCase do
     mix_run!(["do", "clean,", "compile", "--warnings-as-errors"], app_path)
   end
 
+  def drop_test_database(app_path) when is_binary(app_path) do
+    mix_run!(["ecto.drop"], app_path, env: [{"MIX_ENV", "test"}])
+  end
+
   def with_installer_tmp(name, opts \\ [], function)
       when is_list(opts) and is_function(function, 1) do
     autoremove? = Keyword.get(opts, :autoremove?, true)
@@ -77,6 +115,26 @@ defmodule Phoenix.Integration.CodeGeneratorCase do
     Path.expand("../../../installer/tmp", __DIR__)
   end
 
+  def inject_before_final_end(code, code_to_inject)
+      when is_binary(code) and is_binary(code_to_inject) do
+    code
+    |> String.trim_trailing()
+    |> String.trim_trailing("end")
+    |> Kernel.<>(code_to_inject)
+    |> Kernel.<>("end\n")
+  end
+
+  def modify_file(path, function) when is_binary(path) and is_function(function, 1) do
+    path
+    |> File.read!()
+    |> function.()
+    |> write_file!(path)
+  end
+
+  defp write_file!(content, path) do
+    File.write!(path, content)
+  end
+
   defp get_app_root_path(tmp_dir, app_name, opts) do
     app_root_dir =
       if "--umbrella" in opts do
@@ -89,6 +147,6 @@ defmodule Phoenix.Integration.CodeGeneratorCase do
   end
 
   defp random_string(len) do
-    len |> :crypto.strong_rand_bytes() |> Base.encode64() |> binary_part(0, len)
+    len |> :crypto.strong_rand_bytes() |> Base.url_encode64() |> binary_part(0, len)
   end
 end
diff --git a/integration_test/test/test_helper.exs b/integration_test/test/test_helper.exs
index 837fc8c8b..afa010226 100644
--- a/integration_test/test/test_helper.exs
+++ b/integration_test/test/test_helper.exs
@@ -6,5 +6,5 @@ File.cp_r!(
   Path.expand("../_build/dev", __DIR__)
 )
 
-ExUnit.configure(timeout: 180_000)
+ExUnit.configure(timeout: 180_000, exclude: [:database])
 ExUnit.start()