Repository: plotly/dash

Issue:
[BUG] dash 1.11 breaks Dynamic Multipage App Validation
**Describe your context**
Please provide us your environment so we can easily reproduce the issue.

`pip list | grep dash` 
```
dash                               1.11.0              
dash-bootstrap-components          0.9.1               
dash-core-components               1.9.1               
dash-dangerously-set-inner-html    0.0.2               
dash-flow-example                  0.0.5               
dash-html-components               1.0.3               
dash-renderer                      1.4.0               
dash-table                         4.6.2               
```

**Describe the bug**

Copy example *Dynamically Create a Layout for Multi-Page App Validation* from https://dash.plotly.com/urls

<details>
  <summary>Click to expand!</summary>


```python
import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output, State

import flask

app = dash.Dash(
    __name__,
    external_stylesheets=['https://codepen.io/chriddyp/pen/bWLwgP.css']
)

url_bar_and_content_div = html.Div([
    dcc.Location(id='url', refresh=False),
    html.Div(id='page-content')
])

layout_index = html.Div([
    dcc.Link('Navigate to "/page-1"', href='/page-1'),
    html.Br(),
    dcc.Link('Navigate to "/page-2"', href='/page-2'),
])

layout_page_1 = html.Div([
    html.H2('Page 1'),
    dcc.Input(id='input-1-state', type='text', value='Montreal'),
    dcc.Input(id='input-2-state', type='text', value='Canada'),
    html.Button(id='submit-button', n_clicks=0, children='Submit'),
    html.Div(id='output-state'),
    html.Br(),
    dcc.Link('Navigate to "/"', href='/'),
    html.Br(),
    dcc.Link('Navigate to "/page-2"', href='/page-2'),
])

layout_page_2 = html.Div([
    html.H2('Page 2'),
    dcc.Dropdown(
        id='page-2-dropdown',
        options=[{'label': i, 'value': i} for i in ['LA', 'NYC', 'MTL']],
        value='LA'
    ),
    html.Div(id='page-2-display-value'),
    html.Br(),
    dcc.Link('Navigate to "/"', href='/'),
    html.Br(),
    dcc.Link('Navigate to "/page-1"', href='/page-1'),
])


def serve_layout():
    if flask.has_request_context():
        return url_bar_and_content_div
    return html.Div([
        url_bar_and_content_div,
        layout_index,
        layout_page_1,
        layout_page_2,
    ])


app.layout = serve_layout


# Index callbacks
@app.callback(Output('page-content', 'children'),
              [Input('url', 'pathname')])
def display_page(pathname):
    if pathname == "/page-1":
        return layout_page_1
    elif pathname == "/page-2":
        return layout_page_2
    else:
        return layout_index


# Page 1 callbacks
@app.callback(Output('output-state', 'children'),
              [Input('submit-button', 'n_clicks')],
              [State('input-1-state', 'value'),
               State('input-2-state', 'value')])
def update_output(n_clicks, input1, input2):
    return ('The Button has been pressed {} times,'
            'Input 1 is "{}",'
            'and Input 2 is "{}"').format(n_clicks, input1, input2)


# Page 2 callbacks
@app.callback(Output('page-2-display-value', 'children'),
              [Input('page-2-dropdown', 'value')])
def display_value(value):
    print('display_value')
    return 'You have selected "{}"'.format(value)


if __name__ == '__main__':
    app.run_server(debug=True)
```

</details>

**Expected behavior**

No errors.

**Screenshots**

```json
🛑 Errors (4)
⛑️
ID not found in layout
10:53:31 AM
Attempting to connect a callback Input item to component:
  "page-2-dropdown"
but no components with that id exist in the layout.

If you are assigning callbacks to components that are
generated by other callbacks (and therefore not in the
initial layout), you can suppress this exception by setting
`suppress_callback_exceptions=True`.
This ID was used in the callback(s) for Output(s):
  page-2-display-value.children
```
once per ID:
<details>
  <summary>Click to expand!</summary>


```json
⛑️
ID not found in layout
10:53:31 AM
Attempting to connect a callback Input item to component:
  "submit-button"
but no components with that id exist in the layout.

If you are assigning callbacks to components that are
generated by other callbacks (and therefore not in the
initial layout), you can suppress this exception by setting
`suppress_callback_exceptions=True`.
This ID was used in the callback(s) for Output(s):
  output-state.children
⛑️
ID not found in layout
10:53:31 AM
Attempting to connect a callback Output item to component:
  "page-2-display-value"
but no components with that id exist in the layout.

If you are assigning callbacks to components that are
generated by other callbacks (and therefore not in the
initial layout), you can suppress this exception by setting
`suppress_callback_exceptions=True`.
This ID was used in the callback(s) for Output(s):
  page-2-display-value.children
⛑️
ID not found in layout
10:53:31 AM
Attempting to connect a callback Output item to component:
  "output-state"
but no components with that id exist in the layout.

If you are assigning callbacks to components that are
generated by other callbacks (and therefore not in the
initial layout), you can suppress this exception by setting
`suppress_callback_exceptions=True`.
This ID was used in the callback(s) for Output(s):
  output-state.children
```

</details>


Discussion:
note: I think pip upgraded all deps as needed but, this [comment](https://github.com/plotly/dash-docs/pull/831#issuecomment-612177743) about disabling *devtools asserts* made me doubt, it looks related (?):
 
> @chriddyp @alexcjohnson As we discussed in DM, merging this in as there's no averse affect on feature/behavior. Will create a follow up issue to re-enable devtools asserts once we figure it out.


Thanks @eddy-geek - did this example work without errrors prior to v1.11? It looks to me as though it's just missing `suppress_callback_exceptions=True`, which is essentially always needed for multi-page apps and is included in all the other examples on that page. Anyway that fixes it for me locally - we'll get the docs updated (as well as updating them to use the preferred usage of `suppress_callback_exceptions` as a constructor kwarg, rather than the older `app.config.suppress_callback_exceptions = True`)
@alexcjohnson yes the example worked fine, and "validation" -- ie not suppressing exceptions -- is actually the whole point! The doc says it better than I could:

>For full validation, all components within your callback must therefore appear in the initial layout of your app, and you will see an error if they do not. However, in the case of more complex Dash apps that involve dynamic modification of the layout (such as multi-page apps), not every component appearing in your callbacks will be included in the initial layout.
>
>Since this validation is done before flask has any request context, you can create a layout function that checks flask.has_request_context() and returns a complete layout to the validator if there is no request context and returns the incomplete index layout otherwise.

I'm using the same technique in my app (so I am back on 1.10.0) and I think it is popular (judging by related community threads) because any non-toy app is multi-page, and `suppress_callback_exceptions` slows down development/debugging *a lot*. Actually if I understand correctly, wildcards will allow more use-cases to work without `suppress_callback_exceptions` which is going in the good direction :)

So, while the client-side validation and wildcard look great, I would argue it's important to either maintain support for, or replace, this pattern, not just take the "easy" route of `suppress_callback_exceptions`. 
Ah OK! I was unaware of the `flask.has_request_context` technique, thank you for bringing it to my attention! We'll have to think about how to support that with client-side validation, it doesn't seem simple but I do see a way to do it.

Resolving patch:
diff --git a/dash/dash.py b/dash/dash.py
--- a/dash/dash.py
+++ b/dash/dash.py
@@ -331,7 +331,8 @@ def __init__(
         self.routes = []
 
         self._layout = None
-        self._cached_layout = None
+        self._layout_is_function = False
+        self.validation_layout = None
 
         self._setup_dev_tools()
         self._hot_reload = AttributeDict(
@@ -421,18 +422,48 @@ def layout(self):
         return self._layout
 
     def _layout_value(self):
-        if isinstance(self._layout, patch_collections_abc("Callable")):
-            self._cached_layout = self._layout()
-        else:
-            self._cached_layout = self._layout
-        return self._cached_layout
+        return self._layout() if self._layout_is_function else self._layout
 
     @layout.setter
     def layout(self, value):
         _validate.validate_layout_type(value)
-        self._cached_layout = None
+        self._layout_is_function = isinstance(value, patch_collections_abc("Callable"))
         self._layout = value
 
+        # for using flask.has_request_context() to deliver a full layout for
+        # validation inside a layout function - track if a user might be doing this.
+        if (
+            self._layout_is_function
+            and not self.validation_layout
+            and not self.config.suppress_callback_exceptions
+        ):
+
+            def simple_clone(c, children=None):
+                cls = type(c)
+                # in Py3 we can use the __init__ signature to reduce to just
+                # required args and id; in Py2 this doesn't work so we just
+                # empty out children.
+                sig = getattr(cls.__init__, "__signature__", None)
+                props = {
+                    p: getattr(c, p)
+                    for p in c._prop_names  # pylint: disable=protected-access
+                    if hasattr(c, p)
+                    and (
+                        p == "id" or not sig or sig.parameters[p].default == c.REQUIRED
+                    )
+                }
+                if props.get("children", children):
+                    props["children"] = children or []
+                return cls(**props)
+
+            layout_value = self._layout_value()
+            _validate.validate_layout(value, layout_value)
+            self.validation_layout = simple_clone(
+                # pylint: disable=protected-access
+                layout_value,
+                [simple_clone(c) for c in layout_value._traverse_ids()],
+            )
+
     @property
     def index_string(self):
         return self._index_string
@@ -468,6 +499,9 @@ def _config(self):
                 "interval": int(self._dev_tools.hot_reload_interval * 1000),
                 "max_retry": self._dev_tools.hot_reload_max_retry,
             }
+        if self.validation_layout and not self.config.suppress_callback_exceptions:
+            config["validation_layout"] = self.validation_layout
+
         return config
 
     def serve_reload_hash(self):
@@ -602,7 +636,7 @@ def _generate_scripts_html(self):
 
     def _generate_config_html(self):
         return '<script id="_dash-config" type="application/json">{}</script>'.format(
-            json.dumps(self._config())
+            json.dumps(self._config(), cls=plotly.utils.PlotlyJSONEncoder)
         )
 
     def _generate_renderer(self):
diff --git a/dash/development/base_component.py b/dash/development/base_component.py
--- a/dash/development/base_component.py
+++ b/dash/development/base_component.py
@@ -293,11 +293,16 @@ def _traverse_with_paths(self):
                     for p, t in i._traverse_with_paths():
                         yield "\n".join([list_path, p]), t
 
-    def __iter__(self):
-        """Yield IDs in the tree of children."""
+    def _traverse_ids(self):
+        """Yield components with IDs in the tree of children."""
         for t in self._traverse():
             if isinstance(t, Component) and getattr(t, "id", None) is not None:
-                yield t.id
+                yield t
+
+    def __iter__(self):
+        """Yield IDs in the tree of children."""
+        for t in self._traverse_ids():
+            yield t.id
 
     def __len__(self):
         """Return the number of items in the tree."""


Test patch:
diff --git a/tests/integration/devtools/test_callback_validation.py b/tests/integration/devtools/test_callback_validation.py
--- a/tests/integration/devtools/test_callback_validation.py
+++ b/tests/integration/devtools/test_callback_validation.py
@@ -1,3 +1,6 @@
+import flask
+import pytest
+
 import dash_core_components as dcc
 import dash_html_components as html
 from dash import Dash
@@ -695,3 +698,124 @@ def c3(children):
         ]
     ]
     check_errors(dash_duo, specs)
+
+
+def multipage_app(validation=False):
+    app = Dash(__name__, suppress_callback_exceptions=(validation == "suppress"))
+
+    skeleton = html.Div(
+        [dcc.Location(id="url", refresh=False), html.Div(id="page-content")]
+    )
+
+    layout_index = html.Div(
+        [
+            dcc.Link('Navigate to "/page-1"', id="index_p1", href="/page-1"),
+            dcc.Link('Navigate to "/page-2"', id="index_p2", href="/page-2"),
+        ]
+    )
+
+    layout_page_1 = html.Div(
+        [
+            html.H2("Page 1"),
+            dcc.Input(id="input-1-state", type="text", value="Montreal"),
+            dcc.Input(id="input-2-state", type="text", value="Canada"),
+            html.Button(id="submit-button", n_clicks=0, children="Submit"),
+            html.Div(id="output-state"),
+            html.Br(),
+            dcc.Link('Navigate to "/"', id="p1_index", href="/"),
+            dcc.Link('Navigate to "/page-2"', id="p1_p2", href="/page-2"),
+        ]
+    )
+
+    layout_page_2 = html.Div(
+        [
+            html.H2("Page 2"),
+            dcc.Input(id="page-2-input", value="LA"),
+            html.Div(id="page-2-display-value"),
+            html.Br(),
+            dcc.Link('Navigate to "/"', id="p2_index", href="/"),
+            dcc.Link('Navigate to "/page-1"', id="p2_p1", href="/page-1"),
+        ]
+    )
+
+    validation_layout = html.Div([skeleton, layout_index, layout_page_1, layout_page_2])
+
+    def validation_function():
+        return skeleton if flask.has_request_context() else validation_layout
+
+    app.layout = validation_function if validation == "function" else skeleton
+    if validation == "attribute":
+        app.validation_layout = validation_layout
+
+    # Index callbacks
+    @app.callback(Output("page-content", "children"), [Input("url", "pathname")])
+    def display_page(pathname):
+        if pathname == "/page-1":
+            return layout_page_1
+        elif pathname == "/page-2":
+            return layout_page_2
+        else:
+            return layout_index
+
+    # Page 1 callbacks
+    @app.callback(
+        Output("output-state", "children"),
+        [Input("submit-button", "n_clicks")],
+        [State("input-1-state", "value"), State("input-2-state", "value")],
+    )
+    def update_output(n_clicks, input1, input2):
+        return (
+            "The Button has been pressed {} times,"
+            'Input 1 is "{}",'
+            'and Input 2 is "{}"'
+        ).format(n_clicks, input1, input2)
+
+    # Page 2 callbacks
+    @app.callback(
+        Output("page-2-display-value", "children"), [Input("page-2-input", "value")]
+    )
+    def display_value(value):
+        print("display_value")
+        return 'You have selected "{}"'.format(value)
+
+    return app
+
+
+def test_dvcv014_multipage_errors(dash_duo):
+    app = multipage_app()
+    dash_duo.start_server(app, **debugging)
+
+    specs = [
+        [
+            "ID not found in layout",
+            ['"page-2-input"', "page-2-display-value.children"],
+        ],
+        ["ID not found in layout", ['"submit-button"', "output-state.children"]],
+        [
+            "ID not found in layout",
+            ['"page-2-display-value"', "page-2-display-value.children"],
+        ],
+        ["ID not found in layout", ['"output-state"', "output-state.children"]],
+    ]
+    check_errors(dash_duo, specs)
+
+
+@pytest.mark.parametrize("validation", ("function", "attribute", "suppress"))
+def test_dvcv015_multipage_validation_layout(validation, dash_duo):
+    app = multipage_app(validation)
+    dash_duo.start_server(app, **debugging)
+
+    dash_duo.wait_for_text_to_equal("#index_p1", 'Navigate to "/page-1"')
+    dash_duo.find_element("#index_p1").click()
+
+    dash_duo.find_element("#submit-button").click()
+    dash_duo.wait_for_text_to_equal(
+        "#output-state",
+        "The Button has been pressed 1 times,"
+        'Input 1 is "Montreal",and Input 2 is "Canada"',
+    )
+
+    dash_duo.find_element("#p1_p2").click()
+    dash_duo.wait_for_text_to_equal("#page-2-display-value", 'You have selected "LA"')
+
+    assert not dash_duo.get_logs()