Repository: plannigan/columbo (language: python)

Issue:
Store all code examples in documentation as .py files that are evaluated and templated into the docs
We might consider storing all code examples used in the documentation as individual `.py` files. When the docs are built, each example is evaluated and the code (along with whatever is printed to stdout) is templated into the docs.

Projects like Pydantic use this approach and it allows them to:

1. Have confidence that all examples in the documentation work properly (examples that won't run would fail the CI pipeline)
1. Give readers of the documentation more information (they see the output of the code snippet without needing to copy & execute it themself)
1. Minimize maintenance, as the output would be automatically generated.  

Here's one example from Pydantic:

* [Code snippet in the docs](https://pydantic-docs.helpmanual.io/#__code_0:~:text=print(user.dict()),%22%22%22)
* [File where the code snippet is defined](https://github.com/samuelcolvin/pydantic/blob/d5e9d9abc8c8f554d660801ddc14bf6f32c9db45/docs/examples/index_main.py)
* [Code for building the examples and templating them into the markdown docs](https://github.com/samuelcolvin/pydantic/tree/d5e9d9abc8c8f554d660801ddc14bf6f32c9db45/docs/build)


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 530d2af0..4b2b034a 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -152,6 +152,15 @@ jobs:
         with:
           package-import-name: "columbo"
 
+  validate-doc-examples:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v2
+      
+      - name: Validate docs
+        run: ./docker/validate_docs.sh
+
   build-docs:
     runs-on: ubuntu-latest
     if: github.event_name == 'pull_request'
diff --git a/docker-compose.yaml b/docker-compose.yaml
index 0e99cb5a..55be492f 100644
--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -23,6 +23,11 @@ services:
     <<: *devbox
     command: docker/run_tests.sh --format-code
 
+  # test the documentation examples to make sure they can be run with Python
+  validateDocExamples:
+    <<: *devbox
+    command: docker/validate_docs.sh
+
   bump:
     <<: *devbox
     <<: *mount-app-and-user-git-config
diff --git a/docker/validate_docs.sh b/docker/validate_docs.sh
new file mode 100755
index 00000000..db6e3725
--- /dev/null
+++ b/docker/validate_docs.sh
@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+# this script will exit if one of the example files does not execute properly with python
+
+# this is set so that the script fails if one example fails to be executed properly
+set -e
+
+# install columbo
+pip install . -q;
+
+for filename in docs/examples/*.py; do
+    # provide default answers if example python file asks for input
+    yes "" | python ${filename};
+done
+
+# this will only get printed if all examples finish succesfully
+printf "\n\n\nAll of the documentation examples can be run!";
diff --git a/docs/development-guide.md b/docs/development-guide.md
index 87153901..a5d5847f 100644
--- a/docs/development-guide.md
+++ b/docs/development-guide.md
@@ -62,7 +62,7 @@ coverage rather than decreasing it.
 
 We use [pytest][pytest-docs] as our testing framework.
 
-#### Stages
+### Stages
 
 To customize / override a specific testing stage, please read the documentation specific to that tool:
 
@@ -73,7 +73,22 @@ To customize / override a specific testing stage, please read the documentation
 4. [Flake8][flake8-docs]
 5. [Bandit][bandit-docs]
 
-### `setup.py`
+## Validate Examples Used in Documentation
+
+In the `docs/examples/` directory of this repo, there are example Python scripts which we use in our documentation.
+You can validate that the examples run properly using:
+
+```bash
+docker-compose run --rm validateDocExamples
+```
+
+If the script fails (exits with a non-zero status), it will output information about the file that we need to fix.
+
+Note that this script will output some content in the shell every time it runs.
+Just because the script outputs content to the shell does *not* mean it has failed;
+as long as the script finishes successfully (exits with a zero status), there are no problems we need to address.
+
+## `setup.py`
 
 Setuptools is used to packaging the library.
 
diff --git a/docs/examples/README.md b/docs/examples/README.md
new file mode 100644
index 00000000..ea266df8
--- /dev/null
+++ b/docs/examples/README.md
@@ -0,0 +1,5 @@
+# Examples
+
+These are the examples used in the [documentation][documentation].
+
+[documentation]: https://wayfair-incubator.github.io/columbo/
diff --git a/examples/branching_story.py b/docs/examples/branching_story.py
similarity index 93%
rename from examples/branching_story.py
rename to docs/examples/branching_story.py
index 546134d7..0c351240 100644
--- a/examples/branching_story.py
+++ b/docs/examples/branching_story.py
@@ -1,11 +1,14 @@
 import columbo
 
+
 def went_left(answers: columbo.Answers) -> bool:
     return answers["which_door"] == "left"
 
+
 def went_right(answers: columbo.Answers) -> bool:
     return answers["which_door"] == "right"
 
+
 def outcome(answers: columbo.Answers) -> str:
     if answers.get("has_key", False):
         return "You try the the key on the lock. With a little jiggling, it finally opens. You open the gate and leave."
@@ -15,7 +18,7 @@ def outcome(answers: columbo.Answers) -> str:
         "Unable to open the gate yourself, you yell for help. A farmer in the nearby field hears you. "
         "He reaches into his pocket and pulls out a key to unlock the gate and open it. "
         "As you walk through the archway he says, "
-        "\"What I don't understand is how you got in there. This is the only key.\""
+        '"What I don\'t understand is how you got in there. This is the only key."'
     )
 
 
@@ -28,7 +31,7 @@ def outcome(answers: columbo.Answers) -> str:
         "which_door",
         "Which door do you walk through?",
         options=["left", "right"],
-        default="left"
+        default="left",
     ),
     columbo.Confirm(
         "has_key",
@@ -36,7 +39,7 @@ def outcome(answers: columbo.Answers) -> str:
         "As you walk down the hallway, there is a small side table with a key on it.\n"
         "Do you pick up the key before going through the door at the other end?",
         should_ask=went_left,
-        default=True
+        default=True,
     ),
     columbo.Confirm(
         "has_hammer",
@@ -44,7 +47,7 @@ def outcome(answers: columbo.Answers) -> str:
         "The room has a single door on the opposite side of the room and a work bench with a hammer on it.\n"
         "Do you pick up the hammer before going through the door at the other side?",
         should_ask=went_right,
-        default=True
+        default=True,
     ),
     columbo.Echo(
         "You enter a small courtyard with high walls. There is an archway that would allow you to go free, "
diff --git a/docs/examples/index_command_line_answers.py b/docs/examples/index_command_line_answers.py
new file mode 100644
index 00000000..7928342f
--- /dev/null
+++ b/docs/examples/index_command_line_answers.py
@@ -0,0 +1,33 @@
+import columbo
+
+interactions = [
+    columbo.Echo("Welcome to the Columbo example"),
+    columbo.Acknowledge("Press enter to start"),
+    columbo.BasicQuestion(
+        "user",
+        "What is your name?",
+        default="Patrick",
+    ),
+    columbo.BasicQuestion(
+        "user_email",
+        lambda answers: f"""What email address should be used to contact {answers["user"]}?""",
+        default="me@example.com",
+    ),
+    columbo.Choice(
+        "mood",
+        "How are you feeling today?",
+        options=["happy", "sad", "sleepy", "confused"],
+        default="happy",
+    ),
+    columbo.Confirm("likes_dogs", "Do you like dogs?", default=True),
+]
+
+answers = columbo.parse_args(
+    interactions,
+    args=[
+        "--user-email",
+        "patrick@example.com",
+        "--likes-dogs",
+    ],
+)
+print(answers)
diff --git a/docs/examples/index_user_prompts.py b/docs/examples/index_user_prompts.py
new file mode 100644
index 00000000..ee85f6d2
--- /dev/null
+++ b/docs/examples/index_user_prompts.py
@@ -0,0 +1,26 @@
+import columbo
+
+interactions = [
+    columbo.Echo("Welcome to the Columbo example"),
+    columbo.Acknowledge("Press enter to start"),
+    columbo.BasicQuestion(
+        "user",
+        "What is your name?",
+        default="Patrick",
+    ),
+    columbo.BasicQuestion(
+        "user_email",
+        lambda answers: f"""What email address should be used to contact {answers["user"]}?""",
+        default="me@example.com",
+    ),
+    columbo.Choice(
+        "mood",
+        "How are you feeling today?",
+        options=["happy", "sad", "sleepy", "confused"],
+        default="happy",
+    ),
+    columbo.Confirm("likes_dogs", "Do you like dogs?", default=True),
+]
+
+answers = columbo.get_answers(interactions)
+print(answers)
diff --git a/examples/optional_questions.py b/docs/examples/optional_questions.py
similarity index 65%
rename from examples/optional_questions.py
rename to docs/examples/optional_questions.py
index b983cf9a..903dc8b4 100644
--- a/examples/optional_questions.py
+++ b/docs/examples/optional_questions.py
@@ -1,7 +1,7 @@
 import columbo
 
 
-def does_user_have_a_dog(answers: columbo.Answers) -> bool:
+def user_has_dog(answers: columbo.Answers) -> bool:
     return answers["has_dog"]
 
 
@@ -10,15 +10,15 @@ def does_user_have_a_dog(answers: columbo.Answers) -> bool:
     columbo.BasicQuestion(
         "dog_name",
         "What is the name of the dog?",
-        should_ask=does_user_have_a_dog,
-        default="Kaylee"
+        should_ask=user_has_dog,
+        default="Kaylee",
     ),
     columbo.BasicQuestion(
         "dog_breed",
         "What is the breed of the dog?",
-        should_ask=does_user_have_a_dog,
-        default="Basset Hound"
-    )
+        should_ask=user_has_dog,
+        default="Basset Hound",
+    ),
 ]
 
 user_answers = columbo.get_answers(interactions)
diff --git a/docs/examples/validators.py b/docs/examples/validators.py
new file mode 100644
index 00000000..83db5ffc
--- /dev/null
+++ b/docs/examples/validators.py
@@ -0,0 +1,25 @@
+import re
+from typing import List
+
+import columbo
+
+
+def is_email_address(value: str, _: columbo.Answers) -> columbo.ValidationResponse:
+    if not re.match(r"^\w+@\w+", value):
+        error_message = f"{value} is not a valid email address"
+        return columbo.ValidationFailure(error=error_message)
+
+    return columbo.ValidationSuccess()
+
+
+interactions: List[columbo.Interaction] = [
+    columbo.BasicQuestion(
+        "user_email_address",
+        "What email address should be used to contact you?",
+        default="me@example.com",
+        validator=is_email_address,
+    )
+]
+
+user_answers = columbo.get_answers(interactions)
+print(user_answers)
diff --git a/docs/getting-started.md b/docs/getting-started.md
index 53ca1584..401df672 100644
--- a/docs/getting-started.md
+++ b/docs/getting-started.md
@@ -79,34 +79,7 @@ printed next.
 This is the example that appears on the [main page][docs-main] of the documentation.
 
 ```python linenums="1"
-import columbo
-
-interactions = [
-    columbo.Echo("Welcome to the Columbo example"),
-    columbo.Acknowledge(
-        "Press enter to start"
-    ),
-    columbo.BasicQuestion(
-        "user",
-        "What is your name?",
-        default="Patrick",
-    ),
-    columbo.BasicQuestion(
-        "user_email",
-        lambda answers: f"""What email address should be used to contact {answers["user"]}?""",
-        default="me@example.com"
-    ),
-    columbo.Choice(
-        "mood",
-        "How are you feeling today?",
-        options=["happy", "sad", "sleepy", "confused"],
-        default="happy",
-    ),
-    columbo.Confirm("likes_dogs", "Do you like dogs?", default=True),
-]
-
-answers = columbo.get_answers(interactions)
-print(answers)
+{!examples/index_user_prompts.py!}
 ```
 
 * Line 1: Import the `columbo` module.
@@ -152,37 +125,7 @@ print(answers)
     <summary>The full example</summary>
 
 ```python linenums="1" hl_lines="27-30"
-import columbo
-
-interactions = [
-    columbo.Echo("Welcome to the Columbo example"),
-    columbo.Acknowledge(
-        "Press enter to start"
-    ),
-    columbo.BasicQuestion(
-        "user",
-        "What is your name?",
-        default="Patrick",
-    ),
-    columbo.BasicQuestion(
-        "user_email",
-        lambda answers: f"""What email address should be used to contact {answers["user"]}?""",
-        default="me@example.com"
-    ),
-    columbo.Choice(
-        "mood",
-        "How are you feeling today?",
-        options=["happy", "sad", "sleepy", "confused"],
-        default="happy",
-    ),
-    columbo.Confirm("likes_dogs", "Do you like dogs?", default=True),
-]
-
-answers = columbo.parse_args(interactions, args=[
-    "--user-email", "patrick@example.com",
-    "--likes-dogs",
-])
-print(answers)
+{!examples/index_command_line_answers.py!}
 ```
 </details>
 
diff --git a/docs/index.md b/docs/index.md
index f3c0d651..3c515c3f 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -28,34 +28,7 @@ The primary use of `columbo` is to define a sequence of interactions that are us
 using a terminal. Below is a sample which shows some ways this can be used.
 
 ```python
-import columbo
-
-interactions = [
-    columbo.Echo("Welcome to the Columbo example"),
-    columbo.Acknowledge(
-        "Press enter to start"
-    ),
-    columbo.BasicQuestion(
-        "user",
-        "What is your name?",
-        default="Patrick",
-    ),
-    columbo.BasicQuestion(
-        "user_email",
-        lambda answers: f"""What email address should be used to contact {answers["user"]}?""",
-        default="me@example.com"
-    ),
-    columbo.Choice(
-        "mood",
-        "How are you feeling today?",
-        options=["happy", "sad", "sleepy", "confused"],
-        default="happy",
-    ),
-    columbo.Confirm("likes_dogs", "Do you like dogs?", default=True),
-]
-
-answers = columbo.get_answers(interactions)
-print(answers)
+{!examples/index_user_prompts.py!}
 ```
 
 Below shows the output when the user accepts the default values for most of the questions. The user provides a different
@@ -95,34 +68,7 @@ $ python columbo_example.py --user-email patrick@example.com --likes-dogs
     <summary>The full example</summary>
 
 ```python
-import columbo
-
-interactions = [
-    columbo.Echo("Welcome to the Columbo example"),
-    columbo.Acknowledge(
-        "Press enter to start"
-    ),
-    columbo.BasicQuestion(
-        "user",
-        "What is your name?",
-        default="Patrick",
-    ),
-    columbo.BasicQuestion(
-        "user_email",
-        lambda answers: f"""What email address should be used to contact {answers["user"]}?""",
-        default="me@example.com"
-    ),
-    columbo.Choice(
-        "mood",
-        "How are you feeling today?",
-        options=["happy", "sad", "sleepy", "confused"],
-        default="happy",
-    ),
-    columbo.Confirm("likes_dogs", "Do you like dogs?", default=True),
-]
-
-answers = columbo.parse_args(interactions)
-print(answers)
+{!examples/index_command_line_answers.py!}
 ```
 </details>
 
diff --git a/docs/usage-guide/optional-questions-and-branching.md b/docs/usage-guide/optional-questions-and-branching.md
index 749eeedc..9808e4c2 100644
--- a/docs/usage-guide/optional-questions-and-branching.md
+++ b/docs/usage-guide/optional-questions-and-branching.md
@@ -18,29 +18,7 @@ The following is a basic example that has two optional questions that are not as
 question.
 
 ```python
-import columbo
-
-def user_has_dog(answers: columbo.Answers) -> bool:
-    return answers["has_dog"]
-
-interactions = [
-    columbo.Confirm("has_dog", "Do you have a dog?", default=True),
-    columbo.BasicQuestion(
-        "dog_name",
-        "What is the name of the dog?",
-        should_ask=user_has_dog,
-        default="Kaylee"
-    ),
-    columbo.BasicQuestion(
-        "dog_breed",
-        "What is the breed of the dog?",
-        should_ask=user_has_dog,
-        default="Basset Hound"
-    )
-]
-
-user_answers = columbo.get_answers(interactions)
-print(user_answers)
+{!examples/optional_questions.py!}
 ```
 
 If the user accepts the default answers for each of these questions, the output will be:
@@ -69,63 +47,7 @@ isn't different from the optional questions [demonstrated above](#optional-quest
 branching paths by supplying different `should_ask` values that will never both evaluate to `True`.
 
 ```python
-import columbo
-
-def went_left(answers: columbo.Answers) -> bool:
-    return answers["which_door"] == "left"
-
-def went_right(answers: columbo.Answers) -> bool:
-    return answers["which_door"] == "right"
-
-def outcome(answers: columbo.Answers) -> str:
-    if answers.get("has_key", False):
-        return "You try the the key on the lock. With a little jiggling, it finally opens. You open the gate and leave."
-    if answers.get("has_hammer", False):
-        return "You hit the lock with the hammer and it falls to the ground. You open the gate and leave."
-    return (
-        "Unable to open the gate yourself, you yell for help. A farmer in the nearby field hears you. "
-        "He reaches into his pocket and pulls out a key to unlock the gate and open it. "
-        "As you walk through the archway he says, "
-        "\"What I don't understand is how you got in there. This is the only key.\""
-    )
-
-
-interactions = [
-    columbo.Echo(
-        "You wake up in a room that you do not recognize. "
-        "In the dim light, you can see a large door to the left and a small door to the right."
-    ),
-    columbo.Choice(
-        "which_door",
-        "Which door do you walk through?",
-        options=["left", "right"],
-        default="left"
-    ),
-    columbo.Confirm(
-        "has_key",
-        "You step into a short hallway and the door closes behind you, refusing to open again. "
-        "As you walk down the hallway, there is a small side table with a key on it.\n"
-        "Do you pick up the key before going through the door at the other end?",
-        should_ask=went_left,
-        default=True
-    ),
-    columbo.Confirm(
-        "has_hammer",
-        "You step into smaller room and the door closes behind, refusing to open again. "
-        "The room has a single door on the opposite side of the room and a work bench with a hammer on it.\n"
-        "Do you pick up the hammer before going through the door at the other side?",
-        should_ask=went_right,
-        default=True
-    ),
-    columbo.Echo(
-        "You enter a small courtyard with high walls. There is an archway that would allow you to go free, "
-        "but the gate is locked."
-    ),
-    columbo.Echo(outcome),
-]
-
-user_answers = columbo.get_answers(interactions)
-print(user_answers)
+{!examples/branching_story.py!}
 ```
 
 The import thing to note in the example above is that the `Answers` dictionary can have a key-value pair for
diff --git a/docs/usage-guide/validators.md b/docs/usage-guide/validators.md
index 3c3f609f..1308c33a 100644
--- a/docs/usage-guide/validators.md
+++ b/docs/usage-guide/validators.md
@@ -41,29 +41,7 @@ word character on each side then the response is invalid and the user will have
 enter an email address again (hopefully a valid one this time).
 
 ```python
-from typing import List
-import re
-
-import columbo
-
-def is_email_address(value: str, _: columbo.Answers) -> columbo.ValidationResponse:
-    if not re.match(r"^\w+@\w+", value):
-        error_message = f"{value} is not a valid email address"
-        return columbo.ValidationFailure(error=error_message)
-
-    return columbo.ValidationSuccess()
-
-interactions: List[columbo.Interaction] = [
-    columbo.BasicQuestion(
-        "user_email_address",
-        "What email address should be used to contact you?",
-        default="me@example.com",
-        validator=is_email_address,
-    )
-]
-
-user_answers = columbo.get_answers(interactions)
-print(user_answers)
+{!examples/validators.py!}
 ```
 
 [^1]:
diff --git a/examples/README.md b/examples/README.md
deleted file mode 100644
index 5a3d11a8..00000000
--- a/examples/README.md
+++ /dev/null
@@ -1,6 +0,0 @@
-# Examples
-
-A collection of examples that can be run to see how `columbo` works. If it not clear what a specific example is doing,
-please read the [documentation][documentation]. These examples all appear there with detailed explanations.
-
-[documentation]: https://wayfair-incubator.github.io/columbo/
diff --git a/examples/basic.py b/examples/basic.py
deleted file mode 100644
index 3c69b4ec..00000000
--- a/examples/basic.py
+++ /dev/null
@@ -1,62 +0,0 @@
-import re
-import sys
-from typing import List, Optional, cast
-
-from columbo import (
-    Acknowledge,
-    Answers,
-    BasicQuestion,
-    Choice,
-    Confirm,
-    Echo,
-    Interaction,
-    get_answers,
-    parse_args,
-)
-
-
-def user_to_email(answers: Answers) -> str:
-    return f"""{cast(str, answers["user"]).lower().replace(' ', '')}@example.com"""
-
-
-def is_email(value: str, _: Answers) -> Optional[str]:
-    error_message: Optional[str] = None
-
-    if not re.match(r"^\w+@\w+", value):
-        error_message = f"{value} is not a valid email"
-
-    return error_message
-
-
-interactions: List[Interaction] = [
-    Echo("Welcome to the Columbo example"),
-    Acknowledge(
-        "Press enter to start"
-    ),
-    BasicQuestion(
-        "user",
-        "What is your name?",
-        default="Patrick",
-        cli_help="Name of the user providing answers",
-    ),
-    BasicQuestion(
-        "user_email",
-        lambda answers: f"""What email address should be used to contact {answers["user"]}?""",
-        default=user_to_email,
-        validator=is_email,
-    ),
-    Choice(
-        "mood",
-        "How are you feeling today?",
-        options=["happy", "sad", "sleepy", "confused"],
-        default="happy",
-        cli_help="The mood of the user.",
-    ),
-    Confirm("likes_dogs", "Do you like dogs?", default=True),
-]
-
-if __name__ == "__main__":
-    if len(sys.argv) > 1:
-        print(parse_args(interactions, sys.argv[1:]))
-    else:
-        print(get_answers(interactions))
diff --git a/mkdocs.yml b/mkdocs.yml
index b922f0ea..c8ba41e3 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -18,6 +18,8 @@ nav:
   - Changelog: changelog.md
 theme: material
 markdown_extensions:
+  - markdown_include.include:
+      base_path: docs
   - admonition
   - codehilite
   - footnotes


Test patch:
diff --git a/docker/run_tests.sh b/docker/run_tests.sh
index 14076475..895d80ff 100755
--- a/docker/run_tests.sh
+++ b/docker/run_tests.sh
@@ -41,10 +41,10 @@ echo "Running MyPy..."
 mypy columbo tests
 
 echo "Running black..."
-black ${BLACK_ACTION} columbo tests
+black ${BLACK_ACTION} columbo tests docs/examples/
 
 echo "Running iSort..."
-isort ${ISORT_ACTION} columbo tests
+isort ${ISORT_ACTION} columbo tests docs/examples/
 
 echo "Running flake8..."
 flake8 columbo tests
diff --git a/requirements-test.txt b/requirements-test.txt
index 14c390ff..b657e54a 100644
--- a/requirements-test.txt
+++ b/requirements-test.txt
@@ -4,6 +4,7 @@ bump2version==1.0.1
 flake8==3.8.4
 isort==5.7.0
 mike==0.5.5
+markdown-include==0.6.0
 mkdocs-material==7.0.3
 mkdocs-minify-plugin==0.4.0
 mkdocstrings==0.14.0