internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: PrefectHQ/prefect

Issue:
ShellTask docs say it returns string instead of bytes



Discussion:

Resolving patch:
diff --git a/src/prefect/tasks/core/shell.py b/src/prefect/tasks/core/shell.py
--- a/src/prefect/tasks/core/shell.py
+++ b/src/prefect/tasks/core/shell.py
@@ -28,7 +28,7 @@ def run(self, command, env=None):
                 on the system
 
         Returns:
-            - stdout + stderr (string): anything printed to standard out /
+            - stdout + stderr (bytes): anything printed to standard out /
                 standard error during command execution
 
         Raises:


Test patch:
Repository: PrefectHQ/prefect

Issue:
Suppress documentation logging
For some reason, this line is printed many times by `generate_docs.py` when building docs:

`(https://github.com/PrefectHQ/prefect/commit/7fd54f656c76aa6571939dd9b330f2a4c6867140)*</sub>
11:32:22 AM: <sub>*This documentation was auto-generated from [7fd54f6]`


Discussion:

Resolving patch:
diff --git a/docs/generate_docs.py b/docs/generate_docs.py
--- a/docs/generate_docs.py
+++ b/docs/generate_docs.py
@@ -395,9 +395,9 @@ def generate_coverage():
     GIT_SHA = os.getenv("GIT_SHA", "0000000")
     SHORT_SHA = GIT_SHA[:7]
     auto_generated_footer = (
-        "<hr>\n\n<sub>*This documentation was auto-generated from "
-        "[{short_sha}](https://github.com/PrefectHQ/prefect/commit/{git_sha})"
-        "*</sub>".format(short_sha=SHORT_SHA, git_sha=GIT_SHA)
+        "<hr>\n\n<p><small><i>This documentation was auto-generated from "
+        "<a href='https://github.com/PrefectHQ/prefect/commit/{git_sha}'>{short_sha}</a>"
+        "</i></small></p>".format(short_sha=SHORT_SHA, git_sha=GIT_SHA)
     )
 
     front_matter = textwrap.dedent(


Test patch:
Repository: PrefectHQ/prefect

Issue:
Don't allow passing parameters that don't exist
`FlowRunner` raises an error if you run a flow and don't pass a required parameter, but it has no problem with users providing a parameter that doesn't exist. This should raise an error as well to avoid confusion.

```python
f = Flow()
f.run(parameters=dict(a=1)) # no error!
```


Discussion:
Error or warning?  Given that passing unnecessary information doesn't technically hurt anything, I'd lean towards warning, but maybe that's harder to capture on the server side

Resolving patch:
diff --git a/src/prefect/core/flow.py b/src/prefect/core/flow.py
--- a/src/prefect/core/flow.py
+++ b/src/prefect/core/flow.py
@@ -815,6 +815,14 @@ def run(
         """
         runner = prefect.engine.flow_runner.FlowRunner(flow=self)
         parameters = parameters or []
+        unknown_params = [p for p in parameters if p not in self.parameters()]
+        if unknown_params:
+            fmt_params = ", ".join(unknown_params)
+            raise TypeError(
+                "Flow.run received the following unexpected parameters: {}".format(
+                    fmt_params
+                )
+            )
 
         passed_parameters = {}
         for p in self.parameters():


Test patch:
diff --git a/tests/core/test_flow.py b/tests/core/test_flow.py
--- a/tests/core/test_flow.py
+++ b/tests/core/test_flow.py
@@ -666,7 +666,7 @@ def test_sorted_tasks_with_start_task():
     assert set(f.sorted_tasks(root_tasks=[t3])) == set([t3, t4, t5])
 
 
-def test_flow_ignores_irrelevant_user_provided_parameters():
+def test_flow_raises_for_irrelevant_user_provided_parameters():
     class ParameterTask(Task):
         def run(self):
             return prefect.context.get("_parameters")
@@ -677,8 +677,11 @@ def run(self):
         f.add_task(x)
         f.add_task(t)
 
-    state = f.run(return_tasks=[t], parameters=dict(x=10, y=3, z=9))
-    assert state.result[t].result == dict(x=10)
+    with pytest.raises(TypeError):
+        state = f.run(return_tasks=[t], parameters=dict(x=10, y=3, z=9))
+
+    with pytest.raises(TypeError):
+        state = f.run(return_tasks=[t], x=10, y=3, z=9)
 
 
 def test_validate_cycles():
Repository: PrefectHQ/prefect

Issue:
Update `DotDict` docstring
We should update the `DotDict` class docstring to clearly articulate what can / can't be accessed via `.`:
- only valid python identifiers (no numbers, etc.)
- no dunders `__X` gets converted to `_DotDict__X` via mangling
Changelog doesn't update in doc builds
Looking at #297 I realize that our formatted docs Changelog doesn't update (we use a static file); we should start updating this as we do the `README.md` 


Discussion:

Resolving patch:
diff --git a/docs/generate_docs.py b/docs/generate_docs.py
--- a/docs/generate_docs.py
+++ b/docs/generate_docs.py
@@ -488,6 +488,8 @@ def get_class_methods(obj):
     shutil.rmtree("api", ignore_errors=True)
     os.makedirs("api", exist_ok=True)
     generate_coverage()
+
+    ## UPDATE README
     with open("api/README.md", "w+") as f:
         f.write(
             textwrap.dedent(
@@ -516,6 +518,23 @@ def get_class_methods(obj):
             f.write("\n" + readme[readme.index("# Prefect") :])
             f.write(auto_generated_footer)
 
+    ## UPDATE CHANGELOG
+    with open("changelog.md", "w+") as f:
+        f.write(
+            textwrap.dedent(
+                """
+            ---
+            sidebarDepth: 1
+            editLink: false
+            ---
+            """
+            ).lstrip()
+        )
+        with open("../CHANGELOG.md", "r") as g:
+            changelog = g.read()
+            f.write(changelog)
+            f.write(auto_generated_footer)
+
     for page in OUTLINE:
         # collect what to document
         fname, classes, fns = (
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -7,7 +7,7 @@
 
 install_requires = [
     "click >= 6.7, < 7.0",
-    "cloudpickle == 0.5.3",
+    "cloudpickle >= 0.6.0",
     "croniter >= 0.3.23, < 0.4",
     "cryptography >= 2.2.2, < 3.0",
     "dask >= 0.18, < 0.19",
diff --git a/src/prefect/utilities/collections.py b/src/prefect/utilities/collections.py
--- a/src/prefect/utilities/collections.py
+++ b/src/prefect/utilities/collections.py
@@ -36,7 +36,9 @@ def flatten_seq(seq: Iterable) -> Generator:
 class DotDict(MutableMapping):
     """
     A `dict` that also supports attribute ("dot") access. Think of this as an extension
-    to the standard python `dict` object.
+    to the standard python `dict` object.  **Note**: while any hashable object can be added to
+    a `DotDict`, _only_ valid Python identifiers can be accessed with the dot syntax; this excludes
+    strings which begin in numbers, special characters, or double underscores.
 
     Args:
         - init_dict (dict, optional): dictionary to initialize the `DotDict`


Test patch:
Repository: PrefectHQ/prefect

Issue:
Add back `environment.yml` file
I realized why we might want to maintain an `environment.yml` file in parallel with our `requirements.txt` file: `requirements.txt` will be installed via `pip`, whereas if you create an environment via `conda`, the packages will be installed / maintained via `conda`.  This can be useful for those who try to `conda install` everything (since it has different package version logic + handles non-python dependencies).


Discussion:
I recently learned that `conda install requirements.txt` is a valid pattern, so just converting our requirements to a file could be a useful step.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -2,51 +2,39 @@
 
 from setuptools import find_packages, setup
 
+import configparser
 import sys
 import versioneer
 
-install_requires = [
-    "click >= 6.7, < 7.0",
-    "cloudpickle >= 0.6.0",
-    "croniter >= 0.3.23, < 0.4",
-    "cryptography >= 2.2.2, < 3.0",
-    "dask >= 0.18, < 0.19",
-    "distributed >= 1.21.8, < 2.0",
-    "docker >= 3.4.1, < 3.5",
-    "marshmallow == 3.0.0b19",
-    "marshmallow-oneofschema >= 2.0.0b2, < 3.0",
-    "mypy >= 0.600, < 0.700",
-    "mypy_extensions >= 0.4.0, < 0.5",
-    "pendulum >= 2.0.4, < 3.0",
-    "python-dateutil >= 2.7.3, < 3.0",
-    "requests >= 2.20, < 3.0",
-    "toml >= 0.9.4, < 1.0",
-    "typing >= 3.6.4, < 4.0",
-    "typing_extensions >= 3.6.4, < 4.0",
-    "xxhash >= 1.2.0, < 2.0",
-]
+config = configparser.ConfigParser()
+config.read("requirements.ini")
 
-templates = ["jinja2 >= 2.0, < 3.0"]
-viz = ["bokeh == 0.13.0", "graphviz >= 0.8.3"]
-dev = [
-    "pre-commit",
-    "pytest >= 3.8, < 4.0",
-    "pytest-cov",
-    "pytest-env",
-    "pytest-xdist",
-    "Pygments == 2.2.0",
-]
+## base requirements
+install_requires = ["".join(req) for req in config["base"].items()]
 
-if sys.version_info >= (3, 6):
-    dev += ["black"]
+## section dependencies
+includes = {}
+for section in config.sections():
+    includes[section] = config[section].pop("include", "").split(",")
 
 extras = {
-    "dev": dev + viz,
-    "viz": viz,
-    "templates": templates,
-    "all_extras": dev + templates + viz,
+    "dev": ["".join(req) for req in config["dev"].items()],
+    "viz": ["".join(req) for req in config["viz"].items()],
+    "templates": ["".join(req) for req in config["templates"].items()],
 }
 
+## process include keyword for related sections
+for section in extras:
+    for other in includes[section]:
+        extras[section] += extras.get(other.strip(), [])
+
+
+if sys.version_info >= (3, 6):
+    extras["dev"] += ["black"]
+
+extras["all_extras"] = extras["dev"] + extras["viz"] + extras["templates"]
+
+
 setup(
     name="prefect",
     version=versioneer.get_version(),


Test patch:
Repository: PrefectHQ/prefect

Issue:
Include executable notebooks with all tutorials
Each tutorial should include a downloadable IPython notebook for executing all of the tutorial code


Discussion:
Side thought: this would also allows us to doc-test the tutorials by running the notebooks and, at a minimum, ensure they run to completion.
Good point

Resolving patch:
diff --git a/docs/generate_docs.py b/docs/generate_docs.py
--- a/docs/generate_docs.py
+++ b/docs/generate_docs.py
@@ -12,14 +12,15 @@
 On a development installation of Prefect, simply run `python generate_docs.py` from inside the `docs/` folder.
 """
 import inspect
+import nbformat as nbf
 import os
 import re
 import shutil
 import subprocess
 import textwrap
+import toolz
 import warnings
 
-import toolz
 from functools import partial
 
 import prefect
@@ -449,6 +450,34 @@ def get_class_methods(obj):
     return public_members
 
 
+def create_tutorial_notebooks(tutorial):
+    """
+    Utility which automagically creates an .ipynb notebook file from a markdown file consisting
+    of all python code blocks contained within the markdown file.
+
+    Args:
+        - tutorial (str): path to tutorial markdown file
+
+    Will save the resulting notebook in tutorials/notebooks under the same name as the .md file provided.
+    """
+    assert (
+        os.path.basename(os.getcwd()) == "docs"
+    ), "Only run this utility from inside the docs/ directory!"
+
+    os.makedirs(".vuepress/public/notebooks", exist_ok=True)
+    text = open(tutorial, "r").read()
+    code_blocks = re.findall(r"```(.*?)```", text, re.DOTALL)
+    nb = nbf.v4.new_notebook()
+    nb["cells"] = []
+    for code in code_blocks:
+        if not code.startswith("python"):
+            continue
+        code = code[7:]
+        nb["cells"].append(nbf.v4.new_code_cell(code))
+    fname = os.path.basename(tutorial).split(".md")[0] + ".ipynb"
+    nbf.write(nb, f".vuepress/public/notebooks/{fname}")
+
+
 if __name__ == "__main__":
 
     assert (


Test patch:
Repository: PrefectHQ/prefect

Issue:
`client.set_secret()` method
For usability and to mirror UI functionality, we should expose a set_secret method on the Client.  Bonus points for implementing encryption from a key provided in `config.toml`


Discussion:

Resolving patch:
diff --git a/src/prefect/client/client.py b/src/prefect/client/client.py
--- a/src/prefect/client/client.py
+++ b/src/prefect/client/client.py
@@ -4,7 +4,7 @@
 import json
 import logging
 import os
-from typing import TYPE_CHECKING, Optional, Union
+from typing import Any, TYPE_CHECKING, Optional, Union
 
 import prefect
 from prefect.utilities.graphql import (
@@ -125,7 +125,7 @@ def post(self, path: str, server: str = None, **params: BuiltIn) -> dict:
         else:
             return {}
 
-    def graphql(self, query: str, **variables: Union[bool, dict, str]) -> dict:
+    def graphql(self, query: str, **variables: Union[bool, dict, str]) -> GraphQLResult:
         """
         Convenience function for running queries against the Prefect GraphQL API
 
@@ -579,3 +579,24 @@ def set_task_run_state(
         return self.graphql(  # type: ignore
             parse_graphql(mutation), state=serialized_state
         ).setTaskRunState.task_run
+
+    def set_secret(self, name: str, value: Any) -> GraphQLResult:
+        """
+        Set a secret with the given name and value.
+
+        Args:
+            - name (str): the name of the secret; used for retrieving the secret
+                during task runs
+            - value (Any): the value of the secret
+
+        Returns:
+            - GraphQLResult: the resulting GraphQL result from the mutation
+        """
+        mutation = {
+            "mutation": {
+                with_args(
+                    "setSecret", {"input": dict(name=name, value=value)}
+                ): "success"
+            }
+        }
+        return self.graphql(parse_graphql(mutation))


Test patch:
Repository: PrefectHQ/prefect

Issue:
User configs shouldn't be entirely copied into Containers
User configs will tend to have things like local secrets that I don't think should be copied into Docker Containers.  We should have an explicit list of what we need to pull from user configs.


Discussion:
That's an excellent point. Alternatively, we could accept an explicit config file/path/env var and leave it up to the user.
In fact, on further reflection there's only _one single thing_ I can think of that we need to pull from a user's config file: the `[context]` section, ignoring `[context.secrets]`.  All other endpoints / executor settings should be set at the agent / deployment level, no?  

**cc:** @jlowin @joshmeek 
Even so, I'm tempted to say pull nothing automatically. I want to avoid a situation where a user has debug information in their local config and doesn't realize it's been loaded.
Yea, I like that; so are we cool if I strip out the copying of the user config from our container environments and then iteratively add things as they come up?
Yeah taking it out now is fine with me

Resolving patch:
diff --git a/src/prefect/environments.py b/src/prefect/environments.py
--- a/src/prefect/environments.py
+++ b/src/prefect/environments.py
@@ -276,17 +276,6 @@ def build(
 
             self.pull_image()
 
-            path = config.user_config_path
-
-            if Path(path).is_file():
-                config_data = toml.load(path)
-            else:
-                config_data = {}
-
-            # Make copy of config.toml in temporary directory
-            with open("{}/config.toml".format(tempdir), "w") as config_file:
-                toml.dump(config_data, config_file)
-
             self.create_dockerfile(flow=flow, directory=tempdir)
 
             client = docker.from_env()
@@ -433,7 +422,6 @@ def create_dockerfile(self, flow: "prefect.Flow", directory: str = None) -> None
 
                 RUN mkdir /root/.prefect/
                 COPY flow_env.prefect /root/.prefect/flow_env.prefect
-                COPY config.toml /root/.prefect/config.toml
                 {copy_files}
 
                 ENV PREFECT_ENVIRONMENT_FILE="/root/.prefect/flow_env.prefect"


Test patch:
diff --git a/tests/test_environments.py b/tests/test_environments.py
--- a/tests/test_environments.py
+++ b/tests/test_environments.py
@@ -75,7 +75,6 @@ def test_basic_create_dockerfile(self):
         assert " FROM python:3.6" not in dockerfile
         assert "RUN pip install ./prefect" in dockerfile
         assert "RUN mkdir /root/.prefect/" in dockerfile
-        assert "COPY config.toml /root/.prefect/config.toml" in dockerfile
 
     def test_create_dockerfile_with_environment_variables(self):
         container = ContainerEnvironment(
Repository: PrefectHQ/prefect

Issue:
All logs aren't making it through
It seems that the QueueHandler for our logs doesn't block correctly, so some logs don't make it to our logger service.


Discussion:
We simply need to call `.stop()` on the [listener](https://github.com/PrefectHQ/prefect/blob/master/src/prefect/utilities/logging.py#L58), but refactoring to do that smoothly is the tricky part.

Resolving patch:
diff --git a/src/prefect/utilities/logging.py b/src/prefect/utilities/logging.py
--- a/src/prefect/utilities/logging.py
+++ b/src/prefect/utilities/logging.py
@@ -1,4 +1,5 @@
 # Licensed under LICENSE.md; also available at https://www.prefect.io/licenses/alpha-eula
+import atexit
 import logging
 import os
 import queue
@@ -56,6 +57,8 @@ def configure_logging() -> logging.Logger:
         remote_listener = QueueListener(log_queue, remote_handler)
         logger.addHandler(queue_handler)
         remote_listener.start()
+        stopper = lambda listener: listener.stop()
+        atexit.register(stopper, remote_listener)
 
     return logger
 


Test patch:
Repository: PrefectHQ/prefect

Issue:
Overwrite previous line when pushing for `ContainerEnvironment` deploy
Currently, the streaming docker output for `ContainerEnvironment`s (which is awesome and immensely helpful) starts to get a little noisy when Pushing (I think the standard docker out overwrites the Pushing line as it progresses), whereas we keep printing new lines:
```
Pushing [====================>                              ]   22.3MB/55.01MB
Pushing [====================>                              ]  22.85MB/55.01MB
Pushing [=====================>                             ]  23.97MB/55.01MB
Pushing [======================>                            ]  24.52MB/55.01MB
Pushing [=======================>                           ]  25.63MB/55.01MB
```
It would be nice if we updated this so that any line that begins with `Pushing` is only temporarily printed and cleared with the next print.


Discussion:
Yeah I completely agree, `tqdm` seems like the best option. However it will require a lot of time to play around with because the generator output from `push` is really weird in format
As a quick fix, the print function (in Py3) takes an `end` argument which can be made the carriage return:

```
import time

# prints all numbers, overwriting the previous one on the line
for i in range(10):
    time.sleep(0.2)
    print(i, end='\r')
```

And it would be nice if something like ~5-10 lines were displayed (so maybe pull a batch of 10, join with newlines, and then overwrite previous 10 lines)
Note: my previous suggestion is possible but probably more effort than it's worth, because it would require us to actually manipulate the screen using `curses`; we should stick to 1 line for now.

Resolving patch:
diff --git a/src/prefect/environments.py b/src/prefect/environments.py
--- a/src/prefect/environments.py
+++ b/src/prefect/environments.py
@@ -373,7 +373,7 @@ def push(self, image_name: str, image_tag: str) -> None:
         output = client.push(image_name, tag=image_tag, stream=True, decode=True)
         for line in output:
             if line.get("progress"):
-                print(line.get("status"), line.get("progress"))
+                print(line.get("status"), line.get("progress"), end="\r")
 
     def pull_image(self) -> None:
         """Pull the image specified so it can be built.
@@ -387,7 +387,7 @@ def pull_image(self) -> None:
         output = client.pull(self.base_image, stream=True, decode=True)
         for line in output:
             if line.get("progress"):
-                print(line.get("status"), line.get("progress"))
+                print(line.get("status"), line.get("progress"), end="\r")
 
     def create_dockerfile(self, flow: "prefect.Flow", directory: str = None) -> None:
         """Creates a dockerfile to use as the container.


Test patch:
Repository: PrefectHQ/prefect

Issue:
Add timestamp to docs
In a call I had today, it was suggested that it would be useful to add a timestamp to the docs, in addition to the link to the commit that generated them. Currently this only exists in the API reference.


Discussion:

Resolving patch:
diff --git a/docs/generate_docs.py b/docs/generate_docs.py
--- a/docs/generate_docs.py
+++ b/docs/generate_docs.py
@@ -22,6 +22,7 @@
 from functools import partial
 
 import nbformat as nbf
+import pendulum
 import toml
 import toolz
 
@@ -320,9 +321,13 @@ def create_tutorial_notebooks(tutorial):
     GIT_SHA = os.getenv("GIT_SHA", "0000000")
     SHORT_SHA = GIT_SHA[:7]
     auto_generated_footer = (
-        "<hr>\n\n<p><small><i>This documentation was auto-generated from "
-        "<a href='https://github.com/PrefectHQ/prefect/commit/{git_sha}'>{short_sha}</a>"
-        "</i></small></p>".format(short_sha=SHORT_SHA, git_sha=GIT_SHA)
+        '<p style="text-align:center;"i><small><i>This documentation was auto-generated from commit '
+        "<a href='https://github.com/PrefectHQ/prefect/commit/{git_sha}'>{short_sha}</a> "
+        "on {timestamp}</i></small></p>".format(
+            short_sha=SHORT_SHA,
+            git_sha=GIT_SHA,
+            timestamp=pendulum.now("utc").format("MMMM D, YYYY [at] HH:mm:ss [UTC]"),
+        )
     )
 
     front_matter = textwrap.dedent(
@@ -352,9 +357,14 @@ def create_tutorial_notebooks(tutorial):
         )
         f.write("# API Reference\n")
         f.write(
-            "*This documentation was auto-generated from "
-            "[{short_sha}](https://github.com/PrefectHQ/prefect/commit/{git_sha})*".format(
-                short_sha=SHORT_SHA, git_sha=GIT_SHA
+            "*This documentation was auto-generated from commit "
+            "[{short_sha}](https://github.com/PrefectHQ/prefect/commit/{git_sha}) "
+            "on {timestamp}.*".format(
+                short_sha=SHORT_SHA,
+                git_sha=GIT_SHA,
+                timestamp=pendulum.now("utc").format(
+                    "MMMM D, YYYY [at] HH:mm:ss [UTC]"
+                ),
             )
         )
         f.write(


Test patch:
Repository: PrefectHQ/prefect

Issue:
Bad HTML generation at the bottom of the changelog in the docs
![image](https://user-images.githubusercontent.com/153965/52893320-1549dc80-3169-11e9-81b6-41514675bb17.png)

Looks like something is getting messed up between the footer insert and the changelog markdown -- this only happens on this page AFAIK


Discussion:

Resolving patch:
diff --git a/docs/generate_docs.py b/docs/generate_docs.py
--- a/docs/generate_docs.py
+++ b/docs/generate_docs.py
@@ -321,7 +321,7 @@ def create_tutorial_notebooks(tutorial):
     GIT_SHA = os.getenv("GIT_SHA", "0000000")
     SHORT_SHA = GIT_SHA[:7]
     auto_generated_footer = (
-        '<p style="text-align:center;"i><small><i>This documentation was auto-generated from commit '
+        '\n<p style="text-align:center;"i><small><i>This documentation was auto-generated from commit '
         "<a href='https://github.com/PrefectHQ/prefect/commit/{git_sha}'>{short_sha}</a> "
         "on {timestamp}</i></small></p>".format(
             short_sha=SHORT_SHA,


Test patch: