internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: PrefectHQ/prefect

Issue:
Write More Idioms
We should write some more idioms:

- [x] how to define conditional logic using the [new conditional api](https://github.com/PrefectHQ/prefect/pull/2443) and the "old" way
- [x] how to use `target`s (0.11.0+)
- [x] how to configure notifications (three options: write a downstream task, state handler, cloud hook)


Discussion:
Related to #2580. Combining them here:

> Spin off from pycon 2020 sprints with some idioms that pycon contributors might like to try. These would be idioms added in the style of the other Prefect Idioms at https://docs.prefect.io/core/idioms/idioms.html:
> 
- [ ] How do I perform a backfill?
- [ ] What's available in context during a flow run?
- [ ] How do I inspect the output from my tasks?
- [ ] How do I use multiple schedules for a flow?



Resolving patch:
diff --git a/src/prefect/tasks/control_flow/conditional.py b/src/prefect/tasks/control_flow/conditional.py
--- a/src/prefect/tasks/control_flow/conditional.py
+++ b/src/prefect/tasks/control_flow/conditional.py
@@ -4,7 +4,7 @@
 from prefect import Task
 from prefect.engine import signals
 
-__all__ = ["switch", "ifelse"]
+__all__ = ["switch", "ifelse", "merge"]
 
 
 class Merge(Task):
@@ -119,7 +119,7 @@ def as_bool(x):
         switch(condition=as_bool(condition), cases=cases)
 
 
-def merge(*tasks: Task) -> Task:
+def merge(*tasks: Task, flow=None) -> Task:
     """
     Merges conditional branches back together.
 
@@ -145,9 +145,13 @@ def merge(*tasks: Task) -> Task:
         - *tasks (Task): tasks whose results should be merged into a single result. The tasks are
             assumed to all sit downstream of different `switch` branches, such that only
             one of them will contain a result and the others will all be skipped.
+        - flow (Flow, optional): The flow to use, defaults to the current flow
+            in context if no flow is specified
 
     Returns:
         - Task: a Task representing the merged result.
 
     """
-    return Merge().bind(**{"task_{}".format(i + 1): t for i, t in enumerate(tasks)})
+    return Merge().bind(
+        **{"task_{}".format(i + 1): t for i, t in enumerate(tasks)}, flow=flow
+    )


Test patch:
diff --git a/tests/tasks/test_control_flow.py b/tests/tasks/test_control_flow.py
--- a/tests/tasks/test_control_flow.py
+++ b/tests/tasks/test_control_flow.py
@@ -145,6 +145,25 @@ def test_merging_diamond_flow():
         assert isinstance(state.result[merge_task], Success)
 
 
+def test_merge_imperative_flow():
+    flow = Flow("test")
+
+    cond = identity.copy().bind(True, flow=flow)
+    with case(cond, True):
+        a = inc.copy().bind(1, flow=flow)
+
+    with case(cond, False):
+        b = inc.copy().bind(2, flow=flow)
+
+    c = merge(a, b, flow=flow)
+
+    state = flow.run()
+    assert state.result[cond].result == True
+    assert state.result[a].result == 2
+    assert state.result[b].is_skipped()
+    assert state.result[c].result == 2
+
+
 def test_merging_with_objects_that_cant_be_equality_compared():
     class SpecialObject:
         def __eq__(self, other):
Repository: PrefectHQ/prefect

Issue:
0.12.0 Task Names not inferred from function name
## Opened from the [Prefect Public Slack Community](https://join.slack.com/t/prefect-public/shared_invite/enQtNzE5OTU3OTQwNzc1LTQ5M2FkZmQzZjI0ODg1ZTBmOTc0ZjVjYWFjMWExZDAyYzBmYjVmMTE1NTQ1Y2IxZTllOTc4MmI3NzYxMDlhYWU)

**jmaxfieldbrown**: Just curious. After updating to the most recent version of prefect `0.12.0` , all tasks on `dask` / `bokeh` are labeled as "run_task" instead of the function name. Did the `0.12.0` update make it so I need to explicitly attach a name to each task?

`bokeh==2.0.2`
`dask==2.18.1`
`distributed==2.18.0`

**dylan**: Hey <@URRN7598X>!

You’ve found a bug :tada:

Thank you for taking the time to tell us about it. We really appreciate it!

**dylan**: <@ULVA73B9P> open “0.12.0 Task Names not inferred from function name”

Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1592496831014500?thread_ts=1592496831.014500&cid=CL09KU1K7).




Discussion:

Resolving patch:
diff --git a/src/prefect/engine/executors/dask.py b/src/prefect/engine/executors/dask.py
--- a/src/prefect/engine/executors/dask.py
+++ b/src/prefect/engine/executors/dask.py
@@ -255,7 +255,7 @@ def submit(
         from distributed import fire_and_forget, worker_client
 
         extra_context = extra_context or {}
-        task_name = extra_context.get("task_full_name", "")
+        task_name = extra_context.get("task_name", "")
         task_tags = extra_context.get("task_tags", [])
         dask_kwargs = self._prep_dask_kwargs(task_name=task_name, task_tags=task_tags)
         kwargs.update(dask_kwargs)


Test patch:
diff --git a/tests/engine/executors/test_executors.py b/tests/engine/executors/test_executors.py
--- a/tests/engine/executors/test_executors.py
+++ b/tests/engine/executors/test_executors.py
@@ -296,6 +296,13 @@ def test_prep_dask_kwargs(self):
         assert kwargs["key"].startswith("FISH!")
         assert kwargs["resources"] == {"GPU": 1.0}
 
+    def test_submit_sets_task_name(self, mthread):
+        with mthread.start():
+            fut = mthread.submit(lambda x: x + 1, 1, extra_context={"task_name": "inc"})
+            (res,) = mthread.wait([fut])
+            assert fut.key.startswith("inc-")
+            assert res == 2
+
     @pytest.mark.parametrize("executor", ["mproc", "mthread"], indirect=True)
     def test_is_pickleable(self, executor):
         post = cloudpickle.loads(cloudpickle.dumps(executor))
Repository: PrefectHQ/prefect

Issue:
Prefect Docker Agent Continuously Launches Containers when using Concurrency Tags
## Description

Hi everyone, I have been testing with concurrency tags and had a question about some behavior I am seeing with docker. It seems that after a task has been waiting (via concurrency tag limit) for 10 minutes, the prefect docker agent starts continuously spinning up new containers for that flow that is waiting to start. If I do a “docker ps” I could see 20 or more containers running for that flow.

## Expected Behavior
While a flow is paused due to a concurrency tag limit, new docker containers are not continuously spun up by the agent.


## Reproduction
It should be fairly simple to reproduce. 
1. Create two flows that have tasks that take over 10 minutes to run. 
2. Assign those long tasks to use the same concurrency tag.
3. Spin up one prefect docker agent to run the flows.
4. After 10 minutes check docker with "docker ps".

## Environment

Prefect Version: 0.10.6
Docker Engine Version: 19.03.6-ce
Storage Type: Docker
OS: Replicated the issue on both Mac and Amazon Linux 2


Optionally run `prefect diagnostics` from the command line and paste the information here. -->



Discussion:
Tagging @zdhughes for any ideas on this. All the agent is doing is grabbing flow runs from the `getRunsInQueue` mutation so something must be enqueueing them.

Resolving patch:
diff --git a/src/prefect/engine/state.py b/src/prefect/engine/state.py
--- a/src/prefect/engine/state.py
+++ b/src/prefect/engine/state.py
@@ -248,7 +248,7 @@ def is_scheduled(self) -> bool:
         Returns:
             - bool: `True` if the state is skipped, `False` otherwise
         """
-        return isinstance(self, Scheduled)
+        return isinstance(self, (Scheduled, Queued))
 
     def is_submitted(self) -> bool:
         """


Test patch:
diff --git a/tests/engine/test_state.py b/tests/engine/test_state.py
--- a/tests/engine/test_state.py
+++ b/tests/engine/test_state.py
@@ -199,7 +199,6 @@ def test_only_scheduled_states_have_task_run_count_in_context(cls):
         assert state.context["task_run_count"] == 910
     else:
         assert not isinstance(state, Scheduled)
-        assert not state.is_scheduled()
 
 
 def test_retry_stores_loop_count():
@@ -423,7 +422,9 @@ def test_paused_is_scheduled(self):
         dict(state=Mapped(), assert_true={"is_finished", "is_mapped", "is_successful"}),
         dict(state=Paused(), assert_true={"is_pending", "is_scheduled"}),
         dict(state=Pending(), assert_true={"is_pending"}),
-        dict(state=Queued(), assert_true={"is_meta_state", "is_queued"}),
+        dict(
+            state=Queued(), assert_true={"is_meta_state", "is_queued", "is_scheduled"}
+        ),
         dict(state=Resume(), assert_true={"is_pending", "is_scheduled"}),
         dict(
             state=Retrying(), assert_true={"is_pending", "is_scheduled", "is_retrying"}
Repository: PrefectHQ/prefect

Issue:
Add MySQL to readme
Following https://github.com/PrefectHQ/prefect/issues/2124


Discussion:
Also Great Expectations and Google Sheets

Resolving patch:
diff --git a/src/prefect/contrib/tasks/mysql/mysql.py b/src/prefect/contrib/tasks/mysql/mysql.py
--- a/src/prefect/contrib/tasks/mysql/mysql.py
+++ b/src/prefect/contrib/tasks/mysql/mysql.py
@@ -9,6 +9,7 @@
 class MySQLExecute(Task):
     """
     Task for executing a query against a MySQL database.
+
     Args:
         - db_name (str): name of MySQL database
         - user (str): user name used to authenticate


Test patch:
Repository: PrefectHQ/prefect

Issue:
Undefined name: make_env in ./server/src/prefect_server/cli/dev.py
## Description
*A clear description of the bug*

An _undefined name_ like #2235 and #1199
https://github.com/PrefectHQ/prefect/blob/master/server/src/prefect_server/cli/dev.py#L88

`make_env` is an undefined name in this context which will raise a `NameError` at runtime.  Should this be `make_dev_env()` defined on line 36 or is `from prefect.cli.server import make_env` the right solution?

[flake8](http://flake8.pycqa.org) testing of https://github.com/PrefectHQ/prefect on Python 3.8.3

$ __flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics__
```
./server/src/prefect_server/cli/dev.py:88:11: F821 undefined name 'make_env'
    env = make_env()
          ^
1     F821 undefined name 'make_env'
1
```
https://flake8.pycqa.org/en/latest/user/error-codes.html

On the flake8 test selection, this PR does _not_ focus on "_style violations_" (the majority of flake8 error codes that [__psf/black__](https://github.com/psf/black) can autocorrect).  Instead these tests are focus on runtime safety and correctness:
* E9 tests are about Python syntax errors usually raised because flake8 can not build an Abstract Syntax Tree (AST).  Often these issues are a sign of unused code or code that has not been ported to Python 3.  These would be compile-time errors in a compiled language but in a dynamic language like Python they result in the script halting/crashing on the user.
* F63 tests are usually about the confusion between identity and equality in Python.  Use ==/!= to compare str, bytes, and int literals is the classic case.  These are areas where __a == b__ is True but __a is b__ is False (or vice versa).  Python >= 3.8 will raise SyntaxWarnings on these instances.
* F7 tests logic errors and syntax errors in type hints
* F82 tests are almost always _undefined names_ which are usually a sign of a typo, missing imports, or code that has not been ported to Python 3.  These also would be compile-time errors in a compiled language but in Python a __NameError__ is raised which will halt/crash the script on the user.




## Expected Behavior
*What did you expect to happen instead?*




## Reproduction
*A minimal example that exhibits the behavior.*




## Environment
*Any additional information about your environment*

*Optionally run `prefect diagnostics` from the command line and paste the information here*



Discussion:

Resolving patch:
diff --git a/server/src/prefect_server/cli/dev.py b/server/src/prefect_server/cli/dev.py
--- a/server/src/prefect_server/cli/dev.py
+++ b/server/src/prefect_server/cli/dev.py
@@ -85,7 +85,7 @@ def build(version):
     """
     docker_dir = Path(prefect_server.__file__).parents[2] / "docker"
 
-    env = make_env()
+    env = make_dev_env()
 
     if "PREFECT_SERVER_TAG" not in env:
         env.update(PREFECT_SERVER_TAG=version)


Test patch:
Repository: PrefectHQ/prefect

Issue:
Issue with viewing a schematic for example flow
## Archived from the [Prefect Public Slack Community](https://join.slack.com/t/prefect-public/shared_invite/enQtNzE5OTU3OTQwNzc1LTQ5M2FkZmQzZjI0ODg1ZTBmOTc0ZjVjYWFjMWExZDAyYzBmYjVmMTE1NTQ1Y2IxZTllOTc4MmI3NzYxMDlhYWU)

**louisguitton93**: hi everyone,
I'm trying to visualise in the UI the spacy_nlp.py DAG from the examples. I can see the DAG for etl.py just fine.
I'm wondering what I'm doing wrong.
Code to reproduce

```
git clone --depth 1 <https://github.com/PrefectHQ/prefect.git>
cd prefect/examples
sed -i '' 's/flow.run/flow.register/g' etl.py
sed -i '' 's/flow.run/flow.register/g' spacy_nlp.py
prefect server start
python etl.py
python spacy_nlp.py
```


**jeremiah**: Hi Louis, I’m having trouble replicating the setup — however, prior to running `python spacy_nlp.py` i had to run `python -m spacy download en_core_web_sm` in order to make the model available; otherwise the last step had an error. After that, `python spacy_nlp.py` ran and I see:

**louisguitton93**: that's right. I see it too, and it runs fine with an agent. However I can't see the Schematic (that's what I meant sorry if I wasn't clear)
<http://localhost:8080/flow/xxxx-flow-id-xxxx?schematic=>

**jeremiah**: Ah, my apologies for misunderstanding - yes that IS an issue and thanks for raising it, I see it too. I’m sure <@UN6FTLFAS> will get to the bottom of it. I guess one of these tasks must be unexpectedly missing a property.

**jeremiah**: <@ULVA73B9P> archive “Issue with viewing a schematic for example flow”




Discussion:
@znicholasbrown just pinging to see if you saw anything about this while you've been working on schematics lately?
@lauralorenz I think this issue should be resolved in the latest iteration but will double-check before that's released.

Resolving patch:
diff --git a/examples/spacy_nlp.py b/examples/spacy_nlp.py
deleted file mode 100644
--- a/examples/spacy_nlp.py
+++ /dev/null
@@ -1,41 +0,0 @@
-"""
-This example walks through the basics of using Prefect tasks to run spaCy pipelines and interact with components.
-"""
-
-import spacy
-
-from prefect import Flow
-from prefect.tasks.spacy.spacy_tasks import (
-    SpacyComponent,
-    SpacyNER,
-    SpacyNLP,
-    SpacyParser,
-    SpacyTagger,
-)
-
-# load a spacy language model
-nlp = spacy.load("en_core_web_sm")
-
-# add a custom component to language model
-def custom_component(doc):
-    # do something with document
-    return doc
-
-
-nlp.add_pipe(custom_component, name="custom")
-
-# create flow for NLP
-with Flow("Natural Language Processing") as flow:
-
-    # create a spaCy doc from text, the equivalent of nlp('This is some text')
-    doc = SpacyNLP(text="This is some text", nlp=nlp)
-
-    # extract default components from language model pipeline
-    tagger = SpacyTagger(nlp)
-    parser = SpacyParser(nlp)
-    ner = SpacyNER(nlp)
-
-    # extract a custom component from language model pipeline by name
-    custom_pipeline_component = SpacyComponent("custom")
-
-flow.run()


Test patch:
Repository: PrefectHQ/prefect

Issue:
Setting `flow_run_id` in local runs breaks slack notifier
#2868 made some additions to local runs so that targets/results could be templated properly both locally and in runs using a backend API. Setting `flow_run_id` on local runs here:

https://github.com/PrefectHQ/prefect/blob/30c88b315c83748b114228703d42bfe3e7d6a763/src/prefect/core/flow.py#L964-L969

breaks the `slack_notifier` because it has an extra check for a `flow_run_id` in context.

https://github.com/PrefectHQ/prefect/blob/30c88b315c83748b114228703d42bfe3e7d6a763/src/prefect/utilities/notifications/notifications.py#L151

I think this should be resolved by making the backend functionality of the slack notifier opt-in instead of default when a flow_run_id is present.


Discussion:

Resolving patch:
diff --git a/src/prefect/utilities/notifications/notifications.py b/src/prefect/utilities/notifications/notifications.py
--- a/src/prefect/utilities/notifications/notifications.py
+++ b/src/prefect/utilities/notifications/notifications.py
@@ -122,7 +122,9 @@ def email_message_formatter(
 
 
 def slack_message_formatter(
-    tracked_obj: TrackedObjectType, state: "prefect.engine.state.State"
+    tracked_obj: TrackedObjectType,
+    state: "prefect.engine.state.State",
+    backend_info: bool = True,
 ) -> dict:
     # see https://api.slack.com/docs/message-attachments
     fields = []
@@ -148,7 +150,7 @@ def slack_message_formatter(
         "footer": "Prefect notification",
     }
 
-    if prefect.context.get("flow_run_id"):
+    if backend_info and prefect.context.get("flow_run_id"):
         url = None
 
         if isinstance(tracked_obj, prefect.Flow):
@@ -247,6 +249,7 @@ def slack_notifier(
     ignore_states: list = None,
     only_states: list = None,
     webhook_secret: str = None,
+    backend_info: bool = True,
 ) -> "prefect.engine.state.State":
     """
     Slack state change handler; requires having the Prefect slack app installed.  Works as a
@@ -267,6 +270,8 @@ def slack_notifier(
             classes
         - webhook_secret (str, optional): the name of the Prefect Secret that stores your slack
             webhook URL; defaults to `"SLACK_WEBHOOK_URL"`
+        - backend_info (bool, optional): Whether to supply slack notification with urls
+            pointing to backend pages; defaults to True
 
     Returns:
         - State: the `new_state` object that was provided
@@ -302,7 +307,7 @@ def add(x, y):
     # the 'import prefect' time low
     import requests
 
-    form_data = slack_message_formatter(tracked_obj, new_state)
+    form_data = slack_message_formatter(tracked_obj, new_state, backend_info)
     r = requests.post(webhook_url, json=form_data)
     if not r.ok:
         raise ValueError("Slack notification for {} failed".format(tracked_obj))


Test patch:
Repository: PrefectHQ/prefect

Issue:
Add option to remove imagePullSecrets for Azure AKS
## Current behavior

AKS has a build-in system for managing imagePullSecrets for connecting to Azure ACR. This means that when deploying from that private ACR no imagePullSecrets needs to supplied (This is the standard way of setting up AKS. I don't know if this is the case for other cloud providers). However Prefect overwrites this as it sets following which causes jobs to fail. Combined with #3141 this results in a semi-silent fail for the user.

```   
imagePullSecrets:
      - name: ""
```
 

## Proposed behavior
Remove imagePullSecret from deploy file if it is set to "". I wouldn't expect this would lead to breaking changes.



## Example
Would not require user of AKS to setup a Docker Reg Secret in there cluster simplifying the deploy. 



Discussion:

Resolving patch:
diff --git a/src/prefect/agent/kubernetes/agent.py b/src/prefect/agent/kubernetes/agent.py
--- a/src/prefect/agent/kubernetes/agent.py
+++ b/src/prefect/agent/kubernetes/agent.py
@@ -197,9 +197,13 @@ def replace_job_spec_yaml(self, flow_run: GraphQLResult, image: str) -> dict:
             env.append(dict(name=key, value=value))
 
         # Use image pull secrets if provided
-        job["spec"]["template"]["spec"]["imagePullSecrets"][0]["name"] = os.getenv(
-            "IMAGE_PULL_SECRETS", ""
-        )
+        image_pull_secrets = os.getenv("IMAGE_PULL_SECRETS")
+        if image_pull_secrets:
+            job["spec"]["template"]["spec"]["imagePullSecrets"][0][
+                "name"
+            ] = image_pull_secrets
+        else:
+            del job["spec"]["template"]["spec"]["imagePullSecrets"]
 
         # Set resource requirements if provided
         resources = job["spec"]["template"]["spec"]["containers"][0]["resources"]


Test patch:
diff --git a/tests/agent/test_k8s_agent.py b/tests/agent/test_k8s_agent.py
--- a/tests/agent/test_k8s_agent.py
+++ b/tests/agent/test_k8s_agent.py
@@ -351,7 +351,7 @@ def test_k8s_agent_replace_yaml_no_pull_secrets(monkeypatch, cloud_api):
     agent = KubernetesAgent()
     job = agent.replace_job_spec_yaml(flow_run, image="test/name:tag")
 
-    assert not job["spec"]["template"]["spec"]["imagePullSecrets"][0]["name"]
+    assert not job["spec"]["template"]["spec"].get("imagePullSecrets", None)
 
 
 def test_k8s_agent_includes_agent_labels_in_job(monkeypatch, cloud_api):
Repository: PrefectHQ/prefect

Issue:
Support ellipses in GraphQL DSL
Currently, we support GraphQL syntax similar to:

```python
{
  'query': {
    'abc': True
    'xyz': {
        id
    }
  }
}
```

Notice how 'abc' must supply `True` in order to comply with Python syntax. We could consider also making `...` a valid argument, for aesthetic reasons.


```python
{
  'query': {
    'abc': ...
    'xyz': {
        id
    }
  }
}
```

Note these are both equivalent to:

```python
{
  'query': [
    'abc',
    {'xyz': {
        id
    }}
  ]
}
```



Discussion:
Hi, I was looking to start with contributing to open source.
Went through Prefect issues and saw this unpicked.

Would it be possible for me to pick this up?
I have gone through the code and I believe this parer would be needed to be updated for the asked support?

https://github.com/PrefectHQ/prefect/blob/e1b2a585c6d11b85090a936655b29f1337ae5548/src/prefect/utilities/graphql.py#L169-L198

Resolving patch:
diff --git a/src/prefect/utilities/graphql.py b/src/prefect/utilities/graphql.py
--- a/src/prefect/utilities/graphql.py
+++ b/src/prefect/utilities/graphql.py
@@ -117,14 +117,14 @@ def parse_graphql(document: Any) -> str:
     }
     ```
 
-    For convenience, if a dictionary key is True, it is ignored and the key alone is used as
-    a field name
+    For convenience, if a dictionary key is either True or Ellipsis(...), it is ignored and
+    the key alone is used as a field name.
 
     ```python
     {'query':{
         'books': {
             'id': True,
-            'name': True,
+            'name': ...,
             'author': {
                 'id',
                 'name',
@@ -177,7 +177,7 @@ def _parse_graphql_inner(document: Any, delimiter: str) -> str:
     elif isinstance(document, dict):
         result = []
         for key, value in document.items():
-            if value is True:
+            if value in (True, Ellipsis):
                 result.append(key)
             else:
                 result.append(


Test patch:
diff --git a/tests/utilities/test_graphql.py b/tests/utilities/test_graphql.py
--- a/tests/utilities/test_graphql.py
+++ b/tests/utilities/test_graphql.py
@@ -335,6 +335,26 @@ def test_use_true_to_indicate_field_name():
     )
 
 
+def test_use_ellipsis_to_indicate_field_name():
+    inner = OrderedDict()
+    inner["id"] = ...
+    inner["authors"] = {"id"}
+
+    verify(
+        query={"query": {"books": inner}},
+        expected="""
+            query {
+                books {
+                    id
+                    authors {
+                        id
+                    }
+                }
+            }
+        """,
+    )
+
+
 def test_box_query_parsing():
     verify(
         query=Box(query=Box(books={"id"})),