internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: PrefectHQ/prefect

Issue:
Task results are sometimes persisted in undesired scenarios
## Current behavior
<!-- Please describe how the feature works today -->
In a flow like this:
```
@task(checkpoint=False)
def extract_things():
    bar = [1, 2, 3, 4, 5, 6, 7, 8, 9]
    foo = ['a', 'b', 'c']
    data = {
        'bar': bar,
        'foo': foo
    }
    return data
@task(checkpoint=False)
def do_things(data):
    for x in data:
        print(x)
    return None
with Flow(
    "Do it",
) as flow:
    things = extract_things()
    do_things(things['foo'])
    do_things(things['bar'])
```
task results for `things` are persisted because downstream tasks are accessing the result attributes directly and turning them into `Constant` tasks. This leads to undesired results when checkpoint is set to False.


## Proposed behavior
<!-- Please describe your proposed change to the current behavior -->
Either we throw a warning or prevent constants from automatically persisting results when their upstream tasks have checkpointing turned off.



## Example
<!-- Please give an example of how the enhancement would be useful -->
cc @joshmeek 


Discussion:

Resolving patch:
diff --git a/src/prefect/core/task.py b/src/prefect/core/task.py
--- a/src/prefect/core/task.py
+++ b/src/prefect/core/task.py
@@ -806,7 +806,9 @@ def __getitem__(self, key: Any) -> "Task":
         Returns:
             - Task
         """
-        return prefect.tasks.core.operators.GetItem().bind(self, key)
+        return prefect.tasks.core.operators.GetItem(
+            checkpoint=self.checkpoint, result=self.result
+        ).bind(self, key)
 
     def __or__(self, other: object) -> object:
         """


Test patch:
diff --git a/tests/core/test_task_operators.py b/tests/core/test_task_operators.py
--- a/tests/core/test_task_operators.py
+++ b/tests/core/test_task_operators.py
@@ -1,4 +1,5 @@
 from prefect.core import Edge, Flow, Parameter, Task
+from prefect.engine.results import LocalResult
 
 
 class TestMagicInteractionMethods:
@@ -23,6 +24,15 @@ def test_getitem_constant(self):
         state = f.run(parameters=dict(x=dict(a=1, b=2, c=3)))
         assert state.result[z].result == 2
 
+    def test_getitem_preserves_result_info(self):
+        with Flow(name="test") as f:
+            z = Task(checkpoint=False)()[0]
+            y = Task(checkpoint=True, result=LocalResult(dir="home"))[1]
+
+        assert z.checkpoint is False
+        assert isinstance(y.result, LocalResult)
+        assert y.result.dir.endswith("home")
+
     # -----------------------------------------
     # or / pipe / |
Repository: PrefectHQ/prefect

Issue:
Modifying Flows returned by Flow.copy modifies the original flow
## Description
If I modify (using `.replace`) a flow that was created by calling `.copy()` on an existing flow, the original flow is modified also.

## Expected Behavior
Modifying a *copy* of a flow should not modify the original flow.

## Reproduction
<!-- A minimal example that exhibits the behavior. -->
```python
@prefect.task
def f(x):
    return x

with prefect.Flow("foo") as original_flow:
    x = prefect.Parameter(name="x")
    y = f(x)
    
print(original_flow.constants)

copied_flow = original_flow.copy()
copied_flow.replace(x, 1)

print(original_flow.constants)
```

This prints
```
defaultdict(<class 'dict'>, {})
defaultdict(<class 'dict'>, {<Task: f>: {'x': 1}})
```

This means that calling `copied_flow.replace` modifies `original_flow`.


## Environment
<!-- Any additional information about your environment

Optionally run `prefect diagnostics` from the command line and paste the information here. -->
```
{
  "config_overrides": {},
  "env_vars": [],
  "system_information": {
    "platform": "Linux-3.10.0-957.27.2.el7.x86_64-x86_64-with-glibc2.10",
    "prefect_backend": "cloud",
    "prefect_version": "0.13.6",
    "python_version": "3.8.5"
  }
}
```



Discussion:
Hi @jbweston I believe this functionality is expected and intentional where `flow.copy` is not intended to perform a deep copy. Is there a reason using something like `copy.deepcopy(your_flow)` will not suffice?

Resolving patch:
diff --git a/src/prefect/core/flow.py b/src/prefect/core/flow.py
--- a/src/prefect/core/flow.py
+++ b/src/prefect/core/flow.py
@@ -230,6 +230,7 @@ def copy(self) -> "Flow":
         new = copy.copy(self)
         # create a new cache
         new._cache = dict()
+        new.constants = self.constants.copy()
         new.tasks = self.tasks.copy()
         new.edges = self.edges.copy()
         new.set_reference_tasks(self._reference_tasks)


Test patch:
diff --git a/tests/core/test_flow.py b/tests/core/test_flow.py
--- a/tests/core/test_flow.py
+++ b/tests/core/test_flow.py
@@ -586,6 +586,24 @@ def test_eager_cycle_detection_works():
     assert not prefect.config.flows.eager_edge_validation
 
 
+def test_copy_handles_constants():
+    @task
+    def f(x):
+        return x
+
+    with Flow("foo") as original_flow:
+        x = Parameter(name="x")
+        y = f(x)
+
+    assert not original_flow.constants
+
+    copied_flow = original_flow.copy()
+    copied_flow.replace(x, 1)
+
+    assert copied_flow.constants
+    assert not original_flow.constants
+
+
 def test_copy():
     with Flow(name="test") as f:
         t1 = Task()
Repository: PrefectHQ/prefect

Issue:
Raise a better error when using custom schedule filters with cloud
Currently custom filters on schedules aren't supported, since the filters run on cloud's servers. Right now the error message raised is a long `ValidationError` traceback that just notes that the function is "invalid" - we should provide a nicer error message to let users know some context around why.


Discussion:

Resolving patch:
diff --git a/src/prefect/utilities/serialization.py b/src/prefect/utilities/serialization.py
--- a/src/prefect/utilities/serialization.py
+++ b/src/prefect/utilities/serialization.py
@@ -382,19 +382,21 @@ def _serialize(self, value, attr, obj, **kwargs):  # type: ignore
 
         try:
             qual_name = to_qualified_name(value)
-        except Exception as exc:
-            raise ValidationError(
-                f"Invalid function reference, function required, got {value}"
-            ) from exc
-
-        # sort matches such that the longest / most specific match comes first
-        valid_bases = sorted(
-            [fn for fn in self.valid_functions if qual_name.startswith(fn)],
-            key=lambda k: len(k),
-            reverse=True,
-        )
+        except Exception:
+            valid_bases = []
+        else:
+            # sort matches such that the longest / most specific match comes first
+            valid_bases = sorted(
+                [fn for fn in self.valid_functions if qual_name.startswith(fn)],
+                key=lambda k: len(k),
+                reverse=True,
+            )
+
         if not valid_bases and self.reject_invalid:
-            raise ValidationError("Invalid function reference: {}".format(value))
+            raise ValidationError(
+                "When running with Prefect Cloud/Server, custom functions aren't supported "
+                f"for `{type(obj).__name__}.{attr}`. Unable to serialize `{value!r}`."
+            )
         else:
             base_name = next(filter(None, valid_bases)) if valid_bases else qual_name
 


Test patch:
diff --git a/tests/utilities/test_serialization.py b/tests/utilities/test_serialization.py
--- a/tests/utilities/test_serialization.py
+++ b/tests/utilities/test_serialization.py
@@ -240,7 +240,9 @@ def test_serialize_outer_with_state(self):
         assert serialized["f"]["kwargs"] == {"x": 1, "y": 2, "z": 99}
 
     def test_serialize_invalid_fn(self):
-        with pytest.raises(marshmallow.ValidationError):
+        with pytest.raises(
+            marshmallow.ValidationError, match="custom functions aren't supported"
+        ):
             self.Schema().dump(dict(f=fn2))
 
     def test_serialize_invalid_fn_without_validation(self):
@@ -283,7 +285,9 @@ class Foo(object):
             def __call__(self, a, b):
                 return a + b
 
-        with pytest.raises(marshmallow.ValidationError, match="function required"):
+        with pytest.raises(
+            marshmallow.ValidationError, match="custom functions aren't supported"
+        ):
             self.Schema().dump(dict(f=Foo()))
 
     def test_serialize_none(self):
Repository: PrefectHQ/prefect

Issue:
Change "PYTHONPATH" construct for local agent
## Current behavior
Local agent has the "--import-path" feature, but its now availible only for unix enviroment, because PYTHONPATH is concatenated with ":"
current_env["PYTHONPATH"] = ":".join(python_path)

## Proposed behavior
Use  "os.pathsep" instead ":", to get the correct value aswell for Windows users




Discussion:
Thanks for the report @dmitrii-dementevskii 

This would be a really good thing to open a PR for if you have the time! 

Resolving patch:
diff --git a/src/prefect/agent/local/agent.py b/src/prefect/agent/local/agent.py
--- a/src/prefect/agent/local/agent.py
+++ b/src/prefect/agent/local/agent.py
@@ -225,7 +225,7 @@ def populate_env_vars(
             python_path.append(os.environ["PYTHONPATH"])
         if self.import_paths:
             python_path.extend(self.import_paths)
-        env["PYTHONPATH"] = ":".join(python_path)
+        env["PYTHONPATH"] = os.pathsep.join(python_path)
 
         # 4. Values set on the agent via `--env`
         env.update(self.env_vars)


Test patch:
diff --git a/tests/agent/test_local_agent.py b/tests/agent/test_local_agent.py
--- a/tests/agent/test_local_agent.py
+++ b/tests/agent/test_local_agent.py
@@ -87,18 +87,22 @@ def test_local_agent_uses_ip_if_dockerdesktop_hostname(monkeypatch):
     assert "IP" in agent.labels
 
 
-def test_populate_env_vars():
+def test_populate_env_vars(monkeypatch):
     agent = LocalAgent()
 
+    # The python path may be a single item and we want to ensure the correct separator
+    # is added so we will ensure PYTHONPATH has an item in it to start
+    if not os.environ.get("PYTHONPATH", ""):
+        monkeypatch.setenv("PYTHONPATH", "foobar")
+
     env_vars = agent.populate_env_vars(
         GraphQLResult({"id": "id", "name": "name", "flow": {"id": "foo"}})
     )
-    python_path = env_vars.pop("PYTHONPATH", "")
-    assert os.getcwd() in python_path
 
     expected = os.environ.copy()
     expected.update(
         {
+            "PYTHONPATH": os.getcwd() + os.pathsep + expected.get("PYTHONPATH", ""),
             "PREFECT__CLOUD__API": "https://api.prefect.io",
             "PREFECT__CLOUD__AUTH_TOKEN": "TEST_TOKEN",
             "PREFECT__CLOUD__AGENT__LABELS": str(DEFAULT_AGENT_LABELS),
Repository: PrefectHQ/prefect

Issue:
AzureResult read function not working
## Description
<!-- A clear description of the bug -->
Using `AzureResult` to read a file, an exception is raised:

> TypeError: a bytes-like object is required, not 'StorageStreamDownloader'

Looking at the `read` function in `src/prefect/engine/results/azure_result.py` I see the following:

```python
content_string = client.download_blob()

try:
    new.value = new.serializer.deserialize(content_string)
```

`client.download_blob()` returns a `StorageStreamDownloader`, while `new.serailizer.deserialize(content_string)` expects a byte array.

## Expected Behavior
I would expect that the file would be read without raising an exception.

## Reproduction
```python
from prefect.engine.results import AzureResult

con_string = ""
azure_result = AzureResult(container="data", connection_string=con_string)

w = azure_result.write({"Test": "123"})
r = azure_result.read(w.location)
```


## Environment
```json
{
  "config_overrides": {
    "context": {
      "secrets": false
    }
  },
  "env_vars": [],
  "system_information": {
    "platform": "Windows-10-10.0.18362-SP0",
    "prefect_backend": "server",
    "prefect_version": "0.13.19",
    "python_version": "3.9.0"
  }
}
```


Discussion:
Hey @emilbbusch thanks for reporting! I know the fix and can submit a PR. 🙂 

Resolving patch:
diff --git a/src/prefect/engine/results/azure_result.py b/src/prefect/engine/results/azure_result.py
--- a/src/prefect/engine/results/azure_result.py
+++ b/src/prefect/engine/results/azure_result.py
@@ -126,10 +126,10 @@ def read(self, location: str) -> Result:
             client = self.service.get_blob_client(
                 container=self.container, blob=location
             )
-            content_string = client.download_blob()
+            content_bytes = client.download_blob().content_as_bytes()
 
             try:
-                new.value = new.serializer.deserialize(content_string)
+                new.value = new.serializer.deserialize(content_bytes)
             except EOFError:
                 new.value = None
             self.logger.debug("Finished downloading result from {}.".format(location))


Test patch:
diff --git a/tests/engine/results/test_azure_result.py b/tests/engine/results/test_azure_result.py
--- a/tests/engine/results/test_azure_result.py
+++ b/tests/engine/results/test_azure_result.py
@@ -58,7 +58,11 @@ def test_azure_writes_to_blob_using_rendered_template_name(self, monkeypatch):
         assert service.get_blob_client.call_args[1]["container"] == "foo"
 
     def test_azure_reads_and_updates_location(self, monkeypatch):
-        client = MagicMock(download_blob=MagicMock(return_value=b""))
+        client = MagicMock(
+            download_blob=MagicMock(
+                return_value=MagicMock(content_as_bytes=MagicMock(return_value=b""))
+            )
+        )
         service = MagicMock(get_blob_client=MagicMock(return_value=client))
         monkeypatch.setattr(
             "prefect.engine.results.azure_result.AzureResult.service", service
Repository: PrefectHQ/prefect

Issue:
s3 path for agent ecs start does not work
## Description
When specifying a s3 path in the task_definition or run_task_kwargs as part of `agent ecs start`, it does not work. The agent will return back a 403 error (forbidden). 

The cause seems to be from this line: 
https://github.com/PrefectHQ/prefect/blob/96ef85470872593268c9498b57ac9f0b5a268e01/src/prefect/utilities/filesystems.py#L75
The key should be something like this `a/nice/path/file.name`. But it current comes thru as `/a/nice/path/file.name` (note the leading `/`)




## Expected Behavior
A user should be able to specify a s3 path in the agent ecs run, for the task_definition and run_task_kwargs, as shown in the documentation



## Reproduction

```
agent ecs start --task_definition s3://some/file.yaml
```



## Environment
Tested on linux, but probably on all. 





Discussion:

Resolving patch:
diff --git a/src/prefect/utilities/filesystems.py b/src/prefect/utilities/filesystems.py
--- a/src/prefect/utilities/filesystems.py
+++ b/src/prefect/utilities/filesystems.py
@@ -77,7 +77,9 @@ def read_bytes_from_path(path: str) -> bytes:
 
         client = get_boto_client(resource="s3")
         stream = io.BytesIO()
-        client.download_fileobj(Bucket=parsed.netloc, Key=parsed.path, Fileobj=stream)
+        client.download_fileobj(
+            Bucket=parsed.netloc, Key=parsed.path.lstrip("/"), Fileobj=stream
+        )
         return stream.getvalue()
     else:
         raise ValueError(f"Unsupported file scheme {path}")


Test patch:
diff --git a/tests/utilities/test_filesystems.py b/tests/utilities/test_filesystems.py
--- a/tests/utilities/test_filesystems.py
+++ b/tests/utilities/test_filesystems.py
@@ -74,5 +74,5 @@ def test_read_s3(self, monkeypatch):
         )
         res = read_bytes_from_path("s3://mybucket/path/to/thing.yaml")
         assert client.download_fileobj.call_args[1]["Bucket"] == "mybucket"
-        assert client.download_fileobj.call_args[1]["Key"] == "/path/to/thing.yaml"
+        assert client.download_fileobj.call_args[1]["Key"] == "path/to/thing.yaml"
         assert isinstance(res, bytes)
Repository: PrefectHQ/prefect

Issue:
Add ‘Resume’ and other values to context docs
## Opened from the [Prefect Public Slack Community](https://join.slack.com/t/prefect-public/shared_invite/enQtNzE5OTU3OTQwNzc1LTQ5M2FkZmQzZjI0ODg1ZTBmOTc0ZjVjYWFjMWExZDAyYzBmYjVmMTE1NTQ1Y2IxZTllOTc4MmI3NzYxMDlhYWU)

**brettnaul**: quick q on Pause behavior: is there any way to know inside a task (e.g. from the prefect context) whether the current run was restarted manually (i.e. by clicking Approve in the UI)? I'm aiming for something like the `manual_only` trigger but that can be achieved at runtime instead. so something like

```
if condition and not prefect.context["manual_approval"]:
    raise signals.PAUSE()
```


**dylan**: Hey <@UL455076D>,

`resume=True` Should be in the top-level of context whenever manually restarting from Paused

**dylan**: So you should be able to `prefect.context.get("resume")` to determine if it was a manual restart

**brettnaul**: perfect! that's exactly what I was looking for; I think it's missing from <https://docs.prefect.io/api/latest/utilities/context.html> ... maybe there's all sorts of goodies I'm missing out on :sleuth_or_spy:

**dylan**: Good call!

**dylan**: <@ULVA73B9P> open “Add ‘Resume’ and other values to context docs”

Original thread can be found [here](https://prefect-community.slack.com/archives/CL09KU1K7/p1610045539393600?thread_ts=1610045539.393600&cid=CL09KU1K7).




Discussion:

Resolving patch:
diff --git a/src/prefect/utilities/context.py b/src/prefect/utilities/context.py
--- a/src/prefect/utilities/context.py
+++ b/src/prefect/utilities/context.py
@@ -55,6 +55,7 @@
 | `task_id` | the id of the current task |
 | `task_run_id` | the id of the current task run |
 | `task_run_version` | the state version of the current task run |
+| `resume` | boolean showing if the current task run was manually restarted |
 
 Users can also provide values to context at runtime. For more information, see
 the [Context concept


Test patch:
Repository: PrefectHQ/prefect

Issue:
Prefect agent continues to run but doesn't register itself in server
This portion of the code is encountered when trying to register a new agent to the server, both running in a GKE cluster.
https://github.com/PrefectHQ/prefect/blob/4971eca9fbacf0db2e8e1e65691f26ce026160ae/src/prefect/agent/agent.py#L229

What seems to happen is that GKE restarted all the pods, and the server came up at the same time the agent did, but because it failed the registration, the server doesn't show the agent as running and healthy in the UI.

This doesn't seem to be the case for the cloud, so I'd like to understand if there is a reason why the agent should not error and allow the pod to restart vs. continuing on silently. Perhaps it's a case of a missing `raise`?


Discussion:
@alexkoay I believe this line where the exception is not raised was added intentionally because there was a time where server did not yet have agent registration but then it was added in newer versions. We didn't want the agents erroring out and forcing users to upgrade their server instances. That older version of server should be long behind us now and we should now handle the exception accordingly 👍 

Resolving patch:
diff --git a/src/prefect/agent/agent.py b/src/prefect/agent/agent.py
--- a/src/prefect/agent/agent.py
+++ b/src/prefect/agent/agent.py
@@ -220,16 +220,8 @@ def start(self, _loop_intervals: dict = None) -> None:
         if config.backend == "cloud":
             self._verify_token(self.client.get_auth_token())
 
-        try:
-            self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
-        except Exception as exc:
-            if config.backend == "cloud":
-                raise exc
-            else:
-                self.logger.warning(
-                    f"Unable to register agent to {self.client.api_server}. "
-                    f"Make sure the server is running on the latest version."
-                )
+        # Register agent with backend API
+        self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
 
         try:
             self.setup()


Test patch:
Repository: PrefectHQ/prefect

Issue:
Fixes in dask_cloudprovider should be incorporated into Prefect's setup.py
Currently if a user runs a Prefect flow using the `dask_cloudprovider<2021.01.0` for ECS, dask_cloudprovider will deregister  ECS task definitions in AWS for that account.  There is a recent fix in dask_cloudprovider for this issue:

https://github.com/dask/dask-cloudprovider/pull/238

But it is only available in the latest version of dask_cloudprovider, which is explicitly disallowed in prefect's setup.py:

https://github.com/PrefectHQ/prefect/blob/529b7121e438980c9640fbba82fe1ab14a1de00b/setup.py#L50

I'd request updating the required version of dask_cloudprovider to be >= 2021.01.0. Or at a minimum allowing for dask_cloudprovider versions greater than 1.0.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -47,7 +47,7 @@ def run(self):
         "azure-cosmos >= 3.1.1, <3.2",
     ],
     "bitbucket": orchestration_extras["bitbucket"],
-    "dask_cloudprovider": ["dask_cloudprovider[aws] >= 0.2.0, < 1.0"],
+    "dask_cloudprovider": ["dask_cloudprovider[aws] >= 0.2.0"],
     "dev": dev_requires + test_requires,
     "dropbox": ["dropbox ~= 9.0"],
     "ge": ["great_expectations >= 0.11.1"],


Test patch:
Repository: PrefectHQ/prefect

Issue:
Updated doc for DaskExecutor
The doc at https://docs.prefect.io/core/idioms/parallel.html shows few lines to run the flow in  distributed manner.

However, as mentioned in issue #2468, if the `flow.run` does not happen within `if __name__ == "__main__":` block, it fails to run in local mode.

The documentation shall show the way which works.

```python
if __name__ == "__main__":
    executor = DaskExecutor()
    state = flow.run(executor=executor)
```

_Originally posted by @vlcinsky in https://github.com/PrefectHQ/prefect/issues/2460#issuecomment-752740483_


Discussion:
@jcrist For suggestions on how to handle this in that idiom. The code works fine as is in an interactive session but would fail when running from a script. Perhaps we should add a tip/warning about the `__main__`?

Resolving patch:
diff --git a/src/prefect/executors/dask.py b/src/prefect/executors/dask.py
--- a/src/prefect/executors/dask.py
+++ b/src/prefect/executors/dask.py
@@ -97,6 +97,8 @@ class name (e.g. `"distributed.LocalCluster"`), or the class itself.
             potentially useful debug info. Defaults to the `debug` value in
             your Prefect configuration.
 
+    Examples:
+
     Using a temporary local dask cluster:
 
     ```python


Test patch: