Repository: PrefectHQ/prefect

Issue:
`create_flow_run()` fails to run on Prefect Cloud, error when querying for tenant slug
## Description
`create_flow_run()` fails to run on Prefect Cloud. Instead, it's attempt to get the cloud url fails when sending the GraphQL query to get the tenant slug. Looks like a backend issue in how the GraphQL query is constructed. I've tested with a minimal example and the flow runs correctly when executed from a local machine (calling `create_flow_run()` to spawn subflows registered in Prefect Cloud and executed by Prefect Cloud agents. But when attempting to run the same code from Prefect Cloud `create_flow_run()` fails with the following trace.

```
Task 'create_flow_run[1]': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/prefect/engine/task_runner.py", line 865, in get_task_run_state
    logger=self.logger,
  File "/usr/local/lib/python3.6/site-packages/prefect/utilities/executors.py", line 327, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.6/site-packages/prefect/tasks/prefect/flow_run.py", line 142, in create_flow_run
    run_url = client.get_cloud_url("flow-run", flow_run_id)
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 1217, in get_cloud_url
    as_user=(as_user and using_cloud_api and self._api_token is not None)
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 1252, in get_default_tenant_slug
    res = self.graphql(query)
  File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 563, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['user'], 'message': 'field "user" not found in type: \'query_root\'', 'extensions': {'path': '$.selectionSet.user', 'code': 'validation-failed', 'exception': {'message': 'field "user" not found in type: \'query_root\''}}}]
```

## Expected Behavior
create_flow_run() should be able to spawn a subflow run whose id should then be available for further processing. This works with local runs (executed from local machine but spawning subflows via Prefect Cloud Agents).

## Reproduction
This could be simplified but the result would be the same (I think). Register flow 1 and flow2 in project `mylocal`. Execute flow3 locally, no error. Register flow3 in cloud and execute from there to reproduce.

```
from prefect import task, Flow, Parameter
from prefect.engine.results import LocalResult
from prefect import task, Flow, Parameter, apply_map, unmapped, case
from prefect.tasks.control_flow import merge
from prefect.tasks.prefect import create_flow_run, wait_for_flow_run

@task
def noop(x):
    return x

@task
def get_hit_detection_benchmark(s: str, suffix: str) -> str:
    return s+suffix

with Flow("test_crspr", result=LocalResult()) as flow1:
    x = Parameter(name="id_or_label", default='foo_experiment')
    y = x+"_in_y_task"
    z = noop(x)
    w = get_hit_detection_benchmark(z, "_crspr_flow")

with Flow("test_sf", result=LocalResult()) as flow2:
    x = Parameter(name="id_or_label", default='foo_experiment')
    y = x+"_in_y_task"
    z = noop(x)
    w = get_hit_detection_benchmark(z, "_sf_flow")

@task
def print_result(result):
    for item in result:
        print(item)

def spawn_subflows(experiment):
    parameters = {"id_or_label": experiment["experiment_label"],"feature_allow_missing": True}
    with case (experiment['base'], 'CRISPR'):
        subflow_crispr = create_flow_run(flow_name="test_crspr", project_name="mylocal", parameters=parameters)
    with case (experiment['base'], 'SF'):
        subflow_sf = create_flow_run(flow_name="test_sf", project_name="mylocal", parameters=parameters)
    return merge(subflow_crispr, subflow_sf)

with Flow("parent-flow") as flow3:
    inpt = Parameter(name="exp_list", default=[{'experiment_label':'EXP1', 'base':'CRISPR'},{'experiment_label':'EXP2', 'base':'SF'}])
    subflow_runs = apply_map(spawn_subflows, inpt)
    wait_for_flow_run.map(subflow_runs)
    print_result([inpt, subflow_runs])
```

## Environment
Prefect 0.15.1



Discussion:
Hey @pjanowski, this might be an authentication scope issue. How are you authenticating your agent with Cloud?
Let me get back to you on that in a few minutes but submitting flows in all ways including `StartFlowRun` pattern has always worked on this agent (which is a default Kubernetes on GCS agent but I've also reproduced the same error with other agents).
Here's the simpler repro

```
from prefect import task, Flow, Parameter
from prefect.engine.results import LocalResult
from prefect import task, Flow, Parameter, apply_map, unmapped, case
from prefect.tasks.control_flow import merge
from prefect.tasks.prefect import create_flow_run, wait_for_flow_run

with Flow("test_crspr", result=LocalResult()) as flow1:
    x = Parameter(name="id_or_label", default='foo_experiment')
    y = x+"_in_y_task"

@task
def print_result(result):
    for item in result:
        print(item)

with Flow("parent-flow") as flow2:
    inpt = Parameter(name="exp_list", default={"id_or_label": 'EXP1',"feature_allow_missing": True})
    subflow = create_flow_run(flow_name="test_crspr", project_name="mylocal", parameters=inpt)
    wait_for_flow_run(subflow)
    print_result([inpt, subflow])
```
The kubernetes agent is authenticating with Prefect Cloud via an environment variable populated from a k8s secret:
```
        - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
          valueFrom:
            secretKeyRef:
              name: prefect-runner-token
              key: token
```
Interesting, I can't reproduce this locally. I was suspicious of the runner scoped auth token, but I started an agent with one and `create_flow_run` worked without issue (I ran your MRE with both API key and token authentication).

Is your agent on 0.15.1 as well?
Ah nevermind; I've successfully reproduced this in a clean environment. I'll see what's going on. Switching to an API key will almost certainly resolve it in the meantime.

Resolving patch:
diff --git a/src/prefect/tasks/prefect/flow_run.py b/src/prefect/tasks/prefect/flow_run.py
--- a/src/prefect/tasks/prefect/flow_run.py
+++ b/src/prefect/tasks/prefect/flow_run.py
@@ -139,7 +139,7 @@ def create_flow_run(
         scheduled_start_time=scheduled_start_time,
     )
 
-    run_url = client.get_cloud_url("flow-run", flow_run_id)
+    run_url = client.get_cloud_url("flow-run", flow_run_id, as_user=False)
     logger.info(f"Created flow run {run_name_dsp!r}: {run_url}")
     return flow_run_id
 


Test patch:
diff --git a/tests/tasks/prefect/test_flow_run.py b/tests/tasks/prefect/test_flow_run.py
--- a/tests/tasks/prefect/test_flow_run.py
+++ b/tests/tasks/prefect/test_flow_run.py
@@ -130,7 +130,9 @@ def test_displays_flow_run_url(self, MockFlowView, MockClient, caplog):
         MockClient().create_flow_run.return_value = "flow-run-id"
         MockClient().get_cloud_url.return_value = "fake-url"
         create_flow_run.run(flow_id="flow-id")
-        MockClient().get_cloud_url.assert_called_once_with("flow-run", "flow-run-id")
+        MockClient().get_cloud_url.assert_called_once_with(
+            "flow-run", "flow-run-id", as_user=False
+        )
         assert "Created flow run '<generated-name>': fake-url" in caplog.text