Package: kedro-graphql
kedro-graphqlOverviewKedro-graphql is akedro-pluginfor serving kedro projects as a graphql api.  It leveragesStrawberry,FastAPI,
andCeleryto turn anyKedroproject into a GraphqlQL api
with features such as:a distributed task queuesubscribe to pipline events and logs via GraphQL subscriptionsstoragepersist and track all pipelines executed via the APIadditional featuresflowchart  TB
  api[GraphQL API\n<i>strawberry + FastAPI</i>]
  mongodb[(db: 'pipelines'\ncollection: 'pipelines'\n<i>mongdob</i>)]
  redis[(task queue\n<i>redis</i>)]
  worker[worker\n<i>celery</i>]

  api<-->mongodb
  api<-->redis
  worker<-->mongodb
  worker<-->redisFigure 1. ArchitectureQuickstartInstall kedro-graphql into your kedro project environnment.pip install kedro_graphqlStart the redis and mongo services using thisdocker-compose.yaml.docker-compose up -dStart the api server.kedro gqlStart a worker (in another terminal).kedro gql -wNavigate tohttp://127.0.0.1:5000/graphqlto access the graphql interface.Thedocker-compose.yamlincludesmongo-expressandredis-commanderservices
to provide easy acess to MongoDB and redis.Navigate tohttp://127.0.0.1:8082to access mongo-express interface.Navigate tohttp://127.0.0.1:8081to access the redis-commander interface.
One can access the task queues created and managed byCelery.ExampleThe kedro-graphl package contains an very simple example
pipeline called "example00".SetupClone the kedro-graphql repository.git clone git@github.com:opensean/kedro-graphql.gitCreate a virtualenv and activate it.cd kedro-graphql
python3.10 -m venv venv
source venv/bin/activateInstall dependencies.pip install -r src/requirements.txtCreate a text file.echo "Hello" > ./data/01_raw/text_in.txtStart the redis and mongo services.docker-compose up -dStart the api server.kedro gqlStart a worker (in another terminal).kedro gql -wStart a pipelineNavigate tohttp://127.0.0.1:5000/graphqlto access the graphql interface
and execute the following mutation:mutation MyMutation {
  pipeline(
    pipeline: {name: "example00", parameters: [{name: "example", value: "hello"}, {name: "duration", value: "10"}], inputs: {name: "text_in", type: "text.TextDataSet", filepath: "./data/01_raw/text_in.txt"}, outputs: {name: "text_out", type: "text.TextDataSet", filepath: "./data/02_intermediate/text_out.txt"}}
  ) {
    id
    name
  }
}Expected response:{
  "data": {
    "pipeline": {
      "id": "6463991db98d7f8564ab15a0",
      "name": "example00"
    }
  }
}Subscribe to pipeline eventsNow execute the following subscription to track the progress:subscription MySubscription {
  pipeline(id: "6463991db98d7f8564ab15a0") {
    id
    result
    status
    taskId
    timestamp
    traceback
  }
}Susbscribe to pipeline logsExecute the following subscription to recieve log messages:subscription {
   	pipelineLogs(id:"6463991db98d7f8564ab15a0") {
       id
       message
       messageId
       taskId
       time
     }
}Get the pipeline resultFetch the pipeline result with the following query:query MyQuery {
  pipeline(id: "6463991db98d7f8564ab15a0") {
    describe
    id
    name
    outputs {
      filepath
      name
      type
    }
    inputs {
      filepath
      name
      type
    }
    parameters {
      name
      value
    }
    status
    taskEinfo
    taskException
    taskId
    taskKwargs
    taskRequest
    taskName
    taskResult
    taskTraceback
  }
}Expected result:{
  "data": {
    "pipeline": {
      "describe": "#### Pipeline execution order ####\nInputs: parameters, params:example, text_in\n\necho_node\n\nOutputs: text_out\n##################################",
      "id": "6463991db98d7f8564ab15a0",
      "name": "example00",
      "outputs": [
        {
          "filepath": "./data/02_intermediate/text_out.txt",
          "name": "text_out",
          "type": "text.TextDataSet"
        }using 
      ],
      "inputs": [
        {
          "filepath": "./data/01_raw/text_in.txt",
          "name": "text_in",
          "type": "text.TextDataSet"
        }
      ],
      "parameters": [
        {
          "name": "example",
          "value": "hello"
        },
        {
          "name": "duration",
          "value": "10"
        }
      ],
      "status": "SUCCESS",
      "taskEinfo": "None",
      "taskException": null,
      "taskId": "129b4441-6150-4c0b-90df-185c1ec692ea",
      "taskKwargs": "{'name': 'example00', 'inputs': {'text_in': {'type': 'text.TextDataSet', 'filepath': './data/01_raw/text_in.txt'}}, 'outputs': {'text_out': {'type': 'text.TextDataSet', 'filepath': './data/02_intermediate/text_out.txt'}}, 'parameters': {'example': 'hello', 'duration': '10'}}",
      "taskRequest": null,
      "taskName": "<@task: kedro_graphql.tasks.run_pipeline of kedro_graphql at 0x7f29e3e9e500>",
      "taskResult": null,
      "taskTraceback": null
    }
  }
}One can explore how the pipeline is persisted using the mongo-express
interface located herehttp://127.0.0.1:8082.  Pipelines are persisted in the
"pipelines" collection of the "pipelines" database.FeaturesExtensible APIThe api generated by this tool can be extended using decorators.This example adds a query, mutation, and subscription types.## kedro_graphql.plugins.plugins
import asyncio
from kedro_graphql.decorators import gql_query, gql_mutation, gql_subscription
import strawberry
from typing import AsyncGenerator

@gql_query()
@strawberry.type
class ExampleQueryTypePlugin():
    @strawberry.field
    def hello_world(self) -> str:
        return "Hello World"

@gql_mutation()
@strawberry.type
class ExampleMutationTypePlugin():
    @strawberry.mutation
    def hello_world(self, message: str = "World") -> str:
        return "Hello " + message

@gql_subscription()
@strawberry.type
class ExampleSubscriptionTypePlugin():
    @strawberry.subscription
    async def hello_world(self, message: str = "World", target: int = 11) -> AsyncGenerator[str, None]:
        for i in range(target):
            yield str(i) + " Hello " + message
            await asyncio.sleep(0.5)When starting the api server specify the import path using the--importsflag.kedro gql --imports "kedro_graphql.plugins.plugins"Multiple import paths can be specified using comma seperated values.kedro gql --imports "kedro_graphql.plugins.plugins,example_pkg.example.my_types"Alternatively, use a.envfile as described in theGeneral Configurationsection.Configurable ApplicationThe base application is strawberry + FastAPI instance.  One can leverage the
additional features FastAPI offers by defining a custom application class.This example adds aCORSMiddleware.## src/kedro_graphql/example/app.py
from fastapi.middleware.cors import CORSMiddleware
from kedro_graphql import KedroGraphQL



class MyApp(KedroGraphQL):

    def __init__(self): 
        super(MyApp, self).__init__()

        origins = [
            "http://localhost",
            "http://localhost:8080",
        ]
        
        self.add_middleware(
            CORSMiddleware,
            allow_origins=origins,
            allow_credentials=True,
            allow_methods=["*"],
            allow_headers=["*"],
        )
        print("added CORSMiddleware")When starting the api server specify the import path using the--appflag.kedro gql --app "my_kedro_project.app.MyApp"
## example output
added CORSMiddleware
INFO:     Started server process [7032]
INFO:     Waiting for application startup.
Connected to the MongoDB database!
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:5000 (Press CTRL+C to quit)Alternatively, use a.envfile as described in theGeneral Configurationsection.Auto-reloadThe cli interface supports "auto-reloading" in order to make development easier.
When starting the api server and worker specify the-ror--reloadoption
to turn on auto-reloading.  Any changes to the "src" directory of your kedro
project will trigger a reload.Start the api server with auto-reload enabled.kedro gql --reloadStart a worker (in another terminal) with auto-reload enabled.kedro gql -w --reloadThe path to watch can be further refined using the--reload-pathoption.
In the following examples a reload will be triggered when changes are
made to files in thesrc/kedro_graphql/src/runnersdirectory.
Start the api server with auto-reload enabled.kedro gql --reload --reload-path ./src/kedro_graphql/runnersStart a worker (in another terminal) with auto-reload enabled.kedro gql -w --reload --reload-path ./src/kedro_graphql/runnersGeneral ConfigurationConfiguration can be supplied via environment variables or a.envfile.## example .env file
MONGO_URI=mongodb://root:example@localhost:27017/
MONGO_DB_NAME=pipelines
KEDRO_GRAPHQL_IMPORTS=kedro_graphql.plugins.plugins
KEDRO_GRAPHQL_APP=kedro_graphql.asgi.KedroGraphQL
KEDRO_GRAPHQL_BACKEND=kedro_graphql.backends.mongodb.MongoBackend
KEDRO_GRAPHQL_BROKER=redis://localhost
KEDRO_GRAPHQL_CELERY_RESULT_BACKEND=redis://localhost
KEDRO_GRAPHQL_RUNNER=kedro.runner.SequentialRunner
KEDRO_GRAPHQL_ENV=local
KEDRO_GRAPHQL_CONF_SOURCE=NoneThe configuration can also be provided at startup through the cli interface.
Configuration values can be mapped to the the appropriate cli option by
removing the "KEDRO_GRAPHQL" prefix and using a lower case, hyphen format
for the remaining string.  For example:configuration variablecli optionexampleMONGO_URI--mongo-urimongodb://root:example@localhost:27017/MONGO_DB_NAME--mongo-db-namepipelinesKEDRO_GRAPHQL_IMPORTS--importskedro_graphql.plugins.pluginsKEDRO_GRAPHQL_APP--appkedro_graphql.asgi.KedroGraphQLKEDRO_GRAPHQL_BACKEND--backendkedro_graphql.backends.mongodb.MongoBackendKEDRO_GRAPHQL_BROKER--brokerredis://localhostKEDRO_GRAPHQL_CELERY_RESULT_BACKEND--celery-result-backendredis://localhostKEDRO_GRAPHQL_RUNNER--runnerkedro.runner.SequentialRunnerKEDRO_GRAPHQL_ENV--envlocalKEDRO_GRAPHQL_CONF_SOURCE--conf-source$HOME/myproject/confHow to install dependenciesTo install them, run:pip install -r src/requirements.txtHow to testpytest src/testsTo configure the coverage threshold, go to the.coveragercfile.Project dependenciesTo generate or update the dependency requirements for your project:kedro build-reqsThis willpip-compilethe contents ofsrc/requirements.txtinto a new filesrc/requirements.lock. You can see the output of the resolution by openingsrc/requirements.lock.After this, if you'd like to update your project requirements, please updatesrc/requirements.txtand re-runkedro build-reqs.Further information about project dependenciesTO DOsupport custom runners e.g. Argo Workflows, AWS Batch, etc...document plan for supporting custom IOResolverPluginsdocument pipeline tags and implement "search" via tags and/or other fieldsAPI paginations e.g. list pipelines and/or search resultssupport passing credentials via api