Package: fastqlapi
FastQL Inference ServerSpin up a blazing fast Rust GraphQL API to serve your ML model in one line of Python code.Included are examples which can deploy and serve stable diffusion / runwayml / midjourney or any of theHugging facediffusers models in seconds.We've observed about a2x speed upacross the example schema vs a FastAPI/Ariadne Python GraphQL server with identical schema. This is because although the model code is in Python, the API is actually a Rust process runningTechEmpowerno.5 fastest web frameworkActiX.Please noteDoes not include an auth mechanism, you can implement auth using an API Gateway or auth proxy, or using Apollo Federation.Can only create flat / non-nested schema and currently only a single query (support for subscription and mutation coming soon).Make sure you set RUST_ENV to production if you are using it on a remote machine.This is currently a prototype and should not be used in production. It is perfect for prototyping and sharing your models.Why would I need FastQL?You want to be able to spin up and query your model quickly, in order to share, test or demo it.FastQL encourages seperation of your API and model, allowing you to focus on your model code.All the benefits of GraphQL... Fetch data with single API call, no overfetching, autogenerated API documentation, validation and type checking out-of-the-box, API evolution without versioning, subscriptions (websockets) as a first class citizen.FastQL is really fast, faster than a Python GraphQL implementation, the example schema has a latency of < 20ms when tested locally.FastlQL isApollo federationcompliant, which means that you can aggregate many models into a single curated API using schema stitching. This means you can maintain both fixed and dynamic schema's for params to your models, so it is easy to test new features and benefit from federation versioning. You can also use federation to curate, version and productionise a supergraph API.FastQL gives you GraphQL playground for testing and demonstrating your prototype models.Because ML models are typically sequential in terms of request response pattern, requests to the model arealwaysblocking,response time is the important bottleneck FastQL is focused on reducing.A Rust server is free of pythons GIL lock and doesn't suffer from Go's"buffering issues"Installation (Linux any, Mac, Mac M1, Windows)pipinstallfastqlapiUsageYou can callanycallback with each GraphQL query recieved by the server, the callback must return a dict with values for each field you wish to return. The returned type must match the type defined for the field in the fields dict given to the start method (see below).For example:fromfastqlapiimportfastql_serverdefinfer(**kwargs):print(kwargs['input'])return{'output':"test response",}fastql_server.start(callback=infer,query_name="Model",args={"input":{"type":"String","description":"this is my input field"}},fields={"output":{"type":"String"}})Will spin up the below GraphQL Playground ready for requests on localhost:8000/graphiql or you can make a GraphQL request to localhost:8000 like:{Model(input="tosend"){output}}Spin up an example hugging face diffusers GraphQL API using Docker on AWS EC2NB. these examples are just toy examples not suitable for productionLaunch an EC2 GPU powered instance (for example p3.2xlarge)Search for NVIDIA GPU-Optimized PyTorch AMI in the Amazon Machine Image search bar.Select this AMI and configure storage to 64GIB.You will need to set a PEM key-pair, download and chmod the key on your local machine using chmod 400 {{key pair name}}.Launch your instance.Get your current IPV4 IP (for example fromwhatsmyip)In the security tab for your instance select the auto-created or chosen security group and add an inbound rule setting custom TCP and port 8000 (graphql api) with source to {{Your IPV4 IP}} (or 0.0.0.0/0 to make it public) and another identical rule for port 8080 (the image server)Sign up athttps://huggingface.co/to get your free ACCESS_TOKEN.Create an access key in the IAM panel.Setup aws configure with the command aws configure and enter access key details.Run the following commands to set your public IP:EC2_INSTANCE_ID="`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id || die \"wget instance-id has failed:$?\"`"exportPUBLIC_IP=$(awsec2describe-instances--instance-ids$EC2_INSTANCE_ID--query'Reservations[*].Instances[*].PublicIpAddress'--outputtext)echo $PUBLIC_IPand make a note of this public ip you will use to connect to the APIConnect to your instance and use the example docker image on danfreshbc/fastqlapi-diffusers or run your published docker image similar to:dockerrun-p8000:8000-p8080:8080\-ePUBLIC_IP=$PUBLIC_IP\-eMODEL_ID="stabilityai/stable-diffusion-2"\--gpusalldanfreshbc/fastqlapi-diffusers:latestOnce Docker has finished installing...Connect to {{PUBLIC IP}}:8000/graphql to visit your new GraphQL API and make sure that the URL next to the history button contains {{PUBLIC IP}}:8000Try the following example query in the GraphQL Playground console:{Model1(prompt:"a handsome man holding a baby goat"){images}}In the response on the right you will get back a URL you can visit to see the generated image.You can change MODEL_ID to one of the following to try a different diffusers model for example:"Mitsua/mitsua-diffusion-cc0"This is recommended as it trained on totally public images, there is some controversy over the legality of the models below (be warned!)"CompVis/stable-diffusion-v1-4""stabilityai/stable-diffusion-2""runwayml/stable-diffusion-v1-5"The example API surfaces prompt, number_of_images, guidance_scale, number_inference_steps and seed and returns an array of images or a seed.The example uses a ruby webserver to serve up the content of an images directory which it will deposit generated images into on port 8080.A diffusers example implementing fine tuning using dreamboothNB. Mutation support coming soon.This example allows you to download a folder of images from google drive or individual gdrive image ids. It is suggested to upload 10-12 images to a Gdrive folder and make sure you set sharing to 'Anyone with the link' and copy the link. The toy API will then finetune the model on these images so you can run inference against them. This example will overwrite the model each time you upload new files.Setup your instance as above but with a machine withat least16GB ram, the smallest on AWS would be m5.2xlarge. (seeHuggingface dreambooth).Use the following docker command:dockerrun-p8000:8000-p8080:8080\-ePUBLIC_IP=$PUBLIC_IP\-eMODEL_ID="stabilityai/stable-diffusion-2"\--gpusalldanfreshbc/fastqlapi-diffusers:latestbashstart_finetune.shTo start finetuning use a query like below:{Model(fine_tune_photo_description:"Photo of happymachine"gdrive_folder_of_images_link:"https://drive.google.com/drive/folders/17g_m3eaBA6SJQP-xppGkAHXYoKCLObZg"){images}}This query will take around five minutes while the model is tuned and prepared for inference.To infer:{Model(prompt:"happymachine standing in front of an F1 McLaren"){images}}Further infoFastQL currently implements Int, Float, String, ID and array and required versions of those types but not currently
required subtypes (an element of a list).Under the hood FastQL uses the ActiX Rust web server which is currently no.5 fastest web framework according toTech Empower. By comparison, Python'sFastAPIis at no.93. We've observed about a 2x speed up across the example schema here vs a FastAPI/Ariadne Python GraphQL server with the same schema.Environment variablesGRAPHQL_HOSTDefault localhostGRAPHQL_PORTDefault 8000ENABLE_GRAPHIQLWill enable GraphiQL | default to true unless RUST_ENV is set to production (in which case setting ENABLE_GRAPHIQL=true will still overide)CORS_PERMISSIVEShould be used in Development/Test only, allows all headers and origins, credentials supported, maximum age 1 hour does not send wildcard | default trueALLOW_AUTHORIZATION_HEADERdefault falseALLOW_ORIGIN_HEADERdefault falseALLOW_CONTENT_TYPE_HEADERdefault falseMAX_AGE_HEADERdefault 3600RUST_ENVdefault 'development'RUST_LOGRust log level | default 'debug'RUST_BACKTRACEAdd Rust backtrace to log | default 1RUST_QUIETNo Rust logs | default falseTRACINGTurn on Apollo tracing | default falseLicenseThe code in this repo is released under the MIT license.Thank youThe folks atHugging FaceThe team atActixSunli @Async-graphql.rsThe team atMaturinThis is a passion project, we'd love your help, please message me on Twitter @djfreshuk!