Package: langchain-serve
⚡ LangChain Apps on Production with Jina & FastAPI 🚀Jinais an open-source framework for building scalable multi modal AI apps on Production.LangChainis another open-source framework for building applications powered by LLMs.langchain-servehelps you deploy your LangChain apps on Jina AI Cloud in a matter of seconds. You can benefit from the scalability and serverless architecture of the cloud without sacrificing the ease and convenience of local development. And if you prefer, you can also deploy your LangChain apps on your own infrastructure to ensure data privacy. With langchain-serve, you can craft REST/Websocket APIs, spin up LLM-powered conversational Slack bots, or wrap your LangChain apps into FastAPI packages on cloud or on-premises.Give us a :star: and tell us what more you'd like to see!☁️ LLM Apps as-a-servicelangchain-serve currently wraps following apps as a service to be deployed on Jina AI Cloud with one command.🔮 AutoGPT-as-a-serviceAutoGPTis an "AI agent" that given a goal in natural language, will attempt to achieve it by breaking it into sub-tasks and using the internet and other tools in an automatic loop.Show usageDeployautogpton Jina AI Cloud with one commandlc-servedeployautogptShow command output╭──────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ App ID       │                                           autogpt-6cbd489454                                           │
├──────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Phase        │                                                Serving                                                 │
├──────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Endpoint     │                                 wss://autogpt-6cbd489454.wolf.jina.ai                                  │
├──────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ App logs     │                                        dashboards.wolf.jina.ai                                         │
├──────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Swagger UI   │                              https://autogpt-6cbd489454.wolf.jina.ai/docs                              │
├──────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ OpenAPI JSON │                          https://autogpt-6cbd489454.wolf.jina.ai/openapi.json                          │
╰──────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────╯Integrate autogpt with external services using APIs. Get a flavor of the integration on your CLI withlc-serveplaygroundautogptShow playground🧠 Babyagi-as-a-serviceBabyagiis a task-driven autonomous agent that uses LLMs to create, prioritize, and execute tasks. It is a general-purpose AI agent that can be used to automate a wide variety of tasks.Show usageDeploybabyagion Jina AI Cloud with one commandlc-servedeploybabyagiIntegrate babyagi with external services using our Websocket API. Get a flavor of the integration on your CLI withlc-serveplaygroundbabyagiShow playground:panda_face: pandas-ai-as-a-servicepandas-aiintegrates LLM capabilities into Pandas, to make dataframes conversational in Python code. Thanks to langchain-serve, we can now expose pandas-ai APIs on Jina AI Cloud in just a matter of seconds.Show usageDeploypandas-aion Jina AI Cloudlc-servedeploypandas-aiShow command output╭──────────────┬─────────────────────────────────────────────────────────────────────────────────╮
│ App ID       │                               pandasai-06879349ca                               │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ Phase        │                                     Serving                                     │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ Endpoint     │                     wss://pandasai-06879349ca.wolf.jina.ai                      │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ App logs     │                             dashboards.wolf.jina.ai                             │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ Swagger UI   │                  https://pandasai-06879349ca.wolf.jina.ai/docs                  │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ OpenAPI JSON │              https://pandasai-06879349ca.wolf.jina.ai/openapi.json              │
╰──────────────┴─────────────────────────────────────────────────────────────────────────────────╯Upload your DataFrame to Jina AI Cloud (Optional - you can also use a publicly available CSV)Define your DataFrame in a Python file# dataframe.pyimportpandasaspddf=pd.DataFrame(some_data)Upload your DataFrame to Jina AI Cloud using<module>:<variable>syntaxlc-serveutilupload-dfdataframe:dfConversationalize your DataFrame using pandas-ai APIs. Get a flavor of the integration with a local playground on your CLI withlc-serveplaygroundpandas-ai<host>Show playground💬 Question Answer Bot on PDFspdfqnais a simple question answering bot that uses LLMs to answer questions on PDF documents, showcasing the how easy it is to integrate langchain apps on Jina AI Cloud.Show usageDeploypdf_qnaon Jina AI Cloud with one commandlc-servedeploypdf-qnaGet a flavor of the integration with Streamlit playground on your CLI withlc-serveplaygroundpdf-qnaShow playgroundExpand the Q&A bot to multiple languages, different document types & integrate with external services using simple REST APIs.https://github.com/jina-ai/langchain-serve/blob/8f7a9272e99490a5357655becfc5da3569655f38/lcserve/apps/pdf_qna/app.py#L8-L12💪 Features🎉 LLM Apps on production👉Define your API using@servingdecorator👉Build, deploy & distribute Slack bots using@slackbotdecorator👉Bring your own FastAPI app🔥 Secure, Scalable, Serverless, Streaming REST/Websocket APIs onJina AI Cloud.🌎 Globally available REST/Websocket APIs with automatic TLS certs.🌊 Stream LLM interactions in real-time with Websockets.👥 Enable human in the loop for your agents.💬 Build, deploy & distribute Slack bots built with langchain.🔑 Protect your APIs withAPI authorizationusing Bearer tokens.📄 Swagger UI, and OpenAPI spec included with your APIs.⚡️ Serverless, autoscaling apps that scales automatically with your traffic.🗝️ Secure handling of secrets and environment variables.📁 Persistent storage (EFS) mounted on your app for your data.⏱️ Trigger one-time jobs to run asynchronously, allowing for non-blocking execution.📊 Builtin logging, monitoring, and traces for your APIs.🤖 No need to change your code to manage APIs, or manage dockerfiles, or worry about infrastructure!🏠 Self-host LLM Apps with Docker Compose or Kubernetes🚀 Export your apps as Kubernetes or Docker Compose YAMLs with single command.👉lc-serve export app --kind <kubernetes/docker-compose> --path .📦 Deploy your app on your own internal infrastructure with your own security policies.📞Talk to us if you need all the features of Jina AI Cloud on your own infrastructure.🧰 UsageLet's first installlangchain-serveusing pip.pipinstalllangchain-serve🔄 REST APIs using@servingdecorator👉 Let's go through astep-by-step guideto build, deploy and use a REST API using@servingdecorator.🤖💬 Build, Deploy & Distribute Slack bots built with LangChainlangchain-serve exposes a@slackbotdecorator to quickly build, deploy & distribute LLM-powered Slack bots without worrying about the infrastructure. It provides a simple interface to any langchain app on and makes them super accessible to users a platform they're already comfortable with.✨ Ready to dive in?There's astep-by-step guide in the repositoryto help you build your own bot for helping with reasoning.Here's anotherstep-by-step guideto help you chat over own internal HR-realted documents (like onboarding, policies etc.) with your employees right inside your Slack workspace.🔐 Authorize your APIsTo add an extra layer of security, we can integrate any custom API authorization by adding aauthargument to the@servingdecorator.Show code & gotchasfromlcserveimportservingdefauthorizer(token:str)->Any:ifnottoken=='mysecrettoken':# Change this to add your own authorization logicraiseException('Unauthorized')# Raise an exception if the request is not authorizedreturn'userid'# Return any user id or object@serving(auth=authorizer)defask(question:str,**kwargs)->str:auth_response=kwargs['auth_response']# This will be 'userid'return...@serving(websocket=True,auth=authorizer)asyncdeftalk(question:str,**kwargs)->str:auth_response=kwargs['auth_response']# This will be 'userid'return...🤔 Gotchas about theauthfunctionShould accept only one argumenttoken.Should raise an Exception if the request is not authorized.Can return any object, which will be passed to theauth_responseobject underkwargsto the functions.Expects Bearer token in theAuthorizationheader of the request.Sample HTTP request withcurl:curl-X'POST''http://localhost:8080/ask'-H'Authorization: Bearer mysecrettoken'-d'{ "question": "...", "envs": {} }'Sample WebSocket request withwscat:wscat-H"Authorization: Bearer mysecrettoken"-cws://localhost:8080/talk🙋‍♂️ Enable streaming & human-in-the-loop (HITL) with WebSocketsHITL for LangChain agents on production can be challenging since the agents are typically running on servers where humans don't have direct access.langchain-servebridges this gap by enabling websocket APIs that allow for real-time interaction and feedback between the agent and a human operator.Check out thisexampleto see how you can enable HITL for your agents.📁 Persistent storage on Jina AI CloudEvery app deployed on Jina AI Cloud gets a persistent storage (EFS) mounted locally which can be accessed viaworkspacekwarg in the@servingfunction.Show codefromlcserveimportserving@servingdefstore(text:str,**kwargs):workspace:str=kwargs.get('workspace')path=f'{workspace}/store.txt'print(f'Writing to{path}')withopen(path,'a')asf:f.writelines(text+'\n')return'OK'@serving(websocket=True)asyncdefstream(**kwargs):workspace:str=kwargs.get('workspace')websocket:WebSocket=kwargs.get('websocket')path=f'{workspace}/store.txt'print(f'Streaming{path}')asyncwithaiofiles.open(path,'r')asf:asyncforlineinf:awaitwebsocket.send_text(line)return'OK'Here, we are using theworkspaceto store the incoming text in a file via the REST endpoint and streaming the contents of the file via the WebSocket endpoint.🚀 Bring your own FastAPI appIf you already have a FastAPI app with pre-defined endpoints, you can uselc-serveto deploy it on Jina AI Cloud.lc-servedeployjcloud--appfilename:appShow detailsLet's take an example of a simple FastAPI app with directory structure.
└──endpoints.py# endpoints.pyfromtypingimportUnionfromfastapiimportFastAPIapp=FastAPI()@app.get("/status")defread_root():return{"Hello":"World"}@app.get("/items/{item_id}")defread_item(item_id:int,q:Union[str,None]=None):return{"item_id":item_id,"q":q}lc-servedeployjcloud--appendpoints:app🗝️ Using Secrets during DeploymentYou can use secrets during app deployment by passing a secrets file to deployment with the--secretsflag. The secrets file should be a.envfile containing the secrets.lcservedeployjcloudapp--secrets.envShow detailsLet's take an example of a simple app that usesOPENAI_API_KEYstored as secrets.This app directory contains the following files:.
├── main.py             # The app
├── jcloud.yml          # JCloud deployment config file
├── README.md           # This README file
├── requirements.txt    # The requirements file for the app
└── secrets.env         # The secrets file containing the redis credentialsNotesecret.envin this directory is a dummy file. You should replace it with your own secrets after creating a Redis instance. (For example withUpstash), such as:OPENAI_API_KEY=sk-xxxmain.pywill look like:# main.pyfromlcserveimportservingfromlangchain.promptsimportPromptTemplatefromlangchain.chainsimportLLMChainfromlangchain.chat_modelsimportChatOpenAIprompt=PromptTemplate(input_variables=["subject"],template="Write me a short poem about{subject}?",)@serving(openai_tracing=True)defpoem(subject:str,**kwargs):tracing_handler=kwargs.get("tracing_handler")chat=ChatOpenAI(temperature=0.5,callbacks=[tracing_handler])chain=LLMChain(llm=chat,prompt=prompt,callbacks=[tracing_handler])returnchain.run(subject)In the above example, the app will useOPENAI_API_KEYprovided by the secrets to interact with OpenAI.Then you can deploy using the following command and interact with the deployed endpoint.lc-servedeployjcloudmain--secretssecrets.env⏱️ Trigger one-time jobs to run asynchronouslyHere's astep-by-step guideto trigger one-time jobs to run asynchronously using@jobdecorator.💻lc-serveCLIlc-serveis a simple CLI that helps you to deploy your agents on Jina AI Cloud (JCloud)DescriptionCommandDeploy your app locallylc-serve deploy local appExport your app as Kubernetes YAMLlc-serve export app --kind kubernetes --path .Export your app as Docker Compose YAMLlc-serve export app --kind docker-compose --path .Deploy your app on JCloudlc-serve deploy jcloud appDeploy FastAPI app on JCloudlc-serve deploy jcloud --app <app-name>:<app-object>Update existing app on JCloudlc-serve deploy jcloud app --app-id <app-id>Get app status on JCloudlc-serve status <app-id>List all apps on JCloudlc-serve listRemove app on JCloudlc-serve remove <app-id>Pause app on JCloudlc-serve pause <app-id>Resume app on JCloudlc-serve resume <app-id>💡 JCloud Deployment⚙️ ConfigurationsFor JCloud deployment, you can configure your application infrastructure by providing a YAML configuration file using the--configoption. The supported configurations are:Instance type (instance), as defined byJina AI Cloud.Minimum number of replicas for your application (autoscale_min). Setting it 0 enablesserverless.Disk size (disk_size), in GB. The default value is 1 GB.For example:instance: C4
autoscale_min: 0
disk_size: 1.5GYou can alternatively include ajcloud.yamlfile in your application directory with the desired configurations. However, please note that if the--configoption is explicitly used in the command line interface, the local jcloud.yaml file will be disregarded. The command line provided configuration file will take precedence.If you don't provide a configuration file or a specific configuration isn't specified, the following default settings will be applied:instance: C3
autoscale_min: 1
disk_size: 1G💰 PricingApplications hosted on JCloud are priced in two categories:Base creditsBase credits are charged to ensure high availability for your application by maintaining at least one instance running continuously, ready to handle incoming requests. If you wish to stop the serving application, you can either remove the app completely or put it on pause, the latter allows you to resume the app serving based on persisted configurations (refer tolc-serveCLI sectionfor more information). Both options will halt the consumption of credits.Actual credits charged for base credits are calculated based on theinstance type as defined by Jina AI Cloud.By default, instance typeC3is used with a minimum of 1 instance andAmazon EFSdisk of size 1G, which means that if your application is served on JCloud, you will be charged ~10 credits per hour.You can change the instance type and the minimum number of instances by providing a YAML configuration file using the--configoption. For example, if you want to use instance typeC4with a minimum of 0 replicas, and 2G EFS disk, you can provide the following configuration file:instance:C4autoscale_min:0disk_size:2GServing creditsServing credits are charged when your application is actively serving incoming requests.Actual credits charged for serving credits are calculated based on the credits for the instance type multiplied by the duration for which your application serves requests.You are charged for each second your application is serving requests.Total credits charged = Base credits + Serving credits. (Jina AI Clouddefines each credit as €0.005)ExamplesExample 1Consider an HTTP application that has served requests for10minutes in the last hour and uses a custom config:instance: C4
autoscale_min: 0
disk_size: 2GTotal credits per hour charged would be3.33. The calculation is as follows:C4 instance has an hourly credit rate of 20.
EFS has hourly credit rate of 0.104 per GB.
Base credits = 0 + 2 * 0.104 = 0.208 (since `autoscale_min` is 0)
Serving credits = 20 * 10/60 = 3.33
Total credits per hour = 0.208 + 3.33 = 3.538Example 2Consider a WebSocket application that had active connections for 20 minutes in the last hour and uses the default configuration.instance: C3
autoscale_min: 1
disk_size: 1GTotal credits per hour charged would be13.33. The calculation is as follows:C3 instance has an hourly credit rate of 10.
EFS has hourly credit rate of 0.104 per GB.
Base credits = 10 + 1 * 0.104 = 10.104 (since `autoscale_min` is 1)
Serving credits = 10 * 20/60 = 3.33
Total credits per hour = 10.104 + 3.33 = 13.434❓ Frequently Asked Questionslc-servecommand not foundMy client that connects to the JCloud hosted App gets timed-out, what should I do?How to pass environment variables to the app?JCloud deployment failed at pushing image to Jina Hubble, what should I do?Debug babyagi playground request/response for external integrationlc-servecommand not foundExpandlc-servecommand is registered duringlangchain-serveinstallation. If you getcommand not found: lc-serveerror, please replacelc-servecommand withpython -m lcserve& retry.My client that connects to the JCloud hosted App gets timed-out, what should I do?ExpandIf you make long HTTP/ WebSocket requests, the default timeout value (2 minutes) might not be suitable for your use case. You can provide a custom timeout value during JCloud deployment by using the--timeoutargument.Additionally, for HTTP, you may also experience timeouts due to limitations in the OSS we used inlangchain-serve. While we are working to permanently address this issue, we recommend using HTTP/1.1 in your client as a temporary workaround.For WebSocket, please note that the connection will be closed if idle for more than 5 minutes.How to pass environment variables to the app?ExpandWe provide 2 options to pass environment variables:Use--envduring app deployment to load env variables from a.envfile. For example,lc-serve deploy jcloud app --env some.envwill load all env variables fromsome.envfile and pass them to the app. These env variables will be available in the app asos.environ['ENV_VAR_NAME'].You can also pass env variables while sending requests to the app both in HTTP and WebSocket.envsfield in the request body is used to pass env variables. For example{"question":"What is the meaning of life?","envs":{"ENV_VAR_NAME":"ENV_VAR_VALUE"}}JCloud deployment failed at pushing image to Jina Hubble, what should I do?ExpandPlease use--verboseand retry to get more information. If you are operating on computer witharm64arch, please retry with--platform linux/amd64so the image can be built correctly.Debug babyagi playground request/response for external integrationExpand1. Start textual console in a terminal (exclude following groups to reduce the noise in logging)```bash
textual console -x EVENT -x SYSTEM -x DEBUG
```Start the playground with--verboseflag. Start interacting and see the logs in the console.lc-serveplaygroundbabyagi--verbose📣 Reach out to usWant to deploy your LLM apps on your own infrastructure with all capabilities of Jina AI Cloud?ServerlessAutoscalingTLS certsPersistent storageEnd to end LLM observabilityand more on auto-pilot!Join us on Discordand we'd be happy to hear more about your use case.