Package: langserve
🦜️🏓 LangServe🚩 We will be releasing a hosted version of LangServe for one-click deployments of
LangChain applications.Sign up hereto get on the waitlist.OverviewLangServehelps developers
deployLangChainrunnables and chainsas a REST API.This library is integrated withFastAPIand
usespydanticfor data validation.In addition, it provides a client that can be used to call into runnables deployed on a
server.
A JavaScript client is available
inLangChain.js.FeaturesInput and Output schemas automatically inferred from your LangChain object, and
enforced on every API call, with rich error messagesAPI docs page with JSONSchema and Swagger (insert example link)Efficient/invoke/,/batch/and/stream/endpoints with support for many
concurrent requests on a single server/stream_log/endpoint for streaming all (or some) intermediate steps from your
chain/agentnewas of 0.0.40, supportsastream_eventsto make it easier to stream without needing to parse the output ofstream_log.Playground page at/playground/with streaming output and intermediate stepsBuilt-in (optional) tracing toLangSmith, just
add your API key (seeInstructions)All built with battle-tested open-source Python libraries like FastAPI, Pydantic,
uvloop and asyncio.Use the client SDK to call a LangServe server as if it was a Runnable running
locally (or call the HTTP API directly)LangServe HubLimitationsClient callbacks are not yet supported for events that originate on the serverOpenAPI docs will not be generated when using Pydantic V2. Fast API does not
supportmixing pydantic v1 and v2 namespaces.
See section below for more details.Hosted LangServeWe will be releasing a hosted version of LangServe for one-click deployments of
LangChain
applications.Sign up hereto get on the waitlist.SecurityVulnerability in Versions 0.0.13 - 0.0.15 -- playground endpoint allows accessing
arbitrary files on
server.Resolved in 0.0.16.InstallationFor both client and server:pipinstall"langserve[all]"orpip install "langserve[client]"for client code,
andpip install "langserve[server]"for server code.LangChain CLI 🛠️Use theLangChainCLI to bootstrap aLangServeproject quickly.To use the langchain CLI make sure that you have a recent version oflangchain-cliinstalled. You can install it withpip install -U langchain-cli.langchainappnew../path/to/directoryExamplesGet your LangServe instance started quickly withLangChain Templates.For more examples, see the templatesindexor theexamplesdirectory.DescriptionLinksLLMsMinimal example that reserves OpenAI and Anthropic chat models. Uses async, supports batching and streaming.server,clientRetrieverSimple server that exposes a retriever as a runnable.server,clientConversational RetrieverAConversational Retrieverexposed via LangServeserver,clientAgentwithoutconversation historybased onOpenAI toolsserver,clientAgentwithconversation historybased onOpenAI toolsserver,clientRunnableWithMessageHistoryto implement chat persisted on backend, keyed off asession_idsupplied by client.server,clientRunnableWithMessageHistoryto implement chat persisted on backend, keyed off aconversation_idsupplied by client, anduser_id(see Auth for implementinguser_idproperly).server,clientConfigurable Runnableto create a retriever that supports run time configuration of the index name.server,clientConfigurable Runnablethat shows configurable fields and configurable alternatives.server,clientAPIHandlerShows how to useAPIHandlerinstead ofadd_routes. This provides more flexibility for developers to define endpoints. Works well with all FastAPI patterns, but takes a bit more effort.serverLCEL ExampleExample that uses LCEL to manipulate a dictionary input.server,clientAuthwithadd_routes: Simple authentication that can be applied across all endpoints associated with app. (Not useful on its own for implementing per user logic.)serverAuthwithadd_routes: Simple authentication mechanism based on path dependencies. (No useful on its own for implementing per user logic.)serverAuthwithadd_routes: Implement per user logic and auth for endpoints that use per request config modifier. (Note: At the moment, does not integrate with OpenAPI docs.)server,clientAuthwithAPIHandler: Implement per user logic and auth that shows how to search only within user owned documents.server,clientWidgetsDifferent widgets that can be used with playground (file upload and chat)serverWidgetsFile upload widget used for LangServe playground.server,clientSample ApplicationServerHere's a server that deploys an OpenAI chat model, an Anthropic chat model, and a chain
that uses
the Anthropic model to tell a joke about a topic.#!/usr/bin/env pythonfromfastapiimportFastAPIfromlangchain.promptsimportChatPromptTemplatefromlangchain.chat_modelsimportChatAnthropic,ChatOpenAIfromlangserveimportadd_routesapp=FastAPI(title="LangChain Server",version="1.0",description="A simple api server using Langchain's Runnable interfaces",)add_routes(app,ChatOpenAI(),path="/openai",)add_routes(app,ChatAnthropic(),path="/anthropic",)model=ChatAnthropic()prompt=ChatPromptTemplate.from_template("tell me a joke about{topic}")add_routes(app,prompt|model,path="/joke",)if__name__=="__main__":importuvicornuvicorn.run(app,host="localhost",port=8000)If you intend to call your endpoint from the browser, you will also need to set CORS headers.
You can use FastAPI's built-in middleware for that:fromfastapi.middleware.corsimportCORSMiddleware# Set all CORS enabled originsapp.add_middleware(CORSMiddleware,allow_origins=["*"],allow_credentials=True,allow_methods=["*"],allow_headers=["*"],expose_headers=["*"],)DocsIf you've deployed the server above, you can view the generated OpenAPI docs using:⚠️ If using pydantic v2, docs will not be generated forinvoke,batch,stream,stream_log. SeePydanticsection below for more details.curllocalhost:8000/docsmake sure toaddthe/docssuffix.⚠️ Index page/is not defined bydesign, socurl localhost:8000or visiting
the URL
will return a 404. If you want content at/define an endpoint@app.get("/").ClientPython SDKfromlangchain.schemaimportSystemMessage,HumanMessagefromlangchain.promptsimportChatPromptTemplatefromlangchain.schema.runnableimportRunnableMapfromlangserveimportRemoteRunnableopenai=RemoteRunnable("http://localhost:8000/openai/")anthropic=RemoteRunnable("http://localhost:8000/anthropic/")joke_chain=RemoteRunnable("http://localhost:8000/joke/")joke_chain.invoke({"topic":"parrots"})# or asyncawaitjoke_chain.ainvoke({"topic":"parrots"})prompt=[SystemMessage(content='Act like either a cat or a parrot.'),HumanMessage(content='Hello!')]# Supports astreamasyncformsginanthropic.astream(prompt):print(msg,end="",flush=True)prompt=ChatPromptTemplate.from_messages([("system","Tell me a long story about{topic}")])# Can define custom chainschain=prompt|RunnableMap({"openai":openai,"anthropic":anthropic,})chain.batch([{"topic":"parrots"},{"topic":"cats"}])In TypeScript (requires LangChain.js version 0.0.166 or later):import{RemoteRunnable}from"@langchain/core/runnables/remote";constchain=newRemoteRunnable({url:`http://localhost:8000/joke/`,});constresult=awaitchain.invoke({topic:"cats",});Python usingrequests:importrequestsresponse=requests.post("http://localhost:8000/joke/invoke",json={'input':{'topic':'cats'}})response.json()You can also usecurl:curl--location--requestPOST'http://localhost:8000/joke/invoke'\--header'Content-Type: application/json'\--data-raw'{"input": {"topic": "cats"}}'EndpointsThe following code:...add_routes(app,runnable,path="/my_runnable",)adds of these endpoints to the server:POST /my_runnable/invoke- invoke the runnable on a single inputPOST /my_runnable/batch- invoke the runnable on a batch of inputsPOST /my_runnable/stream- invoke on a single input and stream the outputPOST /my_runnable/stream_log- invoke on a single input and stream the output,
including output of intermediate steps as it's generatedPOST /my_runnable/astream_events- invoke on a single input and stream events as they are generated,
including from intermediate steps.GET /my_runnable/input_schema- json schema for input to the runnableGET /my_runnable/output_schema- json schema for output of the runnableGET /my_runnable/config_schema- json schema for config of the runnableThese endpoints match
theLangChain Expression Language interface--
please reference this documentation for more details.PlaygroundYou can find a playground page for your runnable at/my_runnable/playground/. This
exposes a simple UI
toconfigureand invoke your runnable with streaming output and intermediate steps.WidgetsThe playground supportswidgetsand can be used to test your
runnable with different inputs. See thewidgetssection below for more
details.SharingIn addition, for configurable runnables, the playground will allow you to configure the
runnable and share a link with the configuration:Legacy ChainsLangServe works with both Runnables (constructed
viaLangChain Expression Language)
and legacy chains (inheriting fromChain).
However, some of the input schemas for legacy chains may be incomplete/incorrect,
leading to errors.
This can be fixed by updating theinput_schemaproperty of those chains in LangChain.
If you encounter any errors, please open an issue on THIS repo, and we will work to
address it.DeploymentDeploy to AWSYou can deploy to AWS using theAWS Copilot CLIcopilotinit--app[application-name]--name[service-name]--type'Load Balanced Web Service'--dockerfile'./Dockerfile'--deployClickhereto learn more.Deploy to AzureYou can deploy to Azure using Azure Container Apps (Serverless):az containerapp up --name [container-app-name] --source . --resource-group [resource-group-name] --environment  [environment-name] --ingress external --target-port 8001 --env-vars=OPENAI_API_KEY=your_keyYou can find more
infohereDeploy to GCPYou can deploy to GCP Cloud Run using the following command:gcloud run deploy [your-service-name] --source . --port 8001 --allow-unauthenticated --region us-central1 --set-env-vars=OPENAI_API_KEY=your_keyDeploy using Infrastructure as CodePulumiYou can deploy your LangServe server withPulumiusing your preferred general purpose language. Below are some quickstart
examples for deploying LangServe to different cloud providers.These examples are a good starting point for your own infrastructure as code (IaC) projects. You can easily modify them to suit your needs.CloudLanguageRepositoryQuickstartAWSdotnethttps://github.com/pulumi/examples/aws-cs-langserveAWSgolanghttps://github.com/pulumi/examples/aws-go-langserveAWSpythonhttps://github.com/pulumi/examples/aws-py-langserveAWStypescripthttps://github.com/pulumi/examples/aws-ts-langserveAWSjavascripthttps://github.com/pulumi/examples/aws-js-langserveCommunity ContributedDeploy to RailwayExample Railway RepoPydanticLangServe provides support for Pydantic 2 with some limitations.OpenAPI docs will not be generated for invoke/batch/stream/stream_log when using
Pydantic V2. Fast API does not support [mixing pydantic v1 and v2 namespaces].LangChain uses the v1 namespace in Pydantic v2. Please read
thefollowing guidelines to ensure compatibility with LangChainExcept for these limitations, we expect the API endpoints, the playground and any other
features to work as expected.AdvancedHandling AuthenticationIf you need to add authentication to your server, please read Fast API's documentation
aboutdependenciesandsecurity.The below examples show how to wire up authentication logic LangServe endpoints using FastAPI primitives.You are responsible for providing the actual authentication logic, the users table etc.If you're not sure what you're doing, you could try using an existing solutionAuth0.Using add_routesIf you're usingadd_routes, see
exampleshere.DescriptionLinksAuthwithadd_routes: Simple authentication that can be applied across all endpoints associated with app. (Not useful on its own for implementing per user logic.)serverAuthwithadd_routes: Simple authentication mechanism based on path dependencies. (No useful on its own for implementing per user logic.)serverAuthwithadd_routes: Implement per user logic and auth for endpoints that use per request config modifier. (Note: At the moment, does not integrate with OpenAPI docs.)server,clientAlternatively, you can use FastAPI'smiddleware.Using global dependencies and path dependencies has the advantage that auth will be properly supported in the OpenAPI docs page, but
these are not sufficient for implement per user logic (e.g., making an application that can search only within user owned documents).If you need to implement per user logic, you can use theper_req_config_modifierorAPIHandler(below) to implement this logic.Per UserIf you need authorization or logic that is user dependent,
specifyper_req_config_modifierwhen usingadd_routes. Use a callable receives the
rawRequestobject and can extract relevant information from it for authentication and
authorization purposes.Using APIHandlerIf you feel comfortable with FastAPI and python, you can use LangServe'sAPIHandler.DescriptionLinksAuthwithAPIHandler: Implement per user logic and auth that shows how to search only within user owned documents.server,clientAPIHandlerShows how to useAPIHandlerinstead ofadd_routes. This provides more flexibility for developers to define endpoints. Works well with all FastAPI patterns, but takes a bit more effort.server,clientIt's a bit more work, but gives you complete control over the endpoint definitions, so
you can do whatever custom logic you need for auth.FilesLLM applications often deal with files. There are different architectures
that can be made to implement file processing; at a high level:The file may be uploaded to the server via a dedicated endpoint and processed using a
separate endpointThe file may be uploaded by either value (bytes of file) or reference (e.g., s3 url
to file content)The processing endpoint may be blocking or non-blockingIf significant processing is required, the processing may be offloaded to a dedicated
process poolYou should determine what is the appropriate architecture for your application.Currently, to upload files by value to a runnable, use base64 encoding for the
file (multipart/form-datais not supported yet).Here's
anexamplethat shows
how to use base64 encoding to send a file to a remote runnable.Remember, you can always upload files by reference (e.g., s3 url) or upload them as
multipart/form-data to a dedicated endpoint.Custom Input and Output TypesInput and Output types are defined on all runnables.You can access them via theinput_schemaandoutput_schemaproperties.LangServeuses these types for validation and documentation.If you want to override the default inferred types, you can use thewith_typesmethod.Here's a toy example to illustrate the idea:fromtypingimportAnyfromfastapiimportFastAPIfromlangchain.schema.runnableimportRunnableLambdaapp=FastAPI()deffunc(x:Any)->int:"""Mistyped function that should accept an int but accepts anything."""returnx+1runnable=RunnableLambda(func).with_types(input_type=int,)add_routes(app,runnable)Custom User TypesInherit fromCustomUserTypeif you want the data to de-serialize into a
pydantic model rather than the equivalent dict representation.At the moment, this type only worksserverside and is used
to specify desireddecodingbehavior. If inheriting from this type
the server will keep the decoded type as a pydantic model instead
of converting it into a dict.fromfastapiimportFastAPIfromlangchain.schema.runnableimportRunnableLambdafromlangserveimportadd_routesfromlangserve.schemaimportCustomUserTypeapp=FastAPI()classFoo(CustomUserType):bar:intdeffunc(foo:Foo)->int:"""Sample function that expects a Foo type which is a pydantic model"""assertisinstance(foo,Foo)returnfoo.bar# Note that the input and output type are automatically inferred!# You do not need to specify them.# runnable = RunnableLambda(func).with_types( # <-- Not needed in this case#     input_type=Foo,#     output_type=int,#add_routes(app,RunnableLambda(func),path="/foo")Playground WidgetsThe playground allows you to define custom widgets for your runnable from the backend.Here are a few examples:DescriptionLinksWidgetsDifferent widgets that can be used with playground (file upload and chat)server,clientWidgetsFile upload widget used for LangServe playground.server,clientSchemaA widget is specified at the field level and shipped as part of the JSON schema of the
input typeA widget must contain a key calledtypewith the value being one of a well known
list of widgetsOther widget keys will be associated with values that describe paths in a JSON objecttypeJsonPath=number|string|(number|string)[];typeNameSpacedPath={title:string;path:JsonPath};// Using title to mimick json schema, but can use namespacetypeOneOfPath={oneOf:JsonPath[]};typeWidget={type:string// Some well known type (e.g., base64file, chat etc.)[key:string]:JsonPath|NameSpacedPath|OneOfPath;};Available WidgetsThere are only two widgets that the user can specify manually right now:File Upload WidgetChat History WidgetSee below more information about these widgets.All other widgets on the playground UI are created and managed automatically by the UI
based on the config schema of the Runnable. When you create Configurable Runnables,
the playground should create appropriate widgets for you to control the behavior.File Upload WidgetAllows creation of a file upload input in the UI playground for files
that are uploaded as base64 encoded strings. Here's the
fullexample.Snippet:try:frompydantic.v1importFieldexceptImportError:frompydanticimportFieldfromlangserveimportCustomUserType# ATTENTION: Inherit from CustomUserType instead of BaseModel otherwise#            the server will decode it into a dict instead of a pydantic model.classFileProcessingRequest(CustomUserType):"""Request including a base64 encoded file."""# The extra field is used to specify a widget for the playground UI.file:str=Field(...,extra={"widget":{"type":"base64file"}})num_chars:int=100Example widget:Chat WidgetLook
at thewidget example.To define a chat widget, make sure that you pass "type": "chat"."input" is JSONPath to the field in theRequestthat has the new input message."output" is JSONPath to the field in theResponsethat has new output message(s).Don't specify these fields if the entire input or output should be used as they are (
e.g., if the output is a list of chat messages.)Here's a snippet:classChatHistory(CustomUserType):chat_history:List[Tuple[str,str]]=Field(...,examples=[[("human input","ai response")]],extra={"widget":{"type":"chat","input":"question","output":"answer"}},)question:strdef_format_to_messages(input:ChatHistory)->List[BaseMessage]:"""Format the input to a list of messages."""history=input.chat_historyuser_input=input.questionmessages=[]forhuman,aiinhistory:messages.append(HumanMessage(content=human))messages.append(AIMessage(content=ai))messages.append(HumanMessage(content=user_input))returnmessagesmodel=ChatOpenAI()chat_model=RunnableParallel({"answer":(RunnableLambda(_format_to_messages)|model)})add_routes(app,chat_model.with_types(input_type=ChatHistory),config_keys=["configurable"],path="/chat",)Example widget:You can also specify a list of messages as your a parameter directly, as shown in this snippet:prompt=ChatPromptTemplate.from_messages([("system","You are a helpful assisstant named Cob."),MessagesPlaceholder(variable_name="messages"),])chain=prompt|ChatAnthropic(model="claude-2")classMessageListInput(BaseModel):"""Input for the chat endpoint."""messages:List[Union[HumanMessage,AIMessage]]=Field(...,description="The chat messages representing the current conversation.",extra={"widget":{"type":"chat","input":"messages"}},)add_routes(app,chain.with_types(input_type=MessageListInput),path="/chat",)Seethis sample filefor an example.Enabling / Disabling Endpoints (LangServe >=0.0.33)You can enable / disable which endpoints are exposed when adding routes for a given chain.Useenabled_endpointsif you want to make sure to never get a new endpoint when upgrading langserve to a newer
verison.Enable: The code below will only enableinvoke,batchand the
correspondingconfig_hashendpoint variants.add_routes(app,chain,enabled_endpoints=["invoke","batch","config_hashes"],path="/mychain")Disable: The code below will disable the playground for the chainadd_routes(app,chain,disabled_endpoints=["playground"],path="/mychain")