Package: langchain-nvidia-ai-endpoints
langchain-nvidia-ai-endpointsThelangchain-nvidia-ai-endpointspackage contains LangChain integrations for chat models and embeddings powered by theNVIDIA AI Foundation Modelplayground environment.NVIDIA AI Foundation Endpointsgive users easy access to hosted endpoints for generative AI models like Llama-2, SteerLM, Mistral, etc. Using the API, you can query live endpoints available on theNVIDIA GPU Cloud (NGC)to get quick results from a DGX-hosted cloud compute environment. All models are source-accessible and can be deployed on your own compute cluster.Below is an example on how to use some common functionality surrounding text-generative and embedding modelsInstallation%pipinstall-U--quietlangchain-nvidia-ai-endpointsSetupTo get started:Create a free account with theNVIDIA GPU Cloudservice, which hosts AI solution catalogs, containers, models, etc.Navigate toCatalog > AI Foundation Models > (Model with API endpoint).Select theAPIoption and clickGenerate Key.Save the generated key asNVIDIA_API_KEY. From there, you should have access to the endpoints.importgetpassimportosifnotos.environ.get("NVIDIA_API_KEY","").startswith("nvapi-"):nvidia_api_key=getpass.getpass("Enter your NVIDIA AIPLAY API key: ")assertnvidia_api_key.startswith("nvapi-"),f"{nvidia_api_key[:5]}... is not a valid key"os.environ["NVIDIA_API_KEY"]=nvidia_api_key## Core LC Chat Interfacefromlangchain_nvidia_ai_endpointsimportChatNVIDIAllm=ChatNVIDIA(model="mixtral_8x7b")result=llm.invoke("Write a ballad about LangChain.")print(result.content)Stream, Batch, and AsyncThese models natively support streaming, and as is the case with all LangChain LLMs they expose a batch method to handle concurrent requests, as well as async methods for invoke, stream, and batch. Below are a few examples.print(llm.batch(["What's 2*3?","What's 2*6?"]))# Or via the async API# await llm.abatch(["What's 2*3?", "What's 2*6?"])forchunkinllm.stream("How far can a seagull fly in one day?"):# Show the token separationsprint(chunk.content,end="|")asyncforchunkinllm.astream("How long does it take for monarch butterflies to migrate?"):print(chunk.content,end="|")Supported modelsQueryingavailable_modelswill still give you all of the other models offered by your API credentials.Theplayground_prefix is optional.list(llm.available_models)# ['playground_llama2_13b',# 'playground_llama2_code_13b',# 'playground_clip',# 'playground_fuyu_8b',# 'playground_mistral_7b',# 'playground_nvolveqa_40k',# 'playground_yi_34b',# 'playground_nemotron_steerlm_8b',# 'playground_nv_llama2_rlhf_70b',# 'playground_llama2_code_34b',# 'playground_mixtral_8x7b',# 'playground_neva_22b',# 'playground_steerlm_llama_70b',# 'playground_nemotron_qa_8b',# 'playground_sdxl']Model typesAll of these models above are supported and can be accessed viaChatNVIDIA.Some model types support unique prompting techniques and chat messages. We will review a few important ones below.To find out more about a specific model, please navigate to the API section of an AI Foundation Modelas linked here.General ChatModels such asllama2_13bandmixtral_8x7bare good all-around models that you can use for with any LangChain chat messages. Example below.fromlangchain_nvidia_ai_endpointsimportChatNVIDIAfromlangchain_core.promptsimportChatPromptTemplatefromlangchain_core.output_parsersimportStrOutputParserprompt=ChatPromptTemplate.from_messages([("system","You are a helpful AI assistant named Fred."),("user","{input}")])chain=(prompt|ChatNVIDIA(model="llama2_13b")|StrOutputParser())fortxtinchain.stream({"input":"What's your name?"}):print(txt,end="")Code GenerationThese models accept the same arguments and input structure as regular chat models, but they tend to perform better on code-genreation and structured code tasks. An example of this isllama2_code_13b.prompt=ChatPromptTemplate.from_messages([("system","You are an expert coding AI. Respond only in valid python; no narration whatsoever."),("user","{input}")])chain=(prompt|ChatNVIDIA(model="llama2_code_13b")|StrOutputParser())fortxtinchain.stream({"input":"How do I solve this fizz buzz problem?"}):print(txt,end="")Steering LLMsSteerLM-optimized modelssupports "dynamic steering" of model outputs at inference time.This lets you "control" the complexity, verbosity, and creativity of the model via integer labels on a scale from 0 to 9. Under the hood, these are passed as a special type of assistant message to the model.The "steer" models support this type of input, such assteerlm_llama_70bfromlangchain_nvidia_ai_endpointsimportChatNVIDIAllm=ChatNVIDIA(model="steerlm_llama_70b")# Try making it uncreative and not verbosecomplex_result=llm.invoke("What's a PB&J?",labels={"creativity":0,"complexity":3,"verbosity":0})print("Un-creative\n")print(complex_result.content)# Try making it very creative and verboseprint("\n\nCreative\n")creative_result=llm.invoke("What's a PB&J?",labels={"creativity":9,"complexity":3,"verbosity":9})print(creative_result.content)Use within LCELThe labels are passed as invocation params. You canbindthese to the LLM using thebindmethod on the LLM to include it within a declarative, functional chain. Below is an example.fromlangchain_nvidia_ai_endpointsimportChatNVIDIAfromlangchain_core.promptsimportChatPromptTemplatefromlangchain_core.output_parsersimportStrOutputParserprompt=ChatPromptTemplate.from_messages([("system","You are a helpful AI assistant named Fred."),("user","{input}")])chain=(prompt|ChatNVIDIA(model="steerlm_llama_70b").bind(labels={"creativity":9,"complexity":0,"verbosity":9})|StrOutputParser())fortxtinchain.stream({"input":"Why is a PB&J?"}):print(txt,end="")MultimodalNVIDIA also supports multimodal inputs, meaning you can provide both images and text for the model to reason over.These models also acceptlabels, similar to the Steering LLMs above. In addition tocreativity,complexity, andverbosity, these models support aqualitytoggle.An example model supporting multimodal inputs isplayground_neva_22b.These models accept LangChain's standard image formats. Below are examples.importrequestsimage_url="https://picsum.photos/seed/kitten/300/200"image_content=requests.get(image_url).contentInitialize the model like so:fromlangchain_nvidia_ai_endpointsimportChatNVIDIAllm=ChatNVIDIA(model="playground_neva_22b")Passing an image as a URLfromlangchain_core.messagesimportHumanMessagellm.invoke([HumanMessage(content=[{"type":"text","text":"Describe this image:"},{"type":"image_url","image_url":{"url":image_url}},])])### You can specify the labels for steering here as well.  You can try setting a low verbosity, for instancefromlangchain_core.messagesimportHumanMessagellm.invoke([HumanMessage(content=[{"type":"text","text":"Describe this image:"},{"type":"image_url","image_url":{"url":image_url}},])],labels={"creativity":0,"quality":9,"complexity":0,"verbosity":0})Passing an image as a base64 encoded stringimportbase64b64_string=base64.b64encode(image_content).decode('utf-8')llm.invoke([HumanMessage(content=[{"type":"text","text":"Describe this image:"},{"type":"image_url","image_url":{"url":f"data:image/png;base64,{b64_string}"}},])])Directly within the stringThe NVIDIA API uniquely accepts images as base64 images inlined withinHTML tags. While this isn't interoperable with other LLMs, you can directly prompt the model accordingly.base64_with_mime_type=f"data:image/png;base64,{b64_string}"llm.invoke(f'What\'s in this image?\n<img src="{base64_with_mime_type}" />')RAG: Context modelsNVIDIA also has Q&A models that support a special "context" chat message containing retrieved context (such as documents within a RAG chain). This is useful to avoid prompt-injecting the model.Note:Only "user" (human) and "context" chat messages are supported for these models, not system or AI messages useful in conversational flows.The_qa_models likenemotron_qa_8bsupport this.fromlangchain_nvidia_ai_endpointsimportChatNVIDIAfromlangchain_core.promptsimportChatPromptTemplatefromlangchain_core.output_parsersimportStrOutputParserfromlangchain_core.messagesimportChatMessageprompt=ChatPromptTemplate.from_messages([ChatMessage(role="context",content="Parrots and Cats have signed the peace accord."),("user","{input}")])llm=ChatNVIDIA(model="nemotron_qa_8b")chain=(prompt|llm|StrOutputParser())chain.invoke({"input":"What was signed?"})EmbeddingsYou can also connect to embeddings models through this package. Below is an example:from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings

embedder = NVIDIAEmbeddings(model="nvolveqa_40k")
embedder.embed_query("What's the temperature today?")
embedder.embed_documents([
    "The temperature is 42 degrees.",
    "Class is dismissed at 9 PM."
])By default the embedding model will use the "passage" type for documents and "query" type for queries, but you can fix this on the instance.query_embedder=NVIDIAEmbeddings(model="nvolveqa_40k",model_type="query")doc_embeddder=NVIDIAEmbeddings(model="nvolveqa_40k",model_type="passage")