Will: Hi, all, Will here. Let's build a travel assistant chatbot together. In the process, you'll learn some reusable techniques that'll be applicable if you're building any customer support chatbot or really any AI system that has any of the following characteristics. One, it uses tools that take actions on behalf of the user. So it's more than just a Q& A read only kind of chatbot. Two, it uses a large number of tools. And so it has a lot of difficulty, perhaps choosing the right one and also using that without more advanced techniques, three, your chatbot has to support different specific user journeys in your product. So each of these could be maybe a checkout flow management of a specific feature, et cetera. So it has to support a large number of attempts. We'll start by building a really simple travel assistant, show some limitations in its design, and then gradually add some complexity and control to help better support these types of needs. As always, these tutorials can be found in the Langraff documentation. This case is a customer support use case. As context for what we're building, I wanted to give a brief prehistory of chatbots. So traditional chatbots you typically interact with on the phone or through something that you might build in Dialogflow, often take these behavioral tree type structures or graphs, where the user will have some sort of question, the intent will route it to a very specific skill, and then the chatbot will say, That contacts is propagated down on each of these functions, then solves that dedicated task. The problem with this is you can see is it gets really complex and wherever you do want to be jumping between tasks that context isn't fully shared. It just isn't sufficiently flexible and supportive to really make it a great user experience, which is why everyone's frustrated when they end up with a robo. Experience on the phone. Fast forward to 2022, chat GPT jumps on the scene with this radical idea of eliminating all this complexity of intent detection and entity linking and routing and response generation. And it just puts the user directly in contact with a, with a user. very powerful LLM to be able to answer questions and other types of things. And the problem you might see with this is it's not actually grounded in anything. You could ask this same system in a hundred years and it would still think it's 2022. And it isn't able to actually take actions on behalf of the user. So that brings us to today. We're going to go through a series of flight assistant Designs that have increasing complexity, starting from a very simple tool using conversational agent up to a one that has a little bit more specific task specific context aware workflows that more closely mirror. Honestly, this original design to be able to balance this kind of context, sharing and capacity and expressiveness with the ability to design in a way that Good focused user experiences. So we'll start out with this zero shot tool executor. It's just a graph of two nodes. You've got the assistant, which is really just an LLM and a prompt. It's able to decide to call a tool or respond directly. These tools are all provided at the LLM at once. So it has to do two tasks. Really? It has to decide which and. Which tool and if a tool to call, and then also the arguments there. And then it has to be synthesizing response based on the actions from that. And it can do this in a loop as you probably are familiar with, this will have some limitations that we'll discuss later. And so we'll add in this user control flow so that whenever the assistant takes an action, the user is able to be involved in the loop to confirm whether they can actually take that action. So then the assistant isn't going off and. Spending all the user's money trying to book different flights without the user actually having a say on whether or not they want that. And you're not relying purely on the LLM then in this case you're adding some additional code safeguards to make sure that it's not going off the rails. The third part will be adding a little bit more scoping. So a lot of these types of tools that we're giving the flight assistant, they're going to be like searching over flights, searching over user details, car rentals, excursions. Those things don't really need user support or confirmation. So we're going to split those out into a simple safe tools and sensitive tools type of design. And this is all right. This works pretty well. The problem is that if each of these experiences becomes quite complex, maybe you have specific instructions, you know, re including company policies, all of the above, you start to overload this single prompt where it can't handle all of these experiences. And still have this reliable, delightful user experience that you want. So to be able to better handle that, you can separate all of these different focused user journeys into skills or workflows, or you can even imagine this since this is in fact, the design as a multi agent workflow. So, you have this primary assistant that the user can interact with, and anytime the user needs something that can be satisfied with one of these skills, it'll route the intent to these, and then these transparently can interact directly with the user, they can use their own scoped tools, and they can guide the user through that journey. Since this is all a single state graph, it's going to be managing state transitions throughout, and then work really seamlessly. You'll probably notice that we're going from Least to most complex today in this tutorial and I typically recommend following a similar journey as you build out your own application Simply because assistance these these LLMs are quite powerful And all this additional code that you're writing can actually inhibit the performance of the LLM if you're not measuring it quite carefully You are taking trade offs, of course in that you're putting a lot of the code Decision making and experience complexity on the LLM itself. But as the LLM quality improves, you may find that some of these more simpler designs become more and more expressive and capable of handling these things that today require more explicit, explicit workflows. All right, we'll download the notebook from the docs and then get started. If you're following along with the code, I'd recommend liberal use of these contents navigators because there is a lot of code that's boilerplate just to set up all of the tools for this use case. We want it to be a little bit more similar to what your use case might look like by providing a lot of these tools and database accessories here but it does get a little verbose and increase the time it takes to get to the actual agent design parts. All right, so over in our notebook, we'll first start by setting up the prerequisites. Basically before designing our first agent, we're going to create the database, we're going to create all the tools which we'll reuse across all of the sections in the tutorial today, and we'll install all the prerequisites of course. So, I've already run this, I'm going to set the API keys, basically we'll be using Anthropic for the LLM to access Cloud, and we'll also use Tavaleet as an extra tool the agent to optionally do some general web research. Next we'll fetch the database, I pre packaged this simple SQLite database in Google Cloud Storage for us, so you can just fetch it there and reuse it, you don't have to go and find some other location to download it from. And then we'll start defining our tools. The first tool is this retriever over company policies. Again, for this use case, we're going to be designing a flight assistant. And so the company policies are some Swiss Air company policies about when you can rebook, what kinds of cancellation policies they have, et cetera. These are useful for allowing the bot to do simple Q& A and it can also help direct it to know what kind of actions it's allowed to take. It's not. sufficient to provide policies as instructions, since the LLM can always choose to ignore it. So just remember when you're defining the tools to also enforce that these rules are actually applied. All right. The, the next step is really not that important. We're going to be defining all the tools for the flights. The hotels, the excursions, the car rentals. Each of these are basically CRUD end points where you can be searching over the different available things. You can be getting the user's current information. You can be updating their bookings. You can be deleting their bookings, et cetera. I guess one thing I will point out since it's a common question. Of how do you add configuration to tools? We'll be using this in shared config to get configurable information, to see the signed in passenger ID. So in our case, our graph, we're actually passing in the current user or passenger ID directly into the inputs. The other one doesn't really have to know that the current user is signed in and has a specific UUID. We'll just be using this in configurable params. Alright, so do car rental tools, hotels, excursions, and then we'll have a couple utilities just for this tutorial. One is just printing these events, which just makes it look pretty in the notebook. The other two, or really one, is that we're going to add a fallback so that any time the agent generates a, an input to the tool that will raise an error. We're going to have these fallbacks that just adds that error to the message list. So the agent is able to retry. And learn from these mistakes. If you're building this in a real production use case, you'll likely want to find as many of these error cases as possible through a good evaluation harness, like Langsmith and then build this into a few shot examples or better instructions or better designed tools so that it doesn't actually run into these very often. For the sake of the tutorial, it's always good. To have defense in depth and have these fallbacks in there. All right. So now we can finally start defining our agent. This is going to be a simple conversational zero shot agent. It'll take the LLM and prompt as the assistant node. And then we'll be putting all of our tools in a single tool executor node. The assistant can choose to respond directly to the user, or it can choose to pick one or more of the tools, execute them, and then observe the responses to help assist the user. With all these line graph tutorials, I start by defining the state, really. The state determines the interface for every node in the graph. It also defines how that state is updated whenever a node returns some values. In our case, we're going to have a very simple message graph that just has a list of chat messages. And we'll say that any time a node finishes, we're only going to be appending these messages to it. So this is just a function here. And this is Python's annotated syntax, if you're not familiar with it. So we'll run that. Next, we'll be defining the agent or the assistant. Again, this node here. Really, what we're doing here is we're going to say, so the assistant is taking this runnable, which is a prompt in an LLM. We're going to call it. And then in the typical case, it'll just return that new AI message, or assistant message, and add it to the state. Sometimes when you're using haiku or one of the weaker models, it might return an empty response. And so we don't really want that in our, in our message. So I do have a little retry loop just to get it to respond with an actual response. We've got sonnet here we'll be using by default. If you want to be running this cheaply and fast, you can use haiku. And then you have the prompt, the tool and the LLM. Here, we're using the binds tool syntax, which tells the LLM that these are all the schemas that it can choose from in order to generate that JSON payload. The prompt we're defining here is not super optimized, but it instructs the agent who they are. It instructs them that. The user has specific information and it gives the current time so that it's able to reason about relative questions like, Oh, what's happening tomorrow? Like, how long do I have between this? Et cetera. And this, we have a whole bunch of tools here that we're giving to the agent. So we'll see how well it's able to perform with all of these. So we've got this runnable. Now it's time to define the graph. This one's going to be pretty simple. We start by defining this builder object, which is like a state graph. We're saying that all the nodes are going to receive this state object. And then we'll define the two nodes. There's the assistant node and the action node. We'll say that we'll start. Anytime the user sends in an input, it's going to go to the assistant and then we're going to say, based on this tool's condition, if there are tool calls in the output of the assistant node, then we'll route to the action node again, so they're a tool executor. Otherwise, we're going to end. By ending, this means it's just going to cease executing and the user can then decide to respond or the conversation can end. Anytime we do take an action, we can go directly back to the assistant. Finally, we'll be creating this check pointer. The check pointer is memory for the graph. It saves the entire graph state after each step. In our case, this is really just a chat message history, but again, the state can be arbitrarily complex, and all of that is cached in. So your, your agent can just be, be persisted, stopped, resumed at any point in the future. I'll run that. Now we'll be drawing the graph so you can just visualize it again. This is a very simple agent that's just this little loop. That optionally returns as soon as it's taken all the actions it needs. Now, for all of our parts, we're going to have this simulated conversation between the user and the assistant. If the assistant goes off the rails, this might diverge a little bit, but that's okay. We'll catch that in the trace and in the printouts to show that, oh, maybe this isn't the most realistic response in a given point in time. We're also going to be copying this backup database that we copied over previously. And this is really just so that every time we go through these parts, we're starting from the same point and we can really compare apples to apples in a better way. For this tutorial, we'll be this passenger in the database. We currently do have a flight booking, and we want to be asking questions about updating it and making other trip plans. Finally, this thread ID is going to be unique per conversation. This tells the checkpointer that we define up here To save all of the checkpoints under this thread, we can then later update and we can fetch all of these checkpoints by that thread ID. So you can do things like fun, like time travel and resumption and all that. Alright, let's run this cell now. So you can jump over to Langsmith to see some examples. As you can see this most recent invocation. The LLM has a lot of tools to choose from, and it has to know how to use each one, and it has to know how to respond to the user in a really good way which is just a lot for the current crop of LLMs. You can see that all the chat history and state is saved over time, so it is maintaining this conversation over time. But it's like, it's going and it's using its pre trained knowledge to like share different information. It's sometimes going off the rails and how it's deciding certain actions. I could spend a lot of time prompt engineering and maybe get it reasonably good using this setup. But another way to approach this is to really focus on specific floats, which we'll get to in part four. The other things I want to point out are that it's not asking user confirmation and it's, and it's not great on some of these vertical use cases and user journeys. Which brings us to part two. So we had the zero shot react loop. Let's add some user confirmation. This just lets the user go and confirm whether the agent is allowed to take any action on their behalf. So as before, the user sends a message, the assistant can decide whether to respond directly or whether to invoke a tool and then, and then choose, choose one or more tools. It'll send those over to the tool executor. The tool executor actually calls the APIs and then returns the results, the assistant, and then it can decide whether to continue using these tools or whether to respond. In this case, the user actually gets to sit in and listen or listen in on this tool node, and we'll be able to approve, deny, or even update the calls to the tools if they want to have that last say in what's going on. This will be done through the interrupt before action in line graph. And we'll see what that is in a minute. So let's start by defining the state and assistant. This is really almost identical to part one with two exceptions. One, we're going to add this user info field at the beginning. And this is really saying that We're able to preemptively provide some user information to the LLM, so it doesn't have to go and do that that initial query for getting the user's flight details. If you recall in the conversation above, it had to go and search for that, and then that took an extra step, and it really should always be situationally aware about what the user is intending to do as much as possible. The second difference here is that we're going to be using that state directly in the assistant. Rather than giving the user info and popping the prompt there. So we'll define this, mostly identical again. Now we'll define the graph. So there's two new things about this. Again, we're adding this FetchUserInfo here as a separate entry node. And this means that we'll always have the current user's state or context available to the assistant. The second difference here is we're adding this InterruptBefore. What this tells the graph is that any time the control flows into this tool node, we're going to stop and let the user approve or deny it. What it really means is the graph stops executing. And since we're persisting the state to the checkpoint, anytime we invoke again, it'll start from that node rather than from the top, rather than from the start. You can see our graph here. It looks mostly the same. We've added this additional node there. Now we'll run through the same questions. We'll copy over the database again, just so it's starting from the same point. And we've added some additional information here to the database. Let the user actually interact. All right. So despite having that information provided, it's asking for our confirmation. So we'll say, yes, we're going to search for our flight details. You're allowed to do that. Now it's going to be looking at the policy. Sure. You can look up the policy. I think you're getting the sense that. The user being involved in the flow for every single step is maybe a little bit overkill. So am I allowed to change my flight? Yeah, I can. All right. So if you continue running, the agent's going to continue to ask for your permission anytime it needs to do any action from making a reservation all the way to just looking up a simple policy. This is a little bit too much work for the user. We want the agent to be a little more autonomous than this. We just want to make sure that it's not taking actions that make my life as the user worse in the next section. We'll make a conditional interrupt. This is a simple change using a design pattern, just splitting the tool nodes into two so that we can interrupt on the ones that are what we deem sensitive. So right actions, this is one way of, of introducing conditionality on interrupts and user in the loop workflows. So once again, our user will ask a question. You have the assistant, if it's able to choose to respond directly, or it can call a tool. In the assistant's mind, it's still seeing all these same, the same tools here. It doesn't know that it's split here. That, that really doesn't matter. It's the LLM. But whenever it does call a tool, we'll have this conditional node that looks it up and sees is the tool that's called one of these sensitive tools, or is it a safe tool that it can go and just do search or retrieval? If it's a safe tool, it can go there and then return, and then the user can, or the assistant can respond directly. The user doesn't have to be confirming in this workflow. Otherwise, if it's going to make a change, like a cancellation or booking a reservation, it goes to these sensitive tools. Here we have the route, the interrupt again. So the user can approve or it can decide, no, I don't actually want that. I'm going to want something else. Or the user can even update the state directly to change the parameters specifically before they go into the tool. So it really has like three different choices. Once the tool is called or once the user denies it, the assistant can then decide whether to respond saying this question is satisfied or it can decide to try again, get some more information, et cetera. All right. So. The state here and everything is identical to part two. We have the messages that we've defined. We have the user info so that eagerly populating the information about the user. We have the assistant runnable. This is all unchanged. We're going to use like sonnet. We're going to have all these things. The one thing that is different is now we split this tools into two lists. We've had the safe tools, which is the general search engine. Fetching user information, searching, looking up all these read only operations that we consider to be fine, doesn't require user approval. We've then split out these updates and cancellations and creation events into sensitive tools, which will organize that really. As I mentioned before, the LLM actually doesn't know the difference. We're going to have all these tools provided to it. It's still picking the same as before. What's different is that our graphs, our orchestration, actually will. be able to, to federate this out into different nodes so that the user experience is slightly different. All right, time to define the graph. This is again, mostly the same as in part two. We have the state, we're passing it the state we've created. We have this user info node that we're going to enter on and say every time we start this graph, we're going to make sure that we have the most up to date user information available so the LLM can see the user's current flights and etc. This is useful, again, as an aside. If the user goes and makes changes outside of this assistant workflow, it has up to date context. It doesn't have to be getting that all from the conversation. Now we're going to add the assistant, and we're going to add two different nodes, safe nodes and sensitive nodes. So the safe tools here are the ones that we do for read only operations. The sensitive tools here are the ones that have write operations. Now we'll have the edge for the FetchUserInfo to Assistant. This is just saying every time you enter and we get that eager operation of the UserInfo, we'll immediately go to the Assistant, and then we'll define this conditional edge. Previously, we had just been using this pre built tools condition, which looks for the presence of tool invocations in the AI output in the messages list. Now we're going to extend this and say, okay, yeah, if there's no tool calls, we'll still return just as before. However, if We're going to filter on the name of the tool calls and say, if it's in the sensitive tool names, we'll route this to the sensitive tools node. Otherwise, we can give it to the safe tools node and continue as before. Finally, we're going to compile with a checkpointer again so that it has conversational memory and persistence. And again, this checkpointer is saving not only the conversational memory, the messages, it is also caching this user info that we've defined in our state here. And again, the more fields you add in here, all of this is checkpointed alongside it. So we'll create this. You'll see our graph has added a node here. You have the assistant. It's able to route either to the sensitive tools, to the safe tools, or back to the user. Remember, we're interrupting here only on the sensitive tools node. All right, let's go through this conversation again. Hi there, what time is my flight? You see it fetches the user's flight information, and it looks up policy all without me as the user having to get involved. Great. It's autonomously able to execute stuff. Let's make sure that it is actually going to stop for us before it goes and makes some sort of change. Alright, it's trying to cancel my ticket first. Notice that this time it is trying to cancel before it is updating. This is different than before, just because the LLM is a little bit random. Before it was just directly updating it. This leads us to a point of inadequacy of this simple flat design and that it's pretty hard to have the large prompt that completely defines each intended user journey. The LLM is likely to ignore some instructions or confuse instructions from different sections that might not apply across each experience. All right, in this case, I'll just approve and go through. All right, we've made it through the conversation. Let's jump over to Langsmith to check out the trace. We'll inspect this final one. You can see again, we're inputting null. It's just fetching all of the loaded messages from memory, from the check pointer that we've created. And it's calling this sensitive tools or booking excursion in this case. We're inputting null here because we're approving the current plan. All right. As you see before, the LLM has all these different tools to choose from. It goes in it, it does a decent job of answering the user's initial questions. This time it actually chooses to try to first cancel and then rebook afterwards rather than rebooking directly. This is kind of purely LLM. So we might want to have a more consistent user experience in this kind of task, since it is a common intent. It is able to successfully rebook and then it was looking up some of this information. It does successfully use the search tools and car rentals tools there. It gets to the, it gets to the search tools for trip recommendations and it has a hard time properly formatting this. We can improve the tool description or improve the prompts here, but it's kind of hard to do because we have already about 16 tools or so that are provided this LLM. And you can see we get some inconsistent experiences overall. So again, this worked pretty well. The user is able to confirm or deny any actions. So in the end, if the LLM does decide to cancel, then rebook, versus rebook directly, it may not be that important as to the outcome. It does lead to a less delightful and less controlled user experience. And whenever you want to be building a great you want to be able to shape the way that the user interacts with all of your products and make sure that you can really create a differentiated experience on each of these common user journeys. So the next section we want to be showing a simple way to create these specialized workflows. You can think of these as skills if you're thinking about it in the sense of like a serial Alexa type app. If you are coming from an agent background, this is in our case, a multi agent workflow. But this is more powerful in that each of these delegated flows doesn't have to be agentic. It can be a strict user checklist that you're going through. It doesn't even have to use an LLM. Really what this is, is a way of offloading all of the decision making and execution and planning faculties from this main assistant onto specific workflows where you as the designer, as the developer can inject the expected behavior into the prompt tools and sub graph structure. An analogy I'll make is that if you're creating a website, You're not going to load the checkout workflow with the search workflow with the, like, other, other types of experiences all on a single page. You're going to have specific pages for these and specific modals and designs around these user journeys. The current LLMs really aren't good enough to be able to separate concerns in this way so that whenever the user does enter a workflow, it's only able to attend only that relevant context and really. Perform reliably across these large number of user queries on all of these common intents by separating out concerns and creating these workflows You can really target Common intents and make sure that those experiences are great You can optimize these independently. You can evaluate them independently. Without worrying about interrupting or degrading the overall user experience. This does create more code complexity. It's more stuff to manage. And so there is a trade off here. I wouldn't start directly with this pattern if I'm just building a simple Q& A app. But if you have found that your LLM is starting to struggle with separating concerns and handling all these different Various user intents. This is a design pattern that I would consider adapting. All right. As always, we'll start. All right. As always, we'll start by defining the state. The state has the same fields as before. It has the messages as the user info. You'll notice we added this dialogue state variable here. And what this is just a list of strings. What this tells the graph is any time the user sends a message to it, Where should we route it? Where should we give control? Before we were always first writing to the assistant and then it would go to the tool nodes and stuff like that. Now the control flow can be delegated to the flight booking assistant. It could be the car rental assistant, the hotel booking assistant, all these different workflows. And so this, this state lets us condition on that and route it to the appropriate place. As before we're defining how the state is going to be updated using a custom function. Where we're saying if it, the, if the update is saying pop, then we're going back to the primary assistant. Otherwise, we're adding the, to the stack here. This is just a neat, a concise way of representing where we currently are in the graph. Next, we're going to define each workflow. For the assistance that power each workflow we'll make a flight booking one, a hotel booking, one car rental excursion, and finally the primary assistant that again routes between these. If you've watched some of our other videos or read some of the docs, you'll notice that this is actually a supervised design pattern where the primary assistant gets to choose whether to respond directly or whether to delegate work to each of the underlying workflows. I claim that this is more powerful than a multi agent pattern because these don't have to be agents. Agents are a specific way to implement each of these individual flows. We'll define this mostly the same as before. Each of these flows will have a prompt. They'll have a list of safe tools and sensitive tools. So separating out the read only from the right axis ones, as well as the, the LLM where you're going to bind it. So you're going to provide the schemas saying these are the types of functions you're allowed to optionally select from. We're also going to add this additional function here, we define as a pedantic model, called completer escalate which is the name I chose. This basically says, hey, if the user decides to actually change their mind, this is how you go and, and pull the rip, this is how you go and pull the rip cord, or change the context back to the primary assistant. It might be that I, as the user, will go and say, hey, I want to book a flight to New York, On Thursday, the primary assistance is great. I'll connect you to the primary booking assistant. I'll say, sir, what are your flights? Maybe what are the prices? Actually on second thought, I need to save money. I'm not going to do this. So the flight booking assistant needs to way to say, okay, we're going to exit this guilt. That's exactly what this is. We'll be reusing this completed escalate task and all of these delegated workflows. So we'll define the hotel booking assistant the same way, car rental, same way, et cetera. You'll notice that since we've separated concerns, we can have different prompts for each of these ones. And then. If you wanted to take this a step further, you could optimize, provide more demonstrations, make this arbitrarily optimized for these very targeted use cases, because it's all available. If we were using the current zero shot pattern, you'd have to overload all these different things in a single prompt, which perhaps will be possible with later LLMs when they get more powerful. But today, it becomes a little bit confusing and hard to maintain, and definitely hard to evaluate, because you're constantly making these different changes for each feature experience, which might be different teams that you're developing with and It's much easier in code to be separating the concerns and optimizing them separately. Finally, we'll make this primary assistant It'll have all of these two whatever delegated assistant functions that it uses in order to indicate Okay, we should route these different things In our case again, we're using LLM to do the routing and You might get better performance using some sort of semantic routing with an embedding classifier or something like that. I think it's easier to demonstrate this supervisor pattern because I don't have to train a classifier. But we'll release some other examples later on how you could apply that here. The primary assistant also has a couple of these lookup tools, because if the user does want just general factual knowledge, it might be faster if the primary assistant actually is able to directly answer and do some queries on their behalf. If you end up with a lot of these tools again, you might want to just separate those concerns out. This is all stuff that can be empirically verified with an evaluation framework like LangSmith . So, as we mentioned before, and I'll go back up to the agent state. So as we mentioned before, and I'll go back up to the agent state, we have this message list that's continuously appended to. This is shared in each of these workflows. So all the user's interactions with the flight's assistant can be shared with the primary, which then can be shared with the hotels. So then all these workflows actually know the entire user journey as it's gone through each of these different experiences. This is pretty powerful. One of the frustrations that I often have with tools like Siri is that you'll get into the Spotify tool and then you'll get into maps and then you'll ask the change, the, the, the music and it'll forget that you're there. It just, it just doesn't share context across it. One problem is that if you're sharing this huge list, the LLM can still get a little confused as to its current role. Even if you do have a very explicit system prompt because of that, we're going to add this. Entry node to each of these workflows. What this does is it adds a tool message saying the assistant is now whatever assistant it is reflect on the above conversation, use your tools to execute the task. Basically a reminder of the information that we've provided in the system prompt. I found that at least for Claude, having that reminder at the end of the conversation is relatively effective in Reminding it who it is and what it's able to do. It makes it more reliable if you only put this in the system prompt and the conversation gets quite long, it can start acting and trying to invoke or hallucinate tools from the other assistants as it's navigating this complex user experience. So to find this utility, they'll essentially create the entry node for each of these workflows. Finally, it's time to start creating the graph as before. We're going to start with this user info node just to eagerly populate all this information. Next, we'll add each of these delegated workflows, starting with the flight booking assistant. As I mentioned before, you start by having this enter operation again. This will create a tool message that says you are now the flight and updates and booking assistant. Use these tools to help satisfy the intent. We'll add the actual assistant note here. In this case, update flight. We'll add the sensitive and safe tools and we'll add Add the conditional edge. So this itself is an agentic loop saying it can go to one of the tool nodes and leave the skill, or it can respond directly to the user. Even once this workflow does complete, we want to make sure that we update the state so that it is tracking where it is. So we have this pop dialogue state thing that adds a tool message saying, okay, this workflow is now complete, sort of packages it up. And then it also adds this pop operation to the state. Recall when we defined this state before. Pop here is just going to return everything except for the last list here. So if we've started with assistant and we've gone to update flight, it'll then remove the update flight and it's just assistant in the list, for instance. We'll define other, the other delegated workflows in the same way. So car rentals, hotel bookings, excursions, et cetera. And finally, the primary one. A lot of this logic is repeat. You'll see the primary assistant has a more complex switch because each of these tool calls will route to one of the delegated workflows here. You can probably factor this out into some sort of dictionary and make it a little bit more concise. It also can directly respond to the user. It can also actually use its own search tools, if you remember. Finally, we're going to add this check pointer. And we're going to say interrupt before any of these sensitive tools. If you render the graph, it looks something like this. I'm going to try to zoom in a little bit. So you can see it goes to the Fetch User one. Then it goes to the Primary Assistant. The Primary Assistant optionally routes to any of these. Entrance nodes to these workflows, or it can route directly to the user. Once you're in one of these workflows, it can go to one of the tool nodes. It can leave. So go back to the primary system, or it can actually interact directly with the user. And this is mirrored across each of these specific workflows again. So that was a lot. Let's see how it performs. All right. So let's jump over to Langsmith to look at how this new version did. And before I get too far, remember I did have separate prompts for each of them. I didn't really optimize it super well. I think there's a lot of improvements and a lot of gains that are left on the table right now. Just because I hadn't, didn't have time to optimize them. If you're building something real and you've already separated it out, I think you have a lot of room for improvement by just improving the prompts, improving the tool definitions, and then really focus on each individual module that you can then help climb within an evaluation tool like Linksmith. All right, we'll just look at the trace here just to see how it did. You can see the user's going through, it says, what time is my flight? Seems like it already has the information. So it's searching the flights just to get all the additional information, get some stuff back. And it says, okay, we've got all this. It seems like it's doing a decent job of entering these first questions that we've had. Great. I think that perhaps speaks to the reduced burden of choice since the top level primary assistant has fewer decisions it needs to make. Okay. Going into the flight booking flow. The user tries to update to one for later today. It actually tries and the tool itself enforces that we can't actually reschedule this. It's too soon to now. Remember that policy we talked about earlier. What we could do here, since we're already within this flow, we can just front load a lot of those policies, those relevant policies, so that the assistant doesn't have to go through this process. background take extra work before responding to the user, it can be more proactive about that. And that's what a lot of this separation can end up doing that I haven't yet integrated here. So similar to how we put that preemptive user info block before entering the graph as a whole, within each individual workflow, we can have some of these relevant context information, either in the prompt itself or in a retriever step or other sorts of explicitly designed steps, just so that the assistant can. would be as helpful as possible within the explicit context of the user providing. A lot of what this does then is it takes the burden off the user saying this is who I am, this is what I want, etc. The Assistant already knows because it's demonstrates its initial attent Intent? Users navigated to a specific web page or something like that and then it's able to fetch all that relevant context and bring it to the table. Okay, so the, the, the user then goes through, eventually it lists out the other the other possible flights and then updates it successfully. Great. Let's move on to the hotel booking one. Okay. So once it does book it, it calls this completed escalate. It goes back to the primary agent. Remember in this completed escalate one, escalate step. It is a tool message that's injected saying resuming dialogue with a host assistant, the primary assistant here. Please reflect on that. So the user asks about lodging and transportation. Great, so now it routes to the hotel booking assistant. This is another specialized workflow. In the enter step, we've added this tool message. It says the assistant is here, reflects, continue to help, etc. This just cues in this assistant again as to what it's supposed to be seeing. The AI then says this, it goes through, tries to help. Seems like it does a pretty good job. It's pretty good. Focused here. It sees that there's no explicit mid scale hotels, but there's an upper mid skills here, here, and it's able to be doing these queries in the tool goes through this flow. We get to the car options here. It looks like there's actually a little bit of a hiccup in our design that we could improve. Again, since it's all modular, easier to make these improvements. Once we do see them, it seems in this case, the assistant didn't actually execute or finish once it didn't make the booking, we could actually do hard coded this in so that we never run into this problem. So as soon as the booking is made, automatically route back to the primary assistant, and then it's all up to that to decide who to route to next. Haven't put that in right now. Ask about inspirations. It goes through, it doesn't get anything for location based services. Again, we can improve the prompt here if you want. It's all very scoped and then it, you know, it's second search. It gets it a little bit better and then you get all this information from the tools. Again, it's much more reliable than it was whenever it was the full primary assistant trying to interact with all these tools. It's going through, it goes through, it makes some reservations. It succeeded in pretty much all the tasks of this conversation. Alright, so where does this leave us? We've built a travel assistant, starting from a simple zero shot agent with tools that can engage in chat and have memory and all those nice things. We then made the observation that actually this agent shouldn't be fully autonomous. The user should have a say as to whether it wants to take the actions that impact things. So then we added interrupt. The user could go and Make a change, accept or deny, do all these types of things before the assistant makes any actions. We made the observation that actually we don't need these interrupts on all the actions the agent takes, only on a set of sensitive ones that will change the data that the user cares about. So then we designed our graph with two different nodes, one with the sensitive tools, once with the safe tools. We only interrupt on the sensitive tools, so we have the user the right to change those calls. After that we made the observation that actually the assistant is kind of unreliable in routing these things and using the tools. And so, if you're going to be a user coming to it, or if you have thousands of users coming to your assistant every day, probably a large number of them are going to get results that don't make sense. Or if you're the same user doing it a couple of times, you'll have very inconsistent experiences. And I think all of this leads to a degraded experience as a whole. You also, the assistant also has to be relearning all these things anytime it's invoked, because it's not able to be gaining and optimizing for these explicit workflows separately. So after that, when we split all these concerns up into a separate, multi agent stage where you have the primary assistant delegate to different workflows that you can create centered around explicit known user journeys, booking flights, booking cars, hotels, excursions. Then the top layer, you can just route based on the user intent. In this case, we used an LLM. You can use a simpler, cheaper method like an embedding model as well, just to intent classify and route appropriately. And then each workflow you can design and optimize in isolation. You can say once the user has demonstrated they want to commit to buying a hotel, or buying a flight, or renting a hotel What are the types of things we need to add to the prompt or the tools we need to do? How can we optimize these to make this experience as good as possible? And then once you're in these, as long as you have the escape hatch for it to go back to the primary assistant, shift over to the other assistants, the user has a really consistent experience within that, but while still allowing that flexibility that these LLMs offer by sharing context along the entire user journey, even if they do go through multiple flows. I think this gives you something somewhat of the best of both worlds, where you can take advantage of the long context windows and great reasoning ability, in quotes, of LLMs by, by letting it go and manage that conversational state rather than explicitly tracking it too hard while still being able to put a lot of your dark knowledge and design and intent around your user journeys and your product to good use through well defined flows. It also gives your user full control on the actions that it cares about. Letting your assistant automate what the user doesn't need to do. You can prime it with context. So the user doesn't have to be inputting and staring at a blank blank box, wondering what kinds of questions it can ask once you're in specific flows, it can then save. These are the types of things it's designed to, to, to work on, et cetera. So overall, I think this is a set of design patterns, a set of trade offs you can make that can overall give you better control over the experience of, of how a user interacts with your product and help eventually build much more reliable chatbots that can assist both in Q and a tasks, but also in accomplishing actions such as bookings checkouts, et cetera. So, that's all we have for today. If you have any questions or comments, leave them in the comments below. Just as a reminder, I'm Will. We're going to be posting a lot more of this content about LangSmith, about LangSmith, and all of the things that we'll be developing. So, if you have any requests on what you'd want to see, just leave them in the comments. And see you later. Will: Hi, all, Will here. Let's build a travel assistant chatbot together. In the process, you'll learn some reusable techniques that'll be applicable if you're building any customer support chatbot or really any AI system that has any of the following characteristics. One, it uses tools that take actions on behalf of the user. So it's more than just a Q& A read only kind of chatbot. Two, it uses a large number of tools. And so it has a lot of difficulty, perhaps choosing the right one and also using that without more advanced techniques, three, your chatbot has to support different specific user journeys in your product. So each of these could be maybe a checkout flow management of a specific feature, et cetera. So it has to support a large number of attempts. We'll start by building a really simple travel assistant, show some limitations in its design, and then gradually add some complexity and control to help better support these types of needs. As always, these tutorials can be found in the Langraff documentation. This case is a customer support use case. As context for what we're building, I wanted to give a brief prehistory of chatbots. So traditional chatbots you typically interact with on the phone or through something that you might build in Dialogflow, often take these behavioral tree type structures or graphs, where the user will have some sort of question, the intent will route it to a very specific skill, and then the chatbot will say, That contacts is propagated down on each of these functions, then solves that dedicated task. The problem with this is you can see is it gets really complex and wherever you do want to be jumping between tasks that context isn't fully shared. It just isn't sufficiently flexible and supportive to really make it a great user experience, which is why everyone's frustrated when they end up with a robo. Experience on the phone. Fast forward to 2022, chat GPT jumps on the scene with this radical idea of eliminating all this complexity of intent detection and entity linking and routing and response generation. And it just puts the user directly in contact with a, with a user. very powerful LLM to be able to answer questions and other types of things. And the problem you might see with this is it's not actually grounded in anything. You could ask this same system in a hundred years and it would still think it's 2022. And it isn't able to actually take actions on behalf of the user. So that brings us to today. We're going to go through a series of flight assistant Designs that have increasing complexity, starting from a very simple tool using conversational agent up to a one that has a little bit more specific task specific context aware workflows that more closely mirror. Honestly, this original design to be able to balance this kind of context, sharing and capacity and expressiveness with the ability to design in a way that Good focused user experiences. So we'll start out with this zero shot tool executor. It's just a graph of two nodes. You've got the assistant, which is really just an LLM and a prompt. It's able to decide to call a tool or respond directly. These tools are all provided at the LLM at once. So it has to do two tasks. Really? It has to decide which and. Which tool and if a tool to call, and then also the arguments there. And then it has to be synthesizing response based on the actions from that. And it can do this in a loop as you probably are familiar with, this will have some limitations that we'll discuss later. And so we'll add in this user control flow so that whenever the assistant takes an action, the user is able to be involved in the loop to confirm whether they can actually take that action. So then the assistant isn't going off and. Spending all the user's money trying to book different flights without the user actually having a say on whether or not they want that. And you're not relying purely on the LLM then in this case you're adding some additional code safeguards to make sure that it's not going off the rails. The third part will be adding a little bit more scoping. So a lot of these types of tools that we're giving the flight assistant, they're going to be like searching over flights, searching over user details, car rentals, excursions. Those things don't really need user support or confirmation. So we're going to split those out into a simple safe tools and sensitive tools type of design. And this is all right. This works pretty well. The problem is that if each of these experiences becomes quite complex, maybe you have specific instructions, you know, re including company policies, all of the above, you start to overload this single prompt where it can't handle all of these experiences. And still have this reliable, delightful user experience that you want. So to be able to better handle that, you can separate all of these different focused user journeys into skills or workflows, or you can even imagine this since this is in fact, the design as a multi agent workflow. So, you have this primary assistant that the user can interact with, and anytime the user needs something that can be satisfied with one of these skills, it'll route the intent to these, and then these transparently can interact directly with the user, they can use their own scoped tools, and they can guide the user through that journey. Since this is all a single state graph, it's going to be managing state transitions throughout, and then work really seamlessly. You'll probably notice that we're going from Least to most complex today in this tutorial and I typically recommend following a similar journey as you build out your own application Simply because assistance these these LLMs are quite powerful And all this additional code that you're writing can actually inhibit the performance of the LLM if you're not measuring it quite carefully You are taking trade offs, of course in that you're putting a lot of the code Decision making and experience complexity on the LLM itself. But as the LLM quality improves, you may find that some of these more simpler designs become more and more expressive and capable of handling these things that today require more explicit, explicit workflows. All right, we'll download the notebook from the docs and then get started. If you're following along with the code, I'd recommend liberal use of these contents navigators because there is a lot of code that's boilerplate just to set up all of the tools for this use case. We want it to be a little bit more similar to what your use case might look like by providing a lot of these tools and database accessories here but it does get a little verbose and increase the time it takes to get to the actual agent design parts. All right, so over in our notebook, we'll first start by setting up the prerequisites. Basically before designing our first agent, we're going to create the database, we're going to create all the tools which we'll reuse across all of the sections in the tutorial today, and we'll install all the prerequisites of course. So, I've already run this, I'm going to set the API keys, basically we'll be using Anthropic for the LLM to access Cloud, and we'll also use Tavaleet as an extra tool the agent to optionally do some general web research. Next we'll fetch the database, I pre packaged this simple SQLite database in Google Cloud Storage for us, so you can just fetch it there and reuse it, you don't have to go and find some other location to download it from. And then we'll start defining our tools. The first tool is this retriever over company policies. Again, for this use case, we're going to be designing a flight assistant. And so the company policies are some Swiss Air company policies about when you can rebook, what kinds of cancellation policies they have, et cetera. These are useful for allowing the bot to do simple Q& A and it can also help direct it to know what kind of actions it's allowed to take. It's not. sufficient to provide policies as instructions, since the LLM can always choose to ignore it. So just remember when you're defining the tools to also enforce that these rules are actually applied. All right. The, the next step is really not that important. We're going to be defining all the tools for the flights. The hotels, the excursions, the car rentals. Each of these are basically CRUD end points where you can be searching over the different available things. You can be getting the user's current information. You can be updating their bookings. You can be deleting their bookings, et cetera. I guess one thing I will point out since it's a common question. Of how do you add configuration to tools? We'll be using this in shared config to get configurable information, to see the signed in passenger ID. So in our case, our graph, we're actually passing in the current user or passenger ID directly into the inputs. The other one doesn't really have to know that the current user is signed in and has a specific UUID. We'll just be using this in configurable params. Alright, so do car rental tools, hotels, excursions, and then we'll have a couple utilities just for this tutorial. One is just printing these events, which just makes it look pretty in the notebook. The other two, or really one, is that we're going to add a fallback so that any time the agent generates a, an input to the tool that will raise an error. We're going to have these fallbacks that just adds that error to the message list. So the agent is able to retry. And learn from these mistakes. If you're building this in a real production use case, you'll likely want to find as many of these error cases as possible through a good evaluation harness, like Langsmith and then build this into a few shot examples or better instructions or better designed tools so that it doesn't actually run into these very often. For the sake of the tutorial, it's always good. To have defense in depth and have these fallbacks in there. All right. So now we can finally start defining our agent. This is going to be a simple conversational zero shot agent. It'll take the LLM and prompt as the assistant node. And then we'll be putting all of our tools in a single tool executor node. The assistant can choose to respond directly to the user, or it can choose to pick one or more of the tools, execute them, and then observe the responses to help assist the user. With all these line graph tutorials, I start by defining the state, really. The state determines the interface for every node in the graph. It also defines how that state is updated whenever a node returns some values. In our case, we're going to have a very simple message graph that just has a list of chat messages. And we'll say that any time a node finishes, we're only going to be appending these messages to it. So this is just a function here. And this is Python's annotated syntax, if you're not familiar with it. So we'll run that. Next, we'll be defining the agent or the assistant. Again, this node here. Really, what we're doing here is we're going to say, so the assistant is taking this runnable, which is a prompt in an LLM. We're going to call it. And then in the typical case, it'll just return that new AI message, or assistant message, and add it to the state. Sometimes when you're using haiku or one of the weaker models, it might return an empty response. And so we don't really want that in our, in our message. So I do have a little retry loop just to get it to respond with an actual response. We've got sonnet here we'll be using by default. If you want to be running this cheaply and fast, you can use haiku. And then you have the prompt, the tool and the LLM. Here, we're using the binds tool syntax, which tells the LLM that these are all the schemas that it can choose from in order to generate that JSON payload. The prompt we're defining here is not super optimized, but it instructs the agent who they are. It instructs them that. The user has specific information and it gives the current time so that it's able to reason about relative questions like, Oh, what's happening tomorrow? Like, how long do I have between this? Et cetera. And this, we have a whole bunch of tools here that we're giving to the agent. So we'll see how well it's able to perform with all of these. So we've got this runnable. Now it's time to define the graph. This one's going to be pretty simple. We start by defining this builder object, which is like a state graph. We're saying that all the nodes are going to receive this state object. And then we'll define the two nodes. There's the assistant node and the action node. We'll say that we'll start. Anytime the user sends in an input, it's going to go to the assistant and then we're going to say, based on this tool's condition, if there are tool calls in the output of the assistant node, then we'll route to the action node again, so they're a tool executor. Otherwise, we're going to end. By ending, this means it's just going to cease executing and the user can then decide to respond or the conversation can end. Anytime we do take an action, we can go directly back to the assistant. Finally, we'll be creating this check pointer. The check pointer is memory for the graph. It saves the entire graph state after each step. In our case, this is really just a chat message history, but again, the state can be arbitrarily complex, and all of that is cached in. So your, your agent can just be, be persisted, stopped, resumed at any point in the future. I'll run that. Now we'll be drawing the graph so you can just visualize it again. This is a very simple agent that's just this little loop. That optionally returns as soon as it's taken all the actions it needs. Now, for all of our parts, we're going to have this simulated conversation between the user and the assistant. If the assistant goes off the rails, this might diverge a little bit, but that's okay. We'll catch that in the trace and in the printouts to show that, oh, maybe this isn't the most realistic response in a given point in time. We're also going to be copying this backup database that we copied over previously. And this is really just so that every time we go through these parts, we're starting from the same point and we can really compare apples to apples in a better way. For this tutorial, we'll be this passenger in the database. We currently do have a flight booking, and we want to be asking questions about updating it and making other trip plans. Finally, this thread ID is going to be unique per conversation. This tells the checkpointer that we define up here To save all of the checkpoints under this thread, we can then later update and we can fetch all of these checkpoints by that thread ID. So you can do things like fun, like time travel and resumption and all that. Alright, let's run this cell now. So you can jump over to Langsmith to see some examples. As you can see this most recent invocation. The LLM has a lot of tools to choose from, and it has to know how to use each one, and it has to know how to respond to the user in a really good way which is just a lot for the current crop of LLMs. You can see that all the chat history and state is saved over time, so it is maintaining this conversation over time. But it's like, it's going and it's using its pre trained knowledge to like share different information. It's sometimes going off the rails and how it's deciding certain actions. I could spend a lot of time prompt engineering and maybe get it reasonably good using this setup. But another way to approach this is to really focus on specific floats, which we'll get to in part four. The other things I want to point out are that it's not asking user confirmation and it's, and it's not great on some of these vertical use cases and user journeys. Which brings us to part two. So we had the zero shot react loop. Let's add some user confirmation. This just lets the user go and confirm whether the agent is allowed to take any action on their behalf. So as before, the user sends a message, the assistant can decide whether to respond directly or whether to invoke a tool and then, and then choose, choose one or more tools. It'll send those over to the tool executor. The tool executor actually calls the APIs and then returns the results, the assistant, and then it can decide whether to continue using these tools or whether to respond. In this case, the user actually gets to sit in and listen or listen in on this tool node, and we'll be able to approve, deny, or even update the calls to the tools if they want to have that last say in what's going on. This will be done through the interrupt before action in line graph. And we'll see what that is in a minute. So let's start by defining the state and assistant. This is really almost identical to part one with two exceptions. One, we're going to add this user info field at the beginning. And this is really saying that We're able to preemptively provide some user information to the LLM, so it doesn't have to go and do that that initial query for getting the user's flight details. If you recall in the conversation above, it had to go and search for that, and then that took an extra step, and it really should always be situationally aware about what the user is intending to do as much as possible. The second difference here is that we're going to be using that state directly in the assistant. Rather than giving the user info and popping the prompt there. So we'll define this, mostly identical again. Now we'll define the graph. So there's two new things about this. Again, we're adding this FetchUserInfo here as a separate entry node. And this means that we'll always have the current user's state or context available to the assistant. The second difference here is we're adding this InterruptBefore. What this tells the graph is that any time the control flows into this tool node, we're going to stop and let the user approve or deny it. What it really means is the graph stops executing. And since we're persisting the state to the checkpoint, anytime we invoke again, it'll start from that node rather than from the top, rather than from the start. You can see our graph here. It looks mostly the same. We've added this additional node there. Now we'll run through the same questions. We'll copy over the database again, just so it's starting from the same point. And we've added some additional information here to the database. Let the user actually interact. All right. So despite having that information provided, it's asking for our confirmation. So we'll say, yes, we're going to search for our flight details. You're allowed to do that. Now it's going to be looking at the policy. Sure. You can look up the policy. I think you're getting the sense that. The user being involved in the flow for every single step is maybe a little bit overkill. So am I allowed to change my flight? Yeah, I can. All right. So if you continue running, the agent's going to continue to ask for your permission anytime it needs to do any action from making a reservation all the way to just looking up a simple policy. This is a little bit too much work for the user. We want the agent to be a little more autonomous than this. We just want to make sure that it's not taking actions that make my life as the user worse in the next section. We'll make a conditional interrupt. This is a simple change using a design pattern, just splitting the tool nodes into two so that we can interrupt on the ones that are what we deem sensitive. So right actions, this is one way of, of introducing conditionality on interrupts and user in the loop workflows. So once again, our user will ask a question. You have the assistant, if it's able to choose to respond directly, or it can call a tool. In the assistant's mind, it's still seeing all these same, the same tools here. It doesn't know that it's split here. That, that really doesn't matter. It's the LLM. But whenever it does call a tool, we'll have this conditional node that looks it up and sees is the tool that's called one of these sensitive tools, or is it a safe tool that it can go and just do search or retrieval? If it's a safe tool, it can go there and then return, and then the user can, or the assistant can respond directly. The user doesn't have to be confirming in this workflow. Otherwise, if it's going to make a change, like a cancellation or booking a reservation, it goes to these sensitive tools. Here we have the route, the interrupt again. So the user can approve or it can decide, no, I don't actually want that. I'm going to want something else. Or the user can even update the state directly to change the parameters specifically before they go into the tool. So it really has like three different choices. Once the tool is called or once the user denies it, the assistant can then decide whether to respond saying this question is satisfied or it can decide to try again, get some more information, et cetera. All right. So. The state here and everything is identical to part two. We have the messages that we've defined. We have the user info so that eagerly populating the information about the user. We have the assistant runnable. This is all unchanged. We're going to use like sonnet. We're going to have all these things. The one thing that is different is now we split this tools into two lists. We've had the safe tools, which is the general search engine. Fetching user information, searching, looking up all these read only operations that we consider to be fine, doesn't require user approval. We've then split out these updates and cancellations and creation events into sensitive tools, which will organize that really. As I mentioned before, the LLM actually doesn't know the difference. We're going to have all these tools provided to it. It's still picking the same as before. What's different is that our graphs, our orchestration, actually will. be able to, to federate this out into different nodes so that the user experience is slightly different. All right, time to define the graph. This is again, mostly the same as in part two. We have the state, we're passing it the state we've created. We have this user info node that we're going to enter on and say every time we start this graph, we're going to make sure that we have the most up to date user information available so the LLM can see the user's current flights and etc. This is useful, again, as an aside. If the user goes and makes changes outside of this assistant workflow, it has up to date context. It doesn't have to be getting that all from the conversation. Now we're going to add the assistant, and we're going to add two different nodes, safe nodes and sensitive nodes. So the safe tools here are the ones that we do for read only operations. The sensitive tools here are the ones that have write operations. Now we'll have the edge for the FetchUserInfo to Assistant. This is just saying every time you enter and we get that eager operation of the UserInfo, we'll immediately go to the Assistant, and then we'll define this conditional edge. Previously, we had just been using this pre built tools condition, which looks for the presence of tool invocations in the AI output in the messages list. Now we're going to extend this and say, okay, yeah, if there's no tool calls, we'll still return just as before. However, if We're going to filter on the name of the tool calls and say, if it's in the sensitive tool names, we'll route this to the sensitive tools node. Otherwise, we can give it to the safe tools node and continue as before. Finally, we're going to compile with a checkpointer again so that it has conversational memory and persistence. And again, this checkpointer is saving not only the conversational memory, the messages, it is also caching this user info that we've defined in our state here. And again, the more fields you add in here, all of this is checkpointed alongside it. So we'll create this. You'll see our graph has added a node here. You have the assistant. It's able to route either to the sensitive tools, to the safe tools, or back to the user. Remember, we're interrupting here only on the sensitive tools node. All right, let's go through this conversation again. Hi there, what time is my flight? You see it fetches the user's flight information, and it looks up policy all without me as the user having to get involved. Great. It's autonomously able to execute stuff. Let's make sure that it is actually going to stop for us before it goes and makes some sort of change. Alright, it's trying to cancel my ticket first. Notice that this time it is trying to cancel before it is updating. This is different than before, just because the LLM is a little bit random. Before it was just directly updating it. This leads us to a point of inadequacy of this simple flat design and that it's pretty hard to have the large prompt that completely defines each intended user journey. The LLM is likely to ignore some instructions or confuse instructions from different sections that might not apply across each experience. All right, in this case, I'll just approve and go through. All right, we've made it through the conversation. Let's jump over to Langsmith to check out the trace. We'll inspect this final one. You can see again, we're inputting null. It's just fetching all of the loaded messages from memory, from the check pointer that we've created. And it's calling this sensitive tools or booking excursion in this case. We're inputting null here because we're approving the current plan. All right. As you see before, the LLM has all these different tools to choose from. It goes in it, it does a decent job of answering the user's initial questions. This time it actually chooses to try to first cancel and then rebook afterwards rather than rebooking directly. This is kind of purely LLM. So we might want to have a more consistent user experience in this kind of task, since it is a common intent. It is able to successfully rebook and then it was looking up some of this information. It does successfully use the search tools and car rentals tools there. It gets to the, it gets to the search tools for trip recommendations and it has a hard time properly formatting this. We can improve the tool description or improve the prompts here, but it's kind of hard to do because we have already about 16 tools or so that are provided this LLM. And you can see we get some inconsistent experiences overall. So again, this worked pretty well. The user is able to confirm or deny any actions. So in the end, if the LLM does decide to cancel, then rebook, versus rebook directly, it may not be that important as to the outcome. It does lead to a less delightful and less controlled user experience. And whenever you want to be building a great you want to be able to shape the way that the user interacts with all of your products and make sure that you can really create a differentiated experience on each of these common user journeys. So the next section we want to be showing a simple way to create these specialized workflows. You can think of these as skills if you're thinking about it in the sense of like a serial Alexa type app. If you are coming from an agent background, this is in our case, a multi agent workflow. But this is more powerful in that each of these delegated flows doesn't have to be agentic. It can be a strict user checklist that you're going through. It doesn't even have to use an LLM. Really what this is, is a way of offloading all of the decision making and execution and planning faculties from this main assistant onto specific workflows where you as the designer, as the developer can inject the expected behavior into the prompt tools and sub graph structure. An analogy I'll make is that if you're creating a website, You're not going to load the checkout workflow with the search workflow with the, like, other, other types of experiences all on a single page. You're going to have specific pages for these and specific modals and designs around these user journeys. The current LLMs really aren't good enough to be able to separate concerns in this way so that whenever the user does enter a workflow, it's only able to attend only that relevant context and really. Perform reliably across these large number of user queries on all of these common intents by separating out concerns and creating these workflows You can really target Common intents and make sure that those experiences are great You can optimize these independently. You can evaluate them independently. Without worrying about interrupting or degrading the overall user experience. This does create more code complexity. It's more stuff to manage. And so there is a trade off here. I wouldn't start directly with this pattern if I'm just building a simple Q& A app. But if you have found that your LLM is starting to struggle with separating concerns and handling all these different Various user intents. This is a design pattern that I would consider adapting. All right. As always, we'll start. All right. As always, we'll start by defining the state. The state has the same fields as before. It has the messages as the user info. You'll notice we added this dialogue state variable here. And what this is just a list of strings. What this tells the graph is any time the user sends a message to it, Where should we route it? Where should we give control? Before we were always first writing to the assistant and then it would go to the tool nodes and stuff like that. Now the control flow can be delegated to the flight booking assistant. It could be the car rental assistant, the hotel booking assistant, all these different workflows. And so this, this state lets us condition on that and route it to the appropriate place. As before we're defining how the state is going to be updated using a custom function. Where we're saying if it, the, if the update is saying pop, then we're going back to the primary assistant. Otherwise, we're adding the, to the stack here. This is just a neat, a concise way of representing where we currently are in the graph. Next, we're going to define each workflow. For the assistance that power each workflow we'll make a flight booking one, a hotel booking, one car rental excursion, and finally the primary assistant that again routes between these. If you've watched some of our other videos or read some of the docs, you'll notice that this is actually a supervised design pattern where the primary assistant gets to choose whether to respond directly or whether to delegate work to each of the underlying workflows. I claim that this is more powerful than a multi agent pattern because these don't have to be agents. Agents are a specific way to implement each of these individual flows. We'll define this mostly the same as before. Each of these flows will have a prompt. They'll have a list of safe tools and sensitive tools. So separating out the read only from the right axis ones, as well as the, the LLM where you're going to bind it. So you're going to provide the schemas saying these are the types of functions you're allowed to optionally select from. We're also going to add this additional function here, we define as a pedantic model, called completer escalate which is the name I chose. This basically says, hey, if the user decides to actually change their mind, this is how you go and, and pull the rip, this is how you go and pull the rip cord, or change the context back to the primary assistant. It might be that I, as the user, will go and say, hey, I want to book a flight to New York, On Thursday, the primary assistance is great. I'll connect you to the primary booking assistant. I'll say, sir, what are your flights? Maybe what are the prices? Actually on second thought, I need to save money. I'm not going to do this. So the flight booking assistant needs to way to say, okay, we're going to exit this guilt. That's exactly what this is. We'll be reusing this completed escalate task and all of these delegated workflows. So we'll define the hotel booking assistant the same way, car rental, same way, et cetera. You'll notice that since we've separated concerns, we can have different prompts for each of these ones. And then. If you wanted to take this a step further, you could optimize, provide more demonstrations, make this arbitrarily optimized for these very targeted use cases, because it's all available. If we were using the current zero shot pattern, you'd have to overload all these different things in a single prompt, which perhaps will be possible with later LLMs when they get more powerful. But today, it becomes a little bit confusing and hard to maintain, and definitely hard to evaluate, because you're constantly making these different changes for each feature experience, which might be different teams that you're developing with and It's much easier in code to be separating the concerns and optimizing them separately. Finally, we'll make this primary assistant It'll have all of these two whatever delegated assistant functions that it uses in order to indicate Okay, we should route these different things In our case again, we're using LLM to do the routing and You might get better performance using some sort of semantic routing with an embedding classifier or something like that. I think it's easier to demonstrate this supervisor pattern because I don't have to train a classifier. But we'll release some other examples later on how you could apply that here. The primary assistant also has a couple of these lookup tools, because if the user does want just general factual knowledge, it might be faster if the primary assistant actually is able to directly answer and do some queries on their behalf. If you end up with a lot of these tools again, you might want to just separate those concerns out. This is all stuff that can be empirically verified with an evaluation framework like LangSmith . So, as we mentioned before, and I'll go back up to the agent state. So as we mentioned before, and I'll go back up to the agent state, we have this message list that's continuously appended to. This is shared in each of these workflows. So all the user's interactions with the flight's assistant can be shared with the primary, which then can be shared with the hotels. So then all these workflows actually know the entire user journey as it's gone through each of these different experiences. This is pretty powerful. One of the frustrations that I often have with tools like Siri is that you'll get into the Spotify tool and then you'll get into maps and then you'll ask the change, the, the, the music and it'll forget that you're there. It just, it just doesn't share context across it. One problem is that if you're sharing this huge list, the LLM can still get a little confused as to its current role. Even if you do have a very explicit system prompt because of that, we're going to add this. Entry node to each of these workflows. What this does is it adds a tool message saying the assistant is now whatever assistant it is reflect on the above conversation, use your tools to execute the task. Basically a reminder of the information that we've provided in the system prompt. I found that at least for Claude, having that reminder at the end of the conversation is relatively effective in Reminding it who it is and what it's able to do. It makes it more reliable if you only put this in the system prompt and the conversation gets quite long, it can start acting and trying to invoke or hallucinate tools from the other assistants as it's navigating this complex user experience. So to find this utility, they'll essentially create the entry node for each of these workflows. Finally, it's time to start creating the graph as before. We're going to start with this user info node just to eagerly populate all this information. Next, we'll add each of these delegated workflows, starting with the flight booking assistant. As I mentioned before, you start by having this enter operation again. This will create a tool message that says you are now the flight and updates and booking assistant. Use these tools to help satisfy the intent. We'll add the actual assistant note here. In this case, update flight. We'll add the sensitive and safe tools and we'll add Add the conditional edge. So this itself is an agentic loop saying it can go to one of the tool nodes and leave the skill, or it can respond directly to the user. Even once this workflow does complete, we want to make sure that we update the state so that it is tracking where it is. So we have this pop dialogue state thing that adds a tool message saying, okay, this workflow is now complete, sort of packages it up. And then it also adds this pop operation to the state. Recall when we defined this state before. Pop here is just going to return everything except for the last list here. So if we've started with assistant and we've gone to update flight, it'll then remove the update flight and it's just assistant in the list, for instance. We'll define other, the other delegated workflows in the same way. So car rentals, hotel bookings, excursions, et cetera. And finally, the primary one. A lot of this logic is repeat. You'll see the primary assistant has a more complex switch because each of these tool calls will route to one of the delegated workflows here. You can probably factor this out into some sort of dictionary and make it a little bit more concise. It also can directly respond to the user. It can also actually use its own search tools, if you remember. Finally, we're going to add this check pointer. And we're going to say interrupt before any of these sensitive tools. If you render the graph, it looks something like this. I'm going to try to zoom in a little bit. So you can see it goes to the Fetch User one. Then it goes to the Primary Assistant. The Primary Assistant optionally routes to any of these. Entrance nodes to these workflows, or it can route directly to the user. Once you're in one of these workflows, it can go to one of the tool nodes. It can leave. So go back to the primary system, or it can actually interact directly with the user. And this is mirrored across each of these specific workflows again. So that was a lot. Let's see how it performs. All right. So let's jump over to Langsmith to look at how this new version did. And before I get too far, remember I did have separate prompts for each of them. I didn't really optimize it super well. I think there's a lot of improvements and a lot of gains that are left on the table right now. Just because I hadn't, didn't have time to optimize them. If you're building something real and you've already separated it out, I think you have a lot of room for improvement by just improving the prompts, improving the tool definitions, and then really focus on each individual module that you can then help climb within an evaluation tool like Linksmith. All right, we'll just look at the trace here just to see how it did. You can see the user's going through, it says, what time is my flight? Seems like it already has the information. So it's searching the flights just to get all the additional information, get some stuff back. And it says, okay, we've got all this. It seems like it's doing a decent job of entering these first questions that we've had. Great. I think that perhaps speaks to the reduced burden of choice since the top level primary assistant has fewer decisions it needs to make. Okay. Going into the flight booking flow. The user tries to update to one for later today. It actually tries and the tool itself enforces that we can't actually reschedule this. It's too soon to now. Remember that policy we talked about earlier. What we could do here, since we're already within this flow, we can just front load a lot of those policies, those relevant policies, so that the assistant doesn't have to go through this process. background take extra work before responding to the user, it can be more proactive about that. And that's what a lot of this separation can end up doing that I haven't yet integrated here. So similar to how we put that preemptive user info block before entering the graph as a whole, within each individual workflow, we can have some of these relevant context information, either in the prompt itself or in a retriever step or other sorts of explicitly designed steps, just so that the assistant can. would be as helpful as possible within the explicit context of the user providing. A lot of what this does then is it takes the burden off the user saying this is who I am, this is what I want, etc. The Assistant already knows because it's demonstrates its initial attent Intent? Users navigated to a specific web page or something like that and then it's able to fetch all that relevant context and bring it to the table. Okay, so the, the, the user then goes through, eventually it lists out the other the other possible flights and then updates it successfully. Great. Let's move on to the hotel booking one. Okay. So once it does book it, it calls this completed escalate. It goes back to the primary agent. Remember in this completed escalate one, escalate step. It is a tool message that's injected saying resuming dialogue with a host assistant, the primary assistant here. Please reflect on that. So the user asks about lodging and transportation. Great, so now it routes to the hotel booking assistant. This is another specialized workflow. In the enter step, we've added this tool message. It says the assistant is here, reflects, continue to help, etc. This just cues in this assistant again as to what it's supposed to be seeing. The AI then says this, it goes through, tries to help. Seems like it does a pretty good job. It's pretty good. Focused here. It sees that there's no explicit mid scale hotels, but there's an upper mid skills here, here, and it's able to be doing these queries in the tool goes through this flow. We get to the car options here. It looks like there's actually a little bit of a hiccup in our design that we could improve. Again, since it's all modular, easier to make these improvements. Once we do see them, it seems in this case, the assistant didn't actually execute or finish once it didn't make the booking, we could actually do hard coded this in so that we never run into this problem. So as soon as the booking is made, automatically route back to the primary assistant, and then it's all up to that to decide who to route to next. Haven't put that in right now. Ask about inspirations. It goes through, it doesn't get anything for location based services. Again, we can improve the prompt here if you want. It's all very scoped and then it, you know, it's second search. It gets it a little bit better and then you get all this information from the tools. Again, it's much more reliable than it was whenever it was the full primary assistant trying to interact with all these tools. It's going through, it goes through, it makes some reservations. It succeeded in pretty much all the tasks of this conversation. Alright, so where does this leave us? We've built a travel assistant, starting from a simple zero shot agent with tools that can engage in chat and have memory and all those nice things. We then made the observation that actually this agent shouldn't be fully autonomous. The user should have a say as to whether it wants to take the actions that impact things. So then we added interrupt. The user could go and Make a change, accept or deny, do all these types of things before the assistant makes any actions. We made the observation that actually we don't need these interrupts on all the actions the agent takes, only on a set of sensitive ones that will change the data that the user cares about. So then we designed our graph with two different nodes, one with the sensitive tools, once with the safe tools. We only interrupt on the sensitive tools, so we have the user the right to change those calls. After that we made the observation that actually the assistant is kind of unreliable in routing these things and using the tools. And so, if you're going to be a user coming to it, or if you have thousands of users coming to your assistant every day, probably a large number of them are going to get results that don't make sense. Or if you're the same user doing it a couple of times, you'll have very inconsistent experiences. And I think all of this leads to a degraded experience as a whole. You also, the assistant also has to be relearning all these things anytime it's invoked, because it's not able to be gaining and optimizing for these explicit workflows separately. So after that, when we split all these concerns up into a separate, multi agent stage where you have the primary assistant delegate to different workflows that you can create centered around explicit known user journeys, booking flights, booking cars, hotels, excursions. Then the top layer, you can just route based on the user intent. In this case, we used an LLM. You can use a simpler, cheaper method like an embedding model as well, just to intent classify and route appropriately. And then each workflow you can design and optimize in isolation. You can say once the user has demonstrated they want to commit to buying a hotel, or buying a flight, or renting a hotel What are the types of things we need to add to the prompt or the tools we need to do? How can we optimize these to make this experience as good as possible? And then once you're in these, as long as you have the escape hatch for it to go back to the primary assistant, shift over to the other assistants, the user has a really consistent experience within that, but while still allowing that flexibility that these LLMs offer by sharing context along the entire user journey, even if they do go through multiple flows. I think this gives you something somewhat of the best of both worlds, where you can take advantage of the long context windows and great reasoning ability, in quotes, of LLMs by, by letting it go and manage that conversational state rather than explicitly tracking it too hard while still being able to put a lot of your dark knowledge and design and intent around your user journeys and your product to good use through well defined flows. It also gives your user full control on the actions that it cares about. Letting your assistant automate what the user doesn't need to do. You can prime it with context. So the user doesn't have to be inputting and staring at a blank blank box, wondering what kinds of questions it can ask once you're in specific flows, it can then save. These are the types of things it's designed to, to, to work on, et cetera. So overall, I think this is a set of design patterns, a set of trade offs you can make that can overall give you better control over the experience of, of how a user interacts with your product and help eventually build much more reliable chatbots that can assist both in Q and a tasks, but also in accomplishing actions such as bookings checkouts, et cetera. So, that's all we have for today. If you have any questions or comments, leave them in the comments below. Just as a reminder, I'm Will. We're going to be posting a lot more of this content about LangSmith, about LangSmith, and all of the things that we'll be developing. So, if you have any requests on what you'd want to see, just leave them in the comments. And see you later.