[Music] all right welcome everyone how's it going welcome to the first episode of our podcast where we explore the minds of the people who are shaping the future of artificial intelligence technology and Beyond I am your host and today we have the privilege of sitting down with a guest who is at the Forefront of integrating large language models like gp4 mestral Gemini into our everyday lives his name is Harrison Chase if you haven't heard about him he is the Visionary co-founder and CEO of Lang chain now Lang chain is probably the most popular framework that allows software engineers and developers to integrate large language models into the new generation of applications these applications range from assistants that can chat with you over your private data like how we have shown here in the channel and to all the way to autonomous agents that can reason and take action by themselves depending on your instructions and much more today we will dive deep into Harrison SP for the future of AI the challenges and triumphs of building Lang chain and his insights on the evolving landscape of large language models and generative AI so without any further Ado I bring you haris and [Music] [Music] Chase all right there we go great well Harrison thank you for thank you for being here uh it's an honor thanks for having me great it's an honor thank you very much for for coming and um yeah I'll be asking you some some questions about on Lang chain and about your your career and uh your Insight about the entire AI industry and what you see coming the future so um very happy great so well let's start off with um having you describe uh what you do in a few words I mean Lang chain and in general what you have have been uh working on it's been more than a year already that you have been on this so um why don't you start with that yeah it's been a little over a year so Lang chain started as an open source project to make it easy to build llm applications um so uh yeah launched in late October early November of 2022 um as a side project just a python package uh that yeah had a bunch of like Integrations with various components that are common for building Alama applications and then a few specific ways to construct these into what we called chains um hence the name kind of Lang chain language chain um and since then uh it's turned into a company so we started a company around it in uh late January of 2023 so a little over a year ago and now we have python package which is a big Focus we have a JavaScript package which is also a big focus and then we also have laying Smith which is a platform um for logging observability testing eval of uh llm applications in general um so yeah at a high level that's uh that that's what Lang chain is as the company and then I'm sure we'll dive into all the components in more detail yeah I mean it's amazing that um you have managed to go from one package to this entire ecosystem of uh tools that are capable of building this very cool applications I mean I think it speaks to the early nature of like it's just so early in the space right so like I I start I didn't start lingin with the intent of starting a but when I when I um you know after playing around in the space more and after building with it more it became pretty quickly apparent that one like you know llms are going to be a big deal um and and change a lot of things and two it's just like so early on in the space and so because of that there's just so many tools and helpful things that need to be built that are you know Lang train the orch rtion layer that is just one part of what's needed to build llm apps and then the second big part which we saw was basically these apps were you know really hard to get reliable they have these like random non-deterministic things at the center of them and so we recognize that pretty early on built Lan SP pretty early on and you know I'm sure there's going to be more tooling that's needed but yeah I think it just speaks to the fact that it's super early on and so there's a lot of ways to provide value a lot of uh new tools coming on in on the Ling side um I mean you know I think definitely a lot of like changes and improvements to Lane chain and Link Smith like Lan chain right now is very different than L chain when it first launched right it has to be um and so I we'll continue to push it we'll continue to focus on specific areas that we see being kind of like focuses and add more tooling around that for sure and then On LAN Smith there's uh uh yeah there's a lot more features we we are going to add there we we want to add there um again it's just so early that you know observability and testing is a great start but there's just a lot more to be built in this area yeah um about that like Lang chain how how did the idea come to you did I mean was it some some sort of a realization like one day you were like I wish this existed or was it more gradual uh that you started building it how did it go yeah it wasn't um basically I was going to a bunch of hackathons um and meetups in the space talking to folks I knew I was going to leave my previous job but I didn't know what I was going to do um and so I was just exploring um and as part of that exploration you know talk to a bunch of folks who were building early applications and saw them doing similar things that Ling chain would would end up helping enable um and so I think the thought process thought process wasn't really too grandiose it was like hey there's some common abstractions in terms of how people are doing things wouldn't it be cool as a as a weak side project to pull these out into a package and put it out in the world and that's really how it started right yeah I mean it's it's definitely become I mean since then it's become a very mature product I would say um more mature at least I constantly only getting more and more mature I mean after over a year how would you say that your vision of uh Lang chain has changed because I mean I suppose that it's not the same as it was like a few months after you founded a company yeah vision of the open source or vision of the company I mean in general um in general yeah yeah you know I think there are a few like specific areas that we have more confidence in as being like high value like I think like right after we started the company the main thing that we knew is that it was really early on in the space um and so there was a lot of hypotheses we had um and and I think now we know a little bit more about what it means to be able to build reliable app in the space so like on the L train side you know we've been really focused on kind of like structured output so so uh making it really easy to use all LS to Output structur data we see this being a huge use case um especially where Lang chain plays which is like using language models to call other things and do other things you need to Output it in some structured format so that's been a huge focus of ours um we've we've uh added a bunch of stuff around streaming um so we think streaming not just of but of the inter immediate results as well is really important as you have longer running applications so like that's you know something we've updated our belief is is really important and um uh retrieval is really important um I think if you asked me that last year I'd probably say that as well some updated believe you know now now uh one aspect of that that we're really focused on is the query analysis part um so we think like basically taking a raw user query and then converting it into a more like optimized query in some way um we think big piece and likely to be here for a while so we're really focused on that right right can you give an example of what that what that means a little bit more in detail yeah yeah absolutely so I think maybe um the easiest example to understand is so a lot of rag applications take a user query um and then they basically the the you know the typical method is to look it look up against documents that are in a vector store so you have a numerical representation for each document you create a numerical representation for the user query you find the ones that are most similar right uh but often times queries can in and backing up like that embedding similarities really good for kind of like comparing the semantic uh nature of two documents against each other um but oftentimes queries contain information that is not really semantic but rather more precise so an example could be um like you know what are movies about aliens in the Year 1980 so 1980 isn't like a semantic bit you want like you know you're asking for a specific filter and so like a query analysis step would basically take in that query and then spit out uh assuming that in our database we're able to filter on year of of publication for a movie it would spit out an optimized sarey where it would have like a query search string which maybe be like movies about aliens or if you have it's a movie database maybe it's just aliens um and then it has another filter that's uh uh you know release date equals 2010 and then you can pass that optimized query into the into the you know Vector store filtering on the release date and and get back results that are guaranteed to be that release date 201 right okay that's pretty cool at at some point in the I mean I had seen some interviews with you before and you were talking about some of the implementations of um Lang chain some really cool projects that you had seen in the past I heard you mention one say Dungeons and Dragons um master or something yeah um in I mean it's been a while since Lang's been out has your I mean have you encountered new or cool projects that have used langing to create their application or something yeah I mean I think there's a lot of cool projects and cool and different ways um so so I I I think the Dungeons and Dragons one is still one of my favorite for a few reasons like I think like one like uh you know is a creative application right and so I think creative applications are really good for llms because they're not perfect and they can mess up and in Creative applications that's totally okay and it can even sometimes be like a feature um but but I also think there's a lot of um there's a there's a lot of really tricky technical challenges to build a really good kind of like Dungeons and dragon Str Master mainly you need to kind of like maintain like the state of the world and keep track of that over time um and so I think that gets to like the idea of like long-term memory which is something we're really excited about and so so I think that's really interesting um you know I I think another example of kind of like cool applications for a different reason um are some of uh kind of like these are they're kind of like more complex one example would be a more complex kind of like customer support chatbot and that sounds really boring on face value right um and and and much less exciting than Dungeons and Dragons but but I think the reason why it's cool is that you know it's really easy to build a chat bot that's kind of like answering questions about documentation and so if that's what you have as a support bot right that's that's a little bit less exciting but oftentimes um you want your customer support bot to be able to do a multitude of different things it needs to know about the user that it's interacting with and it needs to like only be able to you know if you're if you're asking about like previous transactions you should only have access to data that to your transactions you shouldn't be able to ask questions about my transactions so there's so it's a bit more narrowly scoped in that way and and the way that we see people building these more complex Bots is often as kind of like more complex graphs or state machines kind of um so there's a lot of really uh cool kind of like flow engineering work that goes into creating these basically like State machines where you have a customer support bot that you know maybe you you you enter um and you're talking to like a generalist customer support bot but then you ask a question about a purchase order and it routes you to different one that knows how to deal with kind of like your previous order and then you ask a question about the current inventory and it routes you to a separate run right so you have these multiactor systems with some like flow set up between them and so there's some really complex engineering that goes in there and so well may not be as cool or sexy as a Dungeons and dragon thing um it uh I think it's still a pretty cool example of uh some complex engineering Plus llms have you seen some of those in production already yeah yeah cool uh they're using Lang chain some are yeah you know unfortunately not are but some definitely are yeah cuz I I saw like um on the previous information that you shared in previous interviews that most of these projects had not reached production level yet um so that has changed apparently since uh last year well so you know most uh so okay maybe a few things like uh I would say most still probably haven't reached production level but there definitely are some that are and so we're starting to see like what it takes to get successful and what success looks like um and then more generally yeah we're seeing we're seeing more and more of that recently so my my take on the both space is like you know chat gbd came out like November 2022 right um and so maybe from then until kind of like the summer there's like a massive kind of like rush in people exploring and trying out new things and tons of interests and people like trying all these wacky agent multi-agent type things um and and then around last summer I think people um you know became more focused on like okay so can any of these actually go to production like what are we putting that's in production and you know there were a few right away but those were more like simple single llm calls um and and so yeah I think from the summer so so I'd say like from the summer to maybe like o October November I would say uh uh yeah maybe like from the summer to November the space I think was a little bit quieter as people were trying to like figure this out and then roughly from like November till now I think there's been more and more companies that are putting things in in production so we did a case study with elastic um who launched kind of like a a chatbot answer questions like an an assistant style chatbot um and that's using link chain and I think that launched I forget the exact date but I think that was maybe like December January um right so we see a lot of these applications starting to come to fruition and and finally yeah finally that's awesome um and you mentioned that you're we're starting to see what kind of things um are required for this kind of applications to actually go to production what would you say that are the main challenges for this applications to actually make it to production level yeah great question um I think there's a few one is like what is the right like ux for these applications um so you know chat and I mean that in a few ways one like chat is a fine ux um and so a lot use that but a lot are experimenting with kind of like other uxs as well I'd say most have gone to production have still been chat um but people have been experimenting and trying to figure it out but even within chat um you know interacting with llms is different than interacting with traditional software or with humans and so people kind of like need to figure out how to interact with these things um and so you know there's there's we did we did an article uh with uh folks from Norwegian Cruise Line where they talked about their chatbot and and some of the UI things that they're doing like you can give people example questions they can ask right um so that they know how to use it um you can uh I mean you need to stream tokens so it doesn't seem like it's waiting forever kind of like basic stuff um but yeah there's there's uh some figuring out of that um and and then uh I think probably the biggest blocker is these applications like you can build a prototype really quickly but it's not kind of like at the level of accuracy where it's where that's good enough to send a production and so there's actually a lot of work that goes in to to tightening um that Gap and and improving the application um and so a lot of that is basically prompt engineering and flow engineering making sure you have the right context making sure the prompts are keep on improving maybe making sure you have the right few shot examples and and this is where Lang Smith um building comes in real Handy you can see exactly what's going on you can see the input and output at each step um uh you can you can enter into a playground play around with the prompt mess around with it from there um and and so you can you can create test sets which then allows you to like iterate with confidence as you kind of like uh change a prompt you can see whether it's improving or depring or sorry not improving um and yeah this type of like visibility um I think really provides you know it's still a lot of work don't get me wrong like it's not and that's why it's taking you know that's why it takes a while and but we see the best engineering teams kind of like get more rigorous about how they're testing get more rigorous about how they're debugging issues um and uh launch into production get feedback improve from there have this like tight feedback loop and and that that gets them to a point where they can say launch into production right um for those who are not familiar with it would you mind just giving a um a quick explanation of what Langs Smith is and how it works yeah absolutely so so Lang Smith is a platform that we're building it's independent of Lang chain so it works with or without L chain all right um uh one yeah a lot of people don't know that um but it's yeah it's completely API first so we we have a nice kind of like Python and typescript um sdks for it that uh you we have some methods for like you know patching the open AI client really easily so if you're using open AI directly and not using linkchain you can just use that or you can just log things directly um so basically what it is it's a loging platform you log you log kind of like runs and results there so runs from production you log there um runs from kind of like evaluation runs uh you log there and then you can easily inspect the results and so you can see exactly what's going on you can see the inputs and outputs at each step um you can track kind of like uh you can collect feedback there you can track feedback over time uh you can you can monitor kind of like the latency of your applications you can check it for errors um we also have uh other things like a prompt Hub where you can store inversion prompts um and then also an annotation Quee um where you can easily label some of this data that's coming in so we say that human label human annotation is is pretty needed useful for collect you a lot of the feedback on these data points right sounds like that's kind of a most for uh production application are you the the I mean I I really don't know is this the only application that does this in the market or are there on any other because I I don't really know about that there there so there are a lot of other similar ones I'd say that ours is the most complete in the sense that it does all these things that I just mentioned um I think you have other tools that are really focused on just kind of like the logging of single data points or just the monitoring of of like latency or just the testing of uh data sets um and and I think ours is unique and that it brings all these things together um or our this unique in two ways i' say one it brings all these things together and I think that's really important because I think this like flywheel between logging testing experimenting like all of that is super important to to get really tight and crisp um and then two ours is really focused on like multi-step applications so a lot of other um ones are focused on like like single llm calls right um but you know we're L chain so you know we need to build this to work with L chain um and L chain is a lot of like multi-step applications um and so I think we have by far the best support for that just given the nature of kind of like our Focus right okay that's uh yeah it sounds pretty much like it's a something that you would need for for production definitely I think it's a I think it's a very very good product it's a probably a better product than linkchain um it came we launched it it was in public data for a while but it's now generally available so everyone everyone just sign up and try it cool and um are do you have any other any other products that are in the works right now or is that your main focus for the moment that's the main focus for the moment right um okay continuing about the about the Lang chain ecosystem you have an immense number of um Integrations with super cool providers which one would you say that has uh brought the most value to the Lang chain ecosystem or which which Integrations like you can name a few that are the most interesting to use um I mean this okay a little bit self- serving but I think the one that's brought the most value is the laying Smith integration um but let's I I'll put that aside we already talked about about that um I mean I think the most you know the main new thing here are language models right so like all the Integrations with language models are probably the most important things there and we've invested we've invested a lot of work in our integration ecosystem so Eric on our team has done a fantastic job of you know we now have like integration packages with specific providers so they're like more reliable and more maintained um and so we've invested a lot of work in this particularly around the language models um and so you know probably the most used one is the open AI integration for sure um and so by that measure like you know that that we we actually do have a list of the most uh kind of like downloaded integration packages um openi is number one by far um but I think you know like uh any of the language M providers anthropic Google Nvidia Kier Azure like all of those have got to be up there um some of the more cool ones I think are the local model integration so like AMA is a a thing that can run models locally our integration with them I think is really popular and it's also really cool because it lets you run stuff locally and that's awesome right so I'd shout out them as being one of the more cool ones that people should check out cool and I mean there language models when you're within the langu framework um the idea is to make them interact with other providers not only for language models but for data store for stores for other tools which ones would you say that are the tools that have had the most success within the echosystem like which ones have attracted more people yeah so uh number one by far has got to be all kind of like the vector store tools um so one of the main things we see people building is chat over their documents right and so to do that Vector stores have emerged as the or or or databases that support Vector uh Vector Technology have emerged as kind of like the main way to do that um and so pine cone weeva troma redis like all of these whether they're Vector native or have added in Vector functionality um you know uh uh those have got to be the main class of tools that are used I'll I'll shout out a second one which is also search related tools um so we see applications that can interact with the web being really popular and so uh tavil AI is a great integration partner um and they have a great kind of like a web action same with EXA same with u.com and so those types of tools have have also been really popular um I I'll also say though that for um you know when we see people going into production often times the tools that they use are custom tools that they write um right because you know we have a bunch of generic tools like Google Maps and stuff like that but oftentimes we see people building custom applications with their data and so we've made it really easy to write your own custom tools and so that's often times that's by far what we see people using most in production with with the exception of like these Vector databases because those are there are a few people that maybe right there on Vector databases but for the most part people use ones that are yeah shelves yeah so you mentioned that apart from rag uh from yeah apart from rag you have web like searching the web applications do you see any other kind of applications where Lang chain um is useful for or are those the main two ones um I mean like I think the main so so I'd say the two main things that people build are and they you can think of them in the same way if if you really want to but one's kind of like rad style chop Bots and then the other kind of like assistants that can interact with various things um those things can be apis to search the web um right or to search their own data but there there are a long tale of other tools um but there it's it's like a long taale like a few people use this tool a few people use this other tool um but the the things that they're building are these kind of like assistant co-pilot like things where you say something then thinks about what to do do it takes some action it gets the response back it thinks about what to do again it gets the response back and then it returns to the user and so that that type of architecture um I think we've spent a lot of time making easy to build with Lang chain and Lang graph um and so you know the that that would be the main thing that people are building but there's so many different use cases for it like you can use it to do SQL question answering you can use it to do CSV question answering you can use it to do question answering over your documents you can use it to interact with an API um and then you know interacting with an API is incredibly horizontal and so there's all different types of apis so you know it's it's tough to but but and it's tough to come up with something more specific than that because people are using LMS for everything so I mean just just to to be sure that everyone understands what we're talking about here when you when we say rag would you mind explaining what rag is in simple terms Place yeah absolutely so rag stands for retrieval augmented generation the basic idea is that um when when you're chatting with an llm and you send a message in it responds based on stuff that it knows it's been trained on that's in its weights um and so that's great if it's been trained on data that's relevant it's not great if you're asking questions about data that model has not been trained on and so the most common way to overcome this issue is to use rag retrieval augmented generation which is where you first do a retrieval step and you fetch relevant documents um or relevant data I should say um and then basically you augment the generation of the llm which basically means you pass in these documents as part of the prompt to the llm and you say something like answer this question based only on the data below right so that allows you to chat with your private data and pretty much any data available yeah exactly and I think most people immediately think of chatting with like unstructured data but you could view SQL as a chatting with a SQL database as a type of rag um you could you know then then there's also uh like when I was talking about the assistance earlier you could also view the assistance as a type of rag and and you know I would say that's not the most common way that people do it there's and this gets to um you know probably like the main point of laying chain is that there's all these different architectures that your applications can take on they can take on like I'd say like assistants are specific type of architecture there's also another like the most common architecture is a much more simpler one for rag the most common rag architecture is a much more simpler one where you you don't have this assistant thing you just like look up data pass it to a prompt that's your final answer right and so that's really fast that's really simple but it doesn't handle like it doesn't handle edge cases well and things like that and so then basically people are people are experimenting with all these different architectures and that's what L chain and Lan graph make it easy to do they make it easy to build all these different architectures and and and use examples of these different architectures awesome all right at some point also you mentioned um that you had seen some applications that I mean the most oftenly used interface for these applications was chat what other interfaces do you see or have you seen either in production or in prototyping yeah I mean the the other most common one besides chat is just like there's a single text input and then you get back a response um so like a single turn chat or something like that but then you can like you can populate it with um any information um then there's also some that just like they they weave it into their application UI so they're not actually maybe changing their UI they're just surfacing things so they could be using it to like generate uh uh like summaries of a document and they show those generated summaries in the UI but like they were already showing the document anyway so it's just adding like another bar under it um and other than that like uh yeah chat's by far the most common right I wish there was more I wish there was more um good ones Rell recently announced like uh or they they shared something around like generative UI where you use an llm to generate the UI um but I still think that's pretty new I haven't seen that like actually in production um very cool but everything that most of the things we see in production are the chat right um I would like to know a little bit more about as well about your career and what you what you have done because I I wasn't able to find much information about that um mostly most most of the information online about you is about Lang chain but I would like to know how you got to to where you are today and um and yeah I mean some advice that you may have to people that would like to to go into the same field as you and yeah so if you would mind just sharing a little bit about your academic history or professional history uh what you have been doing before Lang chain yeah absolutely I studied stats and computer science in college um and really started actually doing stuff in uh because of sports analytics so I really liked Sports and I liked math and stats and so I started doing stuff with sports analytics that led to like uh learning uh how to code to do that and things like that um so uh studied stats in cassin college did a bunch of sports analytics um after graduating did not do Sports analytics but instead went to Keno um a fintech startup um was on the machine learning team there for about two and a half years um did some stuff in time series then uh led the entity linking team so some some NLP related stuff um around entity linking uh we were that was like a 60 person startup when I joined we then got acquired by S&P Global after about a year um stayed there for like another year and a half S&P Global very big company um and so you know after after that I knew I wanted to go someplace smaller so joined robust intelligence um as the second employee there my current co-founder anush was actually the third uh employee there so that's how we met each other um and yeah it was there for about three and a half years um by the time I left we were about six people and it's it was a company in the mlop space so basically we worked on like testing and validation of machine learning models um it was really cool the first year you know uh super early on we were interacting with a lot of like R&D Departments of of other companies we spent a lot of time doing like adversarial machine learning um so some some really cool stuff there um then we started building out a platform um so got to got to see that um that that was really fun in in the mlop space on on testing and validation of machine learning models um and yeah I don't know like you know I think I've always just done things that I've I think I've always done things that I enjoyed right like I got into this because Sports analytics and I really enjoyed that you still do some of that uh not really like I follow it Loosely for for a few years after college I did some stuff on the side um but not really anymore um and so I I don't know I'd say like uh do stuff that you like and then you'll you'll work hard at it more naturally and then if if you work hard at something for enough time you'll you'll get pretty good at it then you can you can get good enough to be dangerous and then you can start ACC company in the space or something like that so you've you've always been in in the machine on the machine learning side of uh of these products right yep yep you learning and ml op so more so on like the applied side of that I was I was never I was never like a good researcher or anything like right and would you say that your degree prepared you more for for that kind of job or more for the development software engineering side um so I think I think my degree was very helpful for a lot like the machine learning stuff um but I learned everything I know about software engineering at my first job like you know i' obviously coded before that but like never seriously and so um yeah learn learned everything um for Real about that um out of college right and I had one question here about um becoming an AI engineer like I mean this these new terms like they're since they're new they can mean so many different things depending on the company that's posting the job ad so so how would you define a job engineer and sorry an AI engineer and what they do yeah um I definitely don't have a very good description of that you know I think it's probably been like under a year since people started using that term I I would say like um it probably involves working with language models and integrating them into uh in an application um and so I think like you know the require skills are are probably good engineering skills um and as well as like good problem solving skills because I think working with llms is a lot like solving hard problems they it's it's you you like the space is just moving so fast you need to be able to like pick up new things quickly um debug things quickly and so yeah just like solid engineering skills and then good problem solving skills to to stay up to date with all the llm advances would you say that would you say that you have to stud computer science in order to actually participate in this new industry or what do you what is I don't think so I don't think so I think like um and there's also flavor like you know there there like I think there's some debate around like will prompt engineering alone kind of like be be be a job and I think if if there's a job engineer for or if there's a job wreck for prompt engineer like I don't even know if need to be able to code um but I think if we're talking about AI engineer you need to be able to code but you don't need to have like you can you can learn that without a degree right um and then yeah you you you need to figure out how to work with language models I don't want to say like you need to know how to work with language models because no one really knows how to work with language models and it's constantly changing um so I would not get intimidated by that so I think like I think probably in my mind the requirement is more about kind of like being a solid engineer and having good problem solving skills right and about that like how would you like where would you go to learn all of those those things like would you uh I mean is what I I guess what I'm trying to get to is is studying by yourself enough to get into the industry or do you need some sort of a an official diploma or something like that um yeah I mean I think uh so so for the kind of like understanding how to work with language models kind of things um like just play around with language models work with language models put stuff out on Twitter put stuff out in the open source like all a lot of the most interesting stuff with working with like language models is being done out in the open by random people on Twitter um for the engineering side of things um maybe maybe this is a bit of a uh uh an oldfashioned take but like I think you just need to work in the industry and and start start in a you know in in an entry-level position work with some really strong people get feedback constantly kind of like improve I think there is you know I think there's some stuff in engineering that you can only learn by doing and so I think it's just experience there and you know there's nothing there's no there's no kind of like shortcuts there yeah I totally agree with that totally and yeah I mean studying in college can take you somewhere but it's only by doing that you will learn yeah exactly yeah like that's why I said I really didn't know anything about engineering until I started actually like working full-time in in your journey as a like as an engineer um do have you had any mentors or role models that have shaped the way that you work today in like in engineering's part or like in general yeah yeah absolutely I think I think both of my first bosses at uh Keno were amazing and I look up to them a lot um uh Sam schlier is one of them he's now like he I think he's like founding uh member of the character team so they're doing awesome stuff he's I think he uh him him and and Mikey um Mikey scholman who's the CEO of suo Al another company in the geni space but both of them were like you know I I learned so much from them not only like the obviously like skills of engineering but also like mental models of how to think about engineering and how to think about leading engineering teams um and Lead engineering projects and stuff like that I think probably the biggest thing um that uh I think it was probably Sam mostly but but Mikey as well um I think really important it is this idea of like just like figuring out what the blockers are and doing just the blockers um like I think oftentimes you can over complicate things in engineering a lot um and especially in a space like AI engineering today where it's just so fast moving and you need to just like ship stuff get feedback iterate you really need to focus like okay what's actually the blockies here and what are we just like bike shedding is a complete waste of time um and so I think Sam and Mikey were both I I think that's probably like the strongest mental model that I took from them W yeah it's always I mean you always take from from people you've worked with yeah you want to borrow like death parts from everyone right then try to do that so I should work with people who are like better than you at at something and try to learn from them 100% yeah that would yeah that would for anyone like just like any new grads or anyone just getting into this field like 100% my advice would be like find really good people and just work with them and just focus on that and don't worry about uh you know don't worry about title don't worry about um you know with within reasons don't worry about title don't worry about uh comp work worry about working with really good people and learning and that will pay off like few years on the line yeah totally do you have any advice for like people who want like they know how to code they are starting off um with these new technologies they're creating these new products there prototypes and they want to build a company around it um do you have any advice to PE to people I mean for people who are kind of in this situation yeah I mean I I don't um I don't know if I have any good advice like my my story and I could share that um you know like I don't know how typical my story is I I can share it and the lessons I learned there but like any advice I have would be you know with that context in mind um and what I did was basically you know I I I built something that I enjoyed working on um and thought was useful um and then I shared it with the world you know I put it out on Twitter um I iterated on it quickly I I made like a release every day right um and eventually enough people started using it where it was feasible to start a company around it um and so um you know I I was in a fortunate position where I was you know uh still at another job at the time but I knew I was going to leave so I kind of like had had a a safety net but I also had the freedom to um you know before that I was I I worked a lot I worked hard right so I wasn't doing a lot of like side projects so I I was in a you know definitely unfortunate position I don't know how replicable that is but yeah the takeaways would probably be like ship it like right you got to you got to put it out there um iterate quickly like get feedback and iterate L really seek out feedback like I think I was pretty proactive about seeking out feedback and I think I think that helped improve it um and then yeah if if you build something that people want as cliche or you know much easier much easier said than done but if if if you do that I think um I think that helps yeah I I mean to me it's just uh impressive how so many stories start with uh the first tweet came out like on this day um I mean from the top of my head I can't actually remember a single one but um like it just comes so often in conversations that yeah it all started with the Tweet did you have like many followers or how did how did that like actually took take traction I had like 200 followers 250 um they were mostly from the sports analytics days um so it's actually funny when I look at the people that I follow like you know it's it's mostly AI people now but there's like some Rand Sports analytics people from way back in the day um and yeah I think what I you know I think the first thread I did was like 13 tweets or something right so it was pretty long and detailed um and I tagged a bunch of people in it it was you know I I think I'd implemented like three different papers and like seven different Integrations or something like that so I tagged all the authors I tagged the integration things um and yeah I guess that helped get enough people to see it and enough people liked it and and and retweeted it I think that it it got got some followers from that and then kept on follow uping follow following reading about it and yeah yeah but it was just like consistent right like you got to do something every day you got to make progress like I mean it's it's it's still impressive the thing I mean the open source part right because I mean it's not obvious to me how you turn open source into a into a business has it been like very difficult or how how's that been um you know we knew we were going to build Lang Smith from the beginning and I think Lang Smith has turned out to be a correct bet like I think there's definitely a market there um and I think the you know but but it is difficult because it's not Lan chain right it's a separate thing like we basically have two teams internally and we basically twice the number of competitors right because there's some competitors for the open source some competitors for link Smith um but luckily we have a very good team that moves very quickly and I think we're able to ship and and and so I think we have the best product in both um but yeah you really I I remember talking with the data bricks people early on um and and they did a similar thing where they had they had spark and then they had their datab break platform um and the way they kind of um framed it was you had to you had to hit a home run in each right so like by the time we started the company you know would gotten a hit with l TBD like you know what's a home run versus what's a single but we got in a hit and yeah with doing that with Lang Smith we had to do that again and I do think we've done it again um you know obviously I I don't know if it's single or home run but like I think think um but yeah you have to do it with both and they're different skills one's Source one's a platform um different set of competitors different set of considerations uh at least one kind of takes the the users to the other one right yeah yeah oh I mean I I would actually say both um both like lane lane chain takes people to Lang Smith but then Lange Smith also makes L chain a better experience so there absolutely is like a flywheel there um and that was intentional as well we didn't want to do something like completely separate um we wanted to do something where there was a strong flywheel um but it is still different skill sets and different product right and if it were only the open source part uh do you have like um like a monetization side of it or is it like completely I mean I really don't know how how to monetize open source so just enlighten me here yeah I mean I think a lot of traditional open source is kind of you know we spent a lot of time looking at kind of like comparables so so like a lot of Open Source databases for example monetize a hosted version of that database um but link chain is not a database and so it would be unclear what that would be but if we were to do that that would probably look like a hosted Lang chain experience maybe like a low codee no code version or something like that um and we didn't want to do that because well we we haven't done that yet because we think it's still really early on really fast moving and that wasn't like the blocker for people we saw the blocker being the problems that Lang Smith solves um so that's why we didn't do it but if you're saying like Okay you can't build Lang Smith then like yeah we probably would invest more in like a hosted experience of sorts um I think another like reasonable analogy is kind of like nextjs and versel um and you know we're starting to build some of that with like langing serve and hosted langing serve so like really easy way to deploy Lang chain applications uh but again we we we see the Lang Smith being a bit more blocker for folks and at this point we just want to create as much value for people um as opposed to try to capture awesome great um all right I wanted to talk to you a little bit about what you see in the future of the industry and in AI in general how would you see the this future of large language model applications um do you think that it will look a little bit similar to how it's being done today with the rag applications or do you do you see something completely new coming on yeah it's a really good question and any any predictions I make are probably going to be wrong um I think like I think we'll see more and more multimodal applications I think we'll see more and more um like complex applications um so like multistep the language models deciding what to do and then taking that action um and and more and more steps um and and I think we'll see that for a variet of the reasons the models will get better people will figure out how to work with them they'll get cheaper and faster um and yeah so so so I think uh the and then multimodal I think we just see so much excitement about that the models aren't currently there um but when they get there I I do think that'll be big um and so those are two pretty kind of like easy bets um one thing that I'm really interested in is the role of kind of like long-term memory in all these applications like like what does that look like and I think I think I think that will be needed for a lot of applications to get uh like personalized experiences for users and so so what exactly does that look like you know open AI added it into chat GPT and they added it a specific way right um uh we have some memory modules in Lane chain that think about it in a slightly different way um how how is it different though yeah so so open AI um they basically use uh they let the llm decide when it wants to remember things um and then a bunch of memory modules in linkchain kind of like do the remembering automatically and I think there are pros and cons to both um but it's it's like it's kind of like you know active memory versus kind of like background memory um and uh yeah they're are pros and cons to each um very interested by memory I think I think I don't think it's like the thing holding applications back now I think people are still trying to figure out how to get simpler applications to work but if we're talking long-term I think that'll be a big part of it memory Yeah Yeah you mentioned as well multimodel um multimodel models uh can you can you develop a little bit on what that means and how that would be uh like important for the industry yeah so multimodal working with different modalities so like speech text image so like right now uh gp4 um has a vision variant where you can pass in images and then get back a response and so I think this is important because there's all you can like you know previously if I wanted to uh try to build an application that would look at an image and decide what to do I would have to like convert like if it's just text I have to like convert it into some description of it's kind of janky right and so now you like pass in an image and you can be like okay based on this like what action should that's how like we're mostly interested in it in terms of like input is image video audio but then it returns like what an action to take because you know Lang change just interested in like the action taking aspect of it right or that that's a lot of what we're interested in um and yeah so I think be able to pass in an image and ask for the next button to press if you're looking at like a computer screen or something like that or pass in a video and ask for a summary of it or something like that is is really powerful do you see a future where you have like for different language models interacting with each other from different applications like for example today most of the Integrations that we have within different applications I mean all of them are made through apis like one application exposes one part of its logic to the other application and then they connect each other do you see a world where these kind of connections happen with natural with with human language even within applications or does that seem like not realistic in the industry no I I think that's pretty realistic yeah I can totally see that I think I think the main reason that's realistic is it's very valuable to create agents or applications that are more narrowly scoped um right so you you might have uh I mean even back to the customer support bot case that we were talking about earlier where you have one person that's responsible for talking about their their previous orders another for the inventory those can be two separate you know llms chatting with each other yeah that's what I was thinking when you were saying that I was like like what if the person is an llm as well yeah exactly so from that point of view 100% I think that that'll that so you would have something like one person asking their language model assistant hey do this for me and the language model assistant will communicate with the language model assistant from another person exactly yeah I mean I don't know if that's coming soon but that would definitely be very crazy for the for the world um about um AI what what do you see in the open source sphere of language models because I mean we have seen some amazing language models in the open source scene um do you think that I mean how's your what's your position about that and has it changed uh recently with the newer models um I think my position is that they're still not used as much and they're still not as good as the closed Source models um but I think a lot of people are really interested in them I think the interest yeah I mean the you know I I think the interest for a few reasons one they can run locally and so your data doesn't leave the premises two they're often um if if you host them right and there's a few good providers like together Ai and fireworks they can be like really fast especially because they're smaller um right so so I think there's a few different considerations which is like okay like are the open source models like how good are they um like are they going to would I like would I rather use just purely based on how good they are like and open source model or or gbd4 or something like that and right now like if I just want it for how good it is I'm going to be using gbd4 and because it's early in the industry um we see that people are just using gbd4 because they're still trying to figure out what to build and if you can't build it with the best model you're not going to be able to build it with a weaker model but I do think it's possible that it'll start to change as these things go to production and then you carry about cost and latency and all of that and so that maybe more appealing to look towards the smaller model um and you know there's a lot of good capable smaller models out there that you could fine-tune if if if you wanted to if you have this really high production um and so I think that's really interesting to think about and and that's an area we're very interested in exploring well I wanted to ask you a little bit as well about I mean real quick about the security side side of it because I mean what Lang chain is doing I mean at some degree is fac facilitating the creation of autonomous agents and just that word autonomous agent uh might freak out a lot of people um do what what is your your position or your thoughts about this security issues with programs that can reason and act in the real world yeah it it's a good question uh I I have a few thoughts I mean one like I don't think the current systems are that good right now like I think they're still very narrow focused right and so you know they can and they can be very good at there was narrow focused but I don't think it's like it don't think it's it there like I don't think there at a position where they're gonna run off and cause a lot of harm um I do think though that you absolutely need to be careful when you hook up various things to a language model mod and so there are a lot of things in linkchain like we have a we have a python reppel where you can run python code but if you're running L generated python code that could be unsafe and so we have a lot of warnings like hey this is experimental you probably like want to be careful about this um and so yeah I'd say those are my two main thoughts like yes you I think I think we're we're mostly like encouraging people to be really thoughtful about the types of thing like I think I I I think there's like two separate things when you talk about safety there's like okay the models are getting better and better and so like will they allow people to you know will will will will anyone be able to like learn how to like build a bomb or something right like that would be terrible right and so I think there's a lot of like safety work going into like that um and and that's a little separate from linkchain and so I view that as separate then there's like okay like are people putting these things as a reasoning the engine and creating these systems that could be dangerous and I think that really boils down to like are people giving it like um like what types of tools are people giving it access to then and and they letting it operate and so yeah we you know I I think we try to put a lot of warning around dangerous Tools in L chain and and try to bring attention on that note would you support a more strict regulations of this kind of um uh companies that deal with language models given that probably in the future most companies will be Dev I mean most software companies will be developing their own llm applications at this point no I'm generally not really in favor of Regulation um I Al to be honest I also don't spend a ton of time thinking about it um I For Better or Worse that's right yeah because I mean the what one of the things that shocked me uh the most about Sam Sam Alman when he was in his interview with I think it was with Lex fredman that he I mean he has always been openly open about his position that regulation is required and that's why he's doing this and he wants to be regulated and things like that so that's that's kind of that was kind of a new thing for me especially in in in the software industry and Industry in general right but yeah so I think they're not many people uh leading companies uh thinking like that I suppose yeah I I think uh you know he also probably has a much more nuanced and and detailed view on this than than I do and so I'm I'm I'm I don't think I'm incredibly qualified to speak on let's just take care of our applications and try to be responsible with how we use them right all right um do you have um I mean we're coming to the end of the of the conversation do you have like some other new projects that you would like to share or some other new advances in AI that you're excited about that you see coming in the future something I don't know yeah I mean I mean very short term um a few few of the things that we're working on um one on the open source we're putting a lot of effort into like query analysis like I think the query analysis bit can be really helpful when improving rag applications I think it takes advantage of a lot of things that Lang chain is good at like structured outputs and things like that um and so we'll be putting out a lot of content around retrieval but specifically around query analysis um and then on L Smith we we're doing a big push on kind of like online evaluation and like automatic kind of like f shot example data set curation and stuff like that um and uh so that should we should hopefully have some stuff to share there in a month or so um but but those are kind of like the two very short-term things that we're thinking about right that's about Lang chain in particular and in the world in general do you have like some like some new developments that you are like yeah I hope this comes soon uh we'll hopefully be able to use it in Lang chain or something like that yeah I mean all the model advancements are super exciting like Gemini 1.5 with their million context window length is really exciting Straw's new model that they just released um you know I'm sure others will putting out new models as well and so I think uh I think yeah just very excited for better models which seems to be happening yeah about that actually I forgot to ask you about that um about the huge context windows that these models are starting to have do you think that that's going to uh make obsoletes some of the rag applications or how do you see that happening some of them probably yeah yeah um I think it will become less and less important how you chunk things and split things up for example um I think the parts of rag we we spent a lot of time thinking about rag I think there's three main components of rag that I don't think are going anywhere in some form um so one's indexing so this is just like taking documents and putting them into some sort of data structure some sort of index um and I think long condex will probably change what those data structures are right you may not need like as many small chunks you may not need to be as clever about it but you still need to fetch doents from somewhere whether it be like elastic search traditional elastic search or something like that so I think there's an indexing step then there's a query analysis stuff so like figuring out what documents to fetch still think that's going to be needed um I think like when you deal with large col well so so there is a difference between if you're doing rag over like a single document or rag over a large collection of documents and so rag over a single document like yeah most of the stuff will go away rag over lots of documents all this stuff will stay around and so the quer and the stuff I think is is really important there and then there's a reranking step and I think that'll be important but yeah rag over single document you're just going to pass the document into the C response yeah yeah I can see that happening too all right okay well uh I don't know if you have any extra comments that you would like to make some extra message that you would like to pass to the community or some or something you would like to say check out all Andre's YouTube he does awesome stuff uh and yeah more people should be like all hund and building awesome things and teaching people how to thank you man I appreciate it um yeah well thank you thank you very much Harrison for for being here for coming to talk with me and it's been an amazing time having this conversation with you I've really enjoyed it thanks for having me great thank you see [Music] you [Music]