please summarize this transcript: ""I think there's still more folks streaming in but it's uh stabilizing a little bit let's get started um good morning everyone welcome to another exciting installment of llama index webinar uh I'm Simon from llama index and in today's installment we're going to go beyond topk retrieval and explore the latest tree structure indexing and retrieval techniques proposed in Raptor uh raptor is a recent paper published in ICL 2024 and we're super excited to host authors to give us a deep dive into the latest technique so here we have par uh Anna and I believe ADT as well um so please take it away par and hand it over to your co-authors as necessary go ahead sounds good thank you for already can you can you see my screen thanks and if anyone has questions please um put it into the chat and we'll have a section at the end to answer all the questions thank you that's good hi everyone I'm B um here with my fellow authors Salman a sh Nana um this is Raptor the Raptor stands for recursive abstractive processing for Tre organized retrieval super excited to share this with all of you so let's get started so I like the slide from you know the ACL 2023 tutorial on retrieval based language models and applications about all the exciting questions in the realm of retal augmented language models and index has been working on a lot of them too so it's it's interesting to see the three questions that they frame here here on what on um what to retrieve how to use retrieval and when to retrieve so Raptor sort of falls into this um what to retrieve end on the text chunks and the aim of raptor is to retrieve text chunks that give an holistic understanding so if we sort of look at retrieval augmentation in the current retrieval methods um the way it generally works is that you the way you get these text chunks is that you have a retrieval Corpus and then you split it into chunks you can split this smartly using all the fancy chunking techniques or you can just split it at 100 to level you can make sure that your sentences aren't being cut and then once you have these text chunks you use some sort of bird-based encoder to encode it into different Tings and at query time you get a query you encode it using the same Bo Bas encoder and you do cosign similarity and and you have your top K so depending on what your top K is you give the the highest ranked sort of um text chunks to as context to your language model normally you give it in context to the language model so uh we're all very familiar with this sort of retrieval Paradigm but why might be an issue so let's let's sort of look at an example let's consider the question you know what is the central theme of the story and if we look at relevant chunks from this story that we're doing this question on you might you can read the chunks it's like you know two my turned into footman the clock had struck 11 uh they did not know the cindella you can imagine this is from the storytale cindella and you're um you ask the question what is the central theme of the story you get these chunks and you pass it to your language model and you know you get this amazing answer right the theme of the story is centered on jealousy cruelty justice justice prevails in the end um the stories about cindella uh conveying important moral lessons about the consequences of wickedness and the power of kindness and perance right so seems perfectly fine seems like a great answer but there's a there's a Twist so the story that we're sort of dealing with is actually a modified version of Cinderella where in the end she becomes a tyranical queen so you know if you look at the answer here it's it's you know completely wrong it's uh not about how the power of kindness and perseverance sort of um you know wins over wickedness in fact it's more about how uh you know the story the consequences of mistreatment and abuse lead to this cycle of crty and you can sort of look at this Raptor answer on the same on the same story and sort of gives gives this answer and so the question is you know how let's let's in this presentation we'll dive into how how do we sort of get um we'll talk about how we get a tree structure which is able to capture both this higher level and lower level information so Insight over here is that you need some higher level information about your retrieval corpra that can help us answer answer thematic questions do multihop questions and better understand what your retrieval corpor entails and you obviously need a lower level information for you know creating this higher level Corpus but also um to also answer questions that are that you can sort of answer with one chunk you know what is the name of syrell stepm you should be able to answer probably with just a single chunk so this is our Insight that you know we need both higher level and lower level information and let's uh dive deeper into how we get it just a quick aside um you know there there their models that are coming out every day so you know in recent sort of recent weeks we've had you know CLA 3 and Gemini 1.5 and you all these questions around Rag and long context models so you know there there a lot of there are a lot of opinions on this I'll just share like two small points about this you know obviously models sort of there's this paper called Lost in the middle which sort of shows that models struggle to effectively use long range context especially when the information is buried in extensive text so you know even if you have um if you just have around 4,000 tokens depending on where the information is um you you get widely ring accuracy and secondly you know use of long context is expensive and slow right so if I found this amazing uh you know picture on Twitter which shows that you know if you if you pass 1 million tokens on like Gemini Pro 1.5 is around uh you know 50 cents per call whereas if you just use rag you can significantly reduce that cost so it's long context are expensive and slow and you know your they might they might um they might struggle with your uh information if it's buried in extensive in in extensive text and also you know your retrieval Corpus may just be larger than a million a million tokens so let's let's dive deep into how Raptor works so Raptor um is this technique that sort of uses clustering and summarization um to create a raptor tree and we retrieve from this tree at inference time um to provide both higher level and lower level information to your language model so how do we how do we build a tree right so we start by having this tree and we build we build a tree by starting off chunking uh chunking your retrieval corpa normally so you can you can use any sort of an chunking methods if you want in our experiments um we just decided with a very simple chunking strategy of uh you know chunk every 100 tokens but you know not don't split sentences across different different chunks and you know you can raptor is chunk agnostic you can use whatever sort of chunking method you want and once you have the text chunks of your of your normal retrieval corpa we we do we cluster we cluster the text trunks based on a clustering algorithm and once we have these text chunks that are clustered into different clusters we pass it to an llm to summarize it and there then we get the next layer of our text chunks uh with with the relationships uh between the child nodes and we sort of recursively apply this again so we again you know cluster cluster it we embed it and then we cluster it and then we summarize and we keep on building this tree till we're no longer either we get to a root note or we're no longer able to Cluster anymore so if we have let's say three three nodes at the root layer and we can't cluster anymore so we keep on building this tree till till we're not till either we have a root node or we're not no longer able to Cluster and once we sort of have this tree we can query from it so just uh some quick motivations for the clustering algorithm that we're thinking about so we wanted soft clustering in our clustering algorithm so the inside being a chunk that covers multiple topics should appear in summaries for each of the respective topic for example you know let's say I have my retrieval corpora contains text about let's say par is a singer Simon is a singer you know Jerry is a singer and then it has a few senten about how you know part is a violist Anna as a violist and so on so you might want our our nodes to have one note might talk about part about how part is a singer and part is a violist one not talking about violinist about how par is a violist and Anna is a violinist and one not talking about singer how par a singer you know Simon's a singer Jerry's a singer and so on so we we sort of want this soft clustering in our in our system so that we're able to so that our nodes are able to cover um a chunks that have S of multiple Topics in them right so we want so one one when we're sort of our summaries should contain contain all the information and you shouldn't restrict your chunks to just be part of a single parent node secondly we we don't want a number of clusters to be predetermined because it's sort of hard to know ahead of time how many topics are in your document so we want an clustering algorithm that that is able to that is that is sort of able to um that is able to decide at at build time how many how many clusters you need and thirdly you sort of want every point to be assigned to one cluster even though you know if if it's present in your lead notes that's fine but in general you probably want some sort of path from one a single a single leaf note to any other Leaf node so that you're able to um you're able to we'll see in our treat veral method that you're able to sort of get from one way to another and you have some sort of from any sort of root node you're able to get to any s and you also you don't want information to get lost and be incorporated in your PR structure so the clustering algorithm that that we use uh is the follow so first we you know reduce the dimensions of the data using umap which is a dimensionality reduction technique and we use umap for sort of dual clustering so we postform Global structure clusters using uh umap and then we form local clusters and within each of the global clusters we use G mixture models as our clustering algorithm which gives us the respective clusters that um that we then summarize and you know we tested it against this you know sort of spatial window based gripping and keing clustering and it GMM performs better and we welcome new work on this so you know our our clustering algorithm work um worked really well for us but you know there there definitely improvements possible on this and we actually look forward to what people s of come up with different clustering algorithms hello hello name meeting my name so once um once we cluster the nodes the next step is summarization so once the nodes are clustered we pass it to a language model to summarize you can choose any language model you want in our experiments across the data sets we we chose GPD 3.5 turbo you know a very effective model and we saw that uh just just looking at some of the numbers empirically we saw that or it's a 72 gbt 3.5 is able to do a 72% compression rate so the average ratio of the summary length to the sum of the child no l is point to it and you know with summarizing there there obviously concerns about hallucination if you know if your model mix up something that is not in your not in the child nodes that you provide and this is a concern so we we wanted to sort of test for this and how frequent is this hallucination concern so we we ran a an annotation study and we randomly sampled 150 notes across 50 across 40 stories and while did them and we saw that the hallucination rate is relatively low it's only we only saw other the 150 nod only six nodes had some sort of hallucination and it was like it was minor hallucination it was um it was mostly due to the morel sort of adding in extraneous information so you know if the morel sort of knows about the topic it might add its own um own own information to it so it wasn't the information wasn't correct it's just not that it wasn't um in the in the child nodes there but this was relatively small so Hallucination is not really a huge concern and you know with models getting better I think this will be something that hopefully will be solved soon yeah and let's sort of look at um you know how Raptor scills obviously uh computational feasibility is important and it's great that L Raptor sort of scales linearly in both your uh build time so you know it doesn't have some sort of quadratic uh quadratic factor to it and also in the number of uh tokens that you spend because obviously you will you are summarizing your um your retrieval Corpus so you you want to ensure that you know you don't you don't make too many queries so it scales linearly in the in the size of your t Corpus and this is because as we sort of show saw earlier that the compression rate is 72% so you know for for um for the amount of tokens that you pass into your model you only get um you know 28% back as your output token so it's it's it scales linearly and it's that's why it's sort of competition useful and we ran all these experiments on like a 16 um I think on a 16 uh GB of RAM like apple normal M1 computer so it's these aren't run on some very fancy GPU just your you know everyday Consumer Electronic so it's it's feasible you can sort of build the trees on your on your own laptop and you can use Raptor so once once we build once we build our tree structure now you want to query from it and there are two methods that we explored the tree traversal method and the collapse retrieval method so the tree traversal method the way it works is you know you sort of Traverse the tree from the root nodes all the way to your uh elak nodes and the way it works is that you let's say you have you you decide the the beam size or you know how many how many nodes are you going to keep at every level and your depth size so you know do you want to retrieve only the first two from the first two layers or till end right and the way it works is um so at the root node you sort of do some sort of similarity search your vector based similarity search and let's say if you're here your top is just one so you keep your you you keep your um the top chosen node and then for the next layer you only look at the subset of nodes that are the children of the first layer of the first of the node that you've chosen and then you do similarity search over just these children nodes and let's say you get for example in this you get you choose you again choose your K nodes so here top one and then you again just look at the children of that node and then you sort of choose your node and this this forms your retrieve context that you pass to your language model uh on the other hand the collapse retrial much simpler you you essentially Clap the entire tree into a single into a single layer and you run um you run similarity search on the entire on the entire on the entire tree at once effectively and then again here you can sort of see top three so you know similarity your normal similarity search and you choose the top three chunks and you pass it to your language model again as context so what are sort of the tradeoffs of these two approaches and you know experiments the collapse tree um approach performs better so you can look over here the green curve is the collapse tree approach whereas this one is the tree traversal approach um uh and the reason for this is that you know collap streen naturally provides more flexibility for treat reversal since you're searching across all nodes simultaneously so you know you can if if it's a thematic question you might sort of retrieve nodes that are more towards the top level versus if it's a more granular question you can retrieve nodes at the leaf level and T veral on the other hand you know if you are if you're if you choose consistent values of your depth and your beam size at each level it keeps a constant noish right so at every level for example you will only choose one node at every level so it keeps an equal amount of information of lower level and higher level nodes uh so you know this is this is like question agnostics so you you not like depending on the question you're not changing values but we we I think the benefit of treat reveral is that if you do know what your questions are going to be ahead of time or if you if you know that you're going to be quering just on the leaf just on just on a thematic question or just on a question that is more granular then you can sort of you know change your start layer you can change your U you know beam size and you can choose the layers that you're going to achieve with with more effectively so in our experiments we we didn't want to tune it so we we use collapse 3 and you know that perform better but if you if if you're using this in practice and you have an idea about uh what sort of questions your users are you going to ask then based on that you can sort of decide if you're going to choose a higher level node or a lower level you're going to choose to retrieve from layers that are on the top or from layers that on the bottom so let's look at you know contribution of the tree structure and just a small thing over here is that in our experiments we saw that when we when you compare it against your normal um you know traditional retal augmentation methods we saw an interesting thing that every time um anytime Raptor sort of if you're for the same question U the notes that Raptor that Raptor retrieves and the nodes that your normal traditional augmentation method retrieves we saw that Raptor at always has at least um a parent or an ancestor um in its retrieval uh in its in the in in the chunks that it retrieves from the nodes that uh your traditional retrieval augmentation method would use for example if you look over here if uh for example over here uh your traditional DPR in this case your traditional retrial augmentation method retrieves this node but you also see you know Raptor for example retries its PA node so the high level nodes will the in in collaps 3 your high level nodes uh might always have a path to lower level node so any information that is empirically we saw that any information that your normal retri augment augmentation method retrieves we have a node that has the same information in the form of the summary and so we were looking at you know we we validated is so the number of tokens of range all the way from I think 16,000 sry I think 2 3,000 stories and we test on quality which is um more which is more medium length uh medium length um uh documents so it's around 5,000 on average of 5,000 tokens and then Cas Casper which is uh question answering on actually NLP NLP papers and so that that has a average you know to token length of I think 19 to 20,000 tokens so we're looking at you know three data sets and we wanted and we were building these trees and we were retrieving using the clap tree approach and we were sort of curious to see you know where where are the nodes coming from so interestingly we saw that around 30% of the nodes are the higher level nodes and um and the rest are sort of leaf level nodes right so it's it's interesting that's only of 30% of the nodes are are necessary or were being retrieved that from the high level nodes so we're looking at you know how much how much do these nodes actually help our system so if you look at at um if you look at the tree structure we sort of see if you if you just retrieve from you know your layer zero or your Leaf nodes you you can look at you get let's say 57% accuracy on this but as you add more and more layers and you start retrieving from all these layers you can see that you know the quality consistently uh increases across so this is from from three random stories we have you look at our paper you have a lot more stories there but you can you can see the general trend of as you add more uh more layers and you retrieve across all of these layers you can you can see an increase in accuracy from 57 to 73 here 66 to 83 and then 57 to 68 and it's interesting that you know if you if you look at these results if you let's say only retrieve from an intermediate to let's say you only retrieve from ler one you can sort of see effectively match the performance of just retrieving from layer zero or you're close to that performance so another thing might be if you recall that layer one sort of has uh significantly uh less number of nodes as layer zero so just by sort of you know creating these synthetic chunks or just summary chunks you can even match the performance of your layer Zer nodes um so one one interesting thing over here is that if if your retrieval Corpus is really big and you know all thear search with fa it's really fast if you you sort of want to reduce the number of chunks in your uh in your carpus you can effectively just uh you know cluster and summarize and then just retrieve from that note but obviously adding more and more layers and then retrieving from all the layers helps a lot as you can see here and you know Raptor uh Raptor also has St of the art results on the 3D assets I mentioned so quality uh you know it's around 8% accuracy see on on on narrative QA on on The metor Benchmark um we have the state of theart and on Casper which was a long from question answering on LP on LP papers so one interesting result that we saw in Nar QA is this result with the recursively summarizing books uh model so this was this was a paper by open AI a few years back where they sort of you know just based on the window or spatial positioning the sort of grouped and summarized their summarized summarized the chunks and they were just using sort of the top the top level nodes as context for their model and you know it's it's evaluated on the sort of the same model and the so the unified QA model and we see that you know just just looking at these numbers you can sort of see a big boost in um in in accuracy from just using using all of the nodes the intermediate nodes rather than just using the top level noes so you know the intermediate nodes certainly help and obviously uh this is based on spatial positioning on where like just clustering or just grouping together nodes that appear together um in your t Corpus whereas you know we're we're using a cluster algorithm so how to get started with the Raptor so you know you can check out the Raptor GitHub uh at this link and you can get started raptor is also available in llama index and you can um you can use that too if you're building llama index and there's also this uh great Lang chain tutorial on rapt and CLA 3 so actually you know funny enough it's it seems like long context plus plus retrieval augmentation might uh give you even more B that's just uh rather than just using either long context or augmentation I think them together really form a deadly combo and some concluding thoughts on this so you know pre retrieval we saw gives our retrieval system both higher level and lower level information about our retrial carpus this you know outperforms just methods that um rely on the underlying raw text and the synthetic chunks really help a lot and provide a more holistic understanding of long documents and we're super excited to see what everyone builds with raptor so you thank you and if you have any questions love to hear them thank so much par that was a great presentation I definitely learned a ton from that uh I think we have a lot of questions in chat which we try to read over and paraphrase and we'll love to hear your take on some of those I think most of these are specific to the paper I also have a list of question that I would love to hear just your general thoughts about the space in term of rag research long context LM which I'll try to get into afterwards um I think the first group of question uh so there a bunch of them on the chat is about like the clustering algorithm um so we love to hear a little bit more just intuition about a clustering algorithm and for example I think a specific question is is in a document a cluster of paragraphing itself that has more meaning than a cluster of paragraph on the same subject from multiple different documents so I think what uh the question is getting at is like uh the document itself has some semantic connection between its constituents so it might be a good way to group these segments to begin with and how to think about this in the context of like having a large corpus with multiple documents which You're Building these cluster on top of I love to hear your thoughts about in general like you know these different paradigms in dealing with one document or multiple documents over a large Corpus yeah sure I mean that's a really great question so if we sort of go back to the the clustering the clustering algorithm I think you know we wanted so so the so if you have sort of documents um that that you know have the spatial spatial thing so you know if if they're within the same document they probably have a you know high enough cosci and that the clustering algorithm itself will one catch that and will grip group them in the same uh in the same cluster secondly you can obviously and we we played around with this in the beginning you can you can sort of add a both you know spatial and a clustering based approach so you can you can very easily you know incorporate that you want a certain window size and also a certain cluster you want the the chunks that you sort of um sort of get the chunks that are within um within the cluster to summarize so you can sort of add both we notice that you know if you just do clustering um it itself sort of captures that so there's no real real need of being explicit about choosing um you know documents that or like paragraphs that are you know close to or within the same document or are within the same thing so are the clustering algorithm sort of automatically because if you think about documents that are chunks that are in the same document will'll probably be talking about similar things so they'll automatically uh come within the same cluster yeah this super interesting and just to clarify when you build these clusters they contain chunks of across different documents is that correct yes so you know whatever text you pass in um it'll contain all of that so if you pass in multiple documents it contain chunks across the different documents gotcha I think I just saw a follow-up question uh that's specifically about this like if you have two separate documents for example one is this financial report about 2021 and the other one it's about 2022 and if I ask specifically about 2022 how to make sure the retrieval doesn't pull information about the 2021 report I think that's sort of related to this discussion topic right I think for that you you may want to sort of so there are a few ways of doing this I think one way is obviously you know if you have if you have metadata you can incorporate that secondly I think what might be interesting in this case what I would do if you if you have sort of an updated so we're working on actually how to sort of update these trees but secondly if you have uh you know if you have reports on 2022 and 2021 what you can very easily sort of do is um you know you can look at the nodes and you can look at the paths so if there's any path from the 20202 nodes and from any path from the 2021 nodes you can just ensure that uh you only take nodes that have a path from the 2022 nodes but updating updating the trees is an ongoing workan orever excited to have something for you in some time yeah for that good question yeah that's that's actually uh going to be my next question how do you add new documents to these indexing algorithm um I know it's going to be a follow-up work we love to hear a little bit about just your high level intuitions and what are the directions you might explore sure so I think one one very interesting sort of pattern that we're exploring is that so using the Tre traversal nodes you can sort of figure out what nodes at every level are are you know semantically closer to your to your nodes right so your two to a quy for example so if I have so the line of work that we're sort of exploring is that if we have if you have a retrieval Corpus that is um you know that you have new new information you can figure out you can figure out what are the nodes that are going to be impacted with this information just based on Sim um you know similarity and then once you figure out what are the nodes that are going to be impacted with this you can only you can sort of you can you know resmar eyesee them with the new information so and and you can add a path to their to the leaf nodes so that's that's sort of the line of the work that you can do so you can f figure out the nodes that are going to be impacted by your new information and then only update those nodes gotcha that makes a lot of sense um I also saw in the chat uh a lot of questions about like interesting examples or visualizations as relates to the clustering and a retrieval aspect I'm curious if you have looked into sort of specific queries and how that's kind of like being uh what are the document that retrieves and what are the Clusters that actually look like with respect to those to help Raptor achieve the performance Improvement yeah so I mean we have we have some examples in the paper so you should definitely read the paper secondly um so we're actually working on this visualization thing so you can just call you know Raptor do visualize or audio. visualize and we'll make this amazing diagram for using some graph graph uh libraries so I think that should be going out soon so you know you should stay tuned for that so then you can sort of visualize all the parts and all the things um in itself but the clusters that we saw you know for example um so like one thing we saw that you know with the contrive example I had you know like part of the singer you know part of the violence and as violence and so on you can sort of see that depending on the topics it's really good at picking out different topics and then summarizing it on topics if you look at the Clusters themselves they're they're pretty topical in nature which is very interesting but you can you can if you you can do that yourself using just um so if you're using the rap for getup for example you can ask it to uh return the layer information and you can look at the notes itself but we're also working on a visualization tool so that that should hopefully uh you know you should be able to see some really fancy diagrams and how the tree is being constructed sounds great yeah I definitely will personally uh play around with it a little bit more to understand sort of the behavior in the clustering um another follow-up question uh this is about clustering as well like I think uh Gans is asking about right now we use summarization as the main operator to apply it to the cluster of documents and the goal is to support like Downstream QA that have that needs high level context right do you have any other thoughts on other type of operations on these cluster of documents that can support different use cases perhaps um right so I think so once you have clusters I definitely think uh depending you can you can definitely add you know to your summarization step you can definitely add more um sort of like more more context with so one sort of thing that I was just thinking about yesterday might be po it's like so this is completely you know like uh some discussion I was having with somean actually is like you know what about code drafter for example so you if you want to do this on code maybe then you know you may not want uh you know you may not want just sort of a summarization of there but you might want to sort of you know have uh a you might want to have sort of graphs that sort of the cluster represent function calls between each other so you definitely there there definitely is follow up like potential ways of sort of improving other algorithms so you may not want to summarize your functions but you might may want to just definitely have some sort of you know uh graphs or paths from one one uh one function to another so some something like that might be interesting to Sol solve like get up issues for example maybe yeah that's super interesting uh that's my next question um and also coming from the audience right like um the question specifically was about code but I think in general the concept of like the richness of different documents and different type of granular semi-structure nature uh is very interesting to play with in term of these kind of hierarchical retrieval techniques um and I think like another top of Mind challenge for a lot of practitioners are like dealing with tables in these like PDF documents as well as like rich media documents I'm curious if you have thought about you know like in the context of not just like playing text but having a little bit more semi-structured components inside how does that interact with this hierarchical retrieval and summarization approach curious to hear your thought yeah I think if you have you know images and tables I know there's a lot of you can be very easily sort of incorporate with raptor I know there's a lot of I think chain of table or something on those lines or like papers that that are able to extract information from this so let's say image you can pass it to you can pass it to you know all these models that can take in image inputs so you know I know Gemini um Gemini can take an image inputs you can have obviously gbd4 V that can take in image input so it'll be interesting to what you can sort of you can sort of do is that you have for clustering you have you have models that actually are able to embed images in the same space for example clip right so you can embed images in the same space as your text so you can actually cluster images too and and then in at the summarization step what you can do is for for models that sort of support images you can actually pass in your image and your text together and you can ask it you know summar the image uh what is the things inside the image and also use the text uh you obviously use your text chunks too so I think so they're definitely so I think it's a great idea so you can actually definitely sort of add images and um so I think there's also work on the lines of inventing tables and so on so you can definitely add that with raptor but yeah definitely very interesting line of work there Co thanks for the great answer I think like uh we had a webinar on sort of table QA uh I think in the previous one so definitely a lot of very interesting research there and sort of the integration of those table reasoning and retrieval and sort of these hierarchical and clustering type of work how that work together is a very interesting research Direction in my mind um sort of changing topic a little bit sort of still related to sort of this grouping and retrieval concept I think a lot of challenges in in practice is dealing with documents that either like it's ingested across time or updated across time right so you can imagine like a knowledge base that have you know like documents from like five years ago two years ago this year which might contain similar topics and often time people want to retrieve the most you know relevant information that's like recent right um have you thought about how to kind of uh uh give the ability to control this like sort of boosting or reranking aspect when it comes to time and how that relates to you know clustering and grouping of these information yeah I think for more recent things um you know if you're if you're summarizing it I think uh what you can simply do and you I haven't tested it but what you can simply do is just sort of add your your information about time to it so for example let's say um I have I I created my tree on something that is from 2021 and now I have I I have more documents from 2022 if I just prepend you know this this document is from 2022 and you prent that to every chunk so then when the model is sort of summarizing it again so it sort of knows this is from 2021 and this is from 2022 so it sort of in the suies will sort of like say that okay so from 2021 this thing and from 2022 this thing so when you're retrieving chunks you will have chunks that um have information so the only the only the chunks that again so when you're updating the tree and uh only the only the chunks that have are the chunks that will be updated are the chunks um that sort of need to be up right the the thing that I'm trying to get at is if you have chunks are not time dependent so I have some information in 2021 That Remains the Same in 2022 those if if my report about 2022 will does not contain any information about those things it'll remain the same but based on your cosign similarity if if I have information in 2022 that gets updated um so the chunks that are being updated um are the chunks that have that information therefore you can just prepend some some time information and update it like that gotcha thanks for the great answer um I'm just checking for uh the latest question on uh the chat uh so Bram ask how costly is the method um I know you have some like high level uh sort of takeaway shared up via the slide but we'll have to hear maybe a little bit about little trade-off in term of like amount of additional compute required to do this method and like what are the scenarios in which like this is best suited for to help a practitioner choose like different techniques for their use case that's a that's a great question so how how many tokens I think you can sort of roughly estimate it pretty pretty easily so as as I said you know your compression rate um if I go to the summarization step you can sort of see you know it it summarizes 72% of your thing at every level so you can sort of um you know bounce it um by around I think like a factor of two or three sort of tokens so you know if I have AAL Corpus of a th000 tokens I know the input do output tokens um so if there's 72% compression rate I sort of know it's like bounded by around two or three end so you know if if if that that is the amount of you know thousand if I'm using a th tokens I know that my input plus output tokens roughly going to be on the order of like 2,000 or 3,000 because because you have soft clustering so might be a bit more but you sort of know the bound on it as for you know computational things so we we ran our experiments on on a normal Mac right so just a map Mac with 16 gigs of RAM so it's it's definitely feasible to build uh to build sort of these these trees on your laptop and if you have access to better compute then obviously you can sort of use that but just on your Mac you know we we've created trees that you know raate that go up to 400 ,000 open so pretty pretty big trees about like long long stories and so on so I think uh it's it's definitely computationally feasible to do it on on your own sort of compute and if you have access to better Compu that's better but obviously but it's definitely feasible to just do it on a consumer grid laptop I think the other thing is that it's an offline process so it's one and done right you can create some tree for a set of documents and you can just reuse it over and over again forever maybe you update it yeah that makes a lot of sense thanks sna for adding on I'm just scrolling through the chat to see if we missed anything um if anyone has a burning question please raise your hand or unmute yourself and ask um but I do have a bunch of like higher level question I would love to bring storm with uh auths yeah sounds good cool um maybe uh one of the biggest like um question I have is just um aside from the sort of concrete question we talked about is there any other future directions that you guys are thinking about um that we like to share um be answer of some of the questions we just asked I mean I think you know adding adding information is definitely one but I already talked about that I think improvements to the clustering Alum General to right now we use you know GMM so GMM you know assume that your text is Goran which we saw that empirically you know like text is more sort of sparse and skewed but it it worked really well you know empirically but we're sort of you know exploring maybe approach that might be more so you know like that might be for example GNN based right so it might be might be using a graph neural net to actually form the Clusters as opposed to just uh you know using just using Guardian mixture models so we're exploring on the lines of um on the lines of improving the clustering algorithm secondly I think you gave us a great idea about you know looking into how you can use visual information I know you know images uh using images or tables and so on if you can embed it and you can pass it to your language model you can use Raptor with it so it'll be interesting to sort of see uh you know using information and tables on with raptor but I think updating the tree is probably like a very very heavily requested feature so that that probably has our has our attention on it for now I think someone in the audience raises hand so I'm going to ask him to ask the question hello uh yes uh thank you so much for this webinar very useful um I have one question uh specifically about the data set like in I'm not sure like how to um how to like explain this exactly but like uh what we trying to do is um use a large code base like analyze uh large code bases and like specifically uh function by function so we kind of want to be able to um have a summary of the function and then like find all all the dependencies that are being called in this function or where this function is being called and also like Traverse the tree and summarized like everything and then at the end basically gave an explanation um what uh this part of the code is doing uh or generating the documentation this is specifically very useful for old code bases um and I was wondering if raptor is something that can help us thank you yeah I think code is definitely very interesting and something we're also thinking about as I mentioned uh with with with code so if you if you don't want to sort of if you just want to explain things or if you just want to use the information as to uh find out some things then rafter as is should be useful because it is it is going to summarize so if you use Raptor with code it's going to summarize what your functions are doing and so on it might build um it might build sort of this three structure so an interesting thing would be if you I'm not sure I haven't tested how how well clustering algorithms sort of work on code but but there for example but you can definitely fine tune uh just you know normal normal bird based um bir birth based encoders on on code I think cird for example you can they their repo supports just you know passing in custom Uh custom data sets and then fine-tuning on that I think the the interesting thing over there is so so if you're just sort of using it to answer questions and not exactly run the code then you can definitely use summar summar summarization and so and obviously function calls within themselves have have a u have a hierarchy within them right so I have a main function which is calling my other functions and so on so I would imagine you know the the clustering algorithm picking up something like that so for for example if you have a function call in one function that probably will have a high similarity for example if I have function a calling function B and then a function C also calling function B since they're both calling function B they will probably have a higher um high enough cosine similarity they'll be they'll be grouped together but I think using using U using the information in it without using the without explicitly clustering it um might might be so if you're if you if you want to run the code and if you want to sort of have this hierarchy you can sort of use that as well and that's something that we're also planning on exploring so if it's just so you know the the short answer is if it's just answering questions um I think Raptor just as is should work pretty well because it will cluster things that are calling the same function and so on together but it it it'll be interesting to sort of incorporate the the natural hierarchy in in the code base itself and that that's something that we're also planning on exploring but we'll be excited to see something from the community on that yeah it's a very interesting discussion I think in term of different document type there's a whole spectrum of like unstructured to fully structured and in my mind like there's different aspects of leveraging our reclustering versus like more structured Travers so that uses the inherent structure itself in that document um I think this kind of like uh gets me into a particular question like there's no good answer to this I think otherwise we would have solved this already but you know like um different type of questions and different type of queries can be like inherently very very different tasks right and I think like one of the promise of large language model and some of these structured ritual techniques we're basically saying that like there might be a good General way to do some of these that cover broad spectrum of tasks right am I curious about your thoughts in term of like crafting domain specific retrial techniques that Target specific document types and query types versus like trying to approach a more General generalist way right so this is like almost like do we get AGI to solve this for us or are we going to still go into a more crafting individual modules and routing to the individual modules right like I don't think I don't pretend I know the answer or there's a good answer we love to hear your thoughts here yeah I think for for domain specific things the way you can sort of improve um you know performance on on different queries is one uh just having sort of better embeddings model for sure so if you have some sort of you know data set and you want to you know use so so the embedding model that we use are very generally trained so you as has been train a lot of a lot of different sort of types of uh retrievable corporat so it's very general but if you if you know the sort of um retrial Corpus Ahad of time you can definitely you know fine tune your on embedding model so the Clusters are better and so on secondly I think so it depends on if you have insights on what sort of questions you're going to ask then as I mentioned you can you can sort of tweak the parameters of Tre veral and so on and one one interesting line would be to not do this by yourself but actually just ask a language model is this a higher level question is this something that needs more uh load out information so I know I know there's a lot of work being done on creating creating queries using uh creating like fake like either multihop queries using using language models so an interesting thing here would be to you know ask a language model to either you know create depending on what what sort of the query is to you can obviously ask it to create higher level queries that might answer this question and then restrict your tree siiz to above or you can sort of you know make create lowlevel queries and just uh you know make your you can choose the layers in the tree that are more low level so so you can definitely depending on what sort of thing is you can definitely use the language model as an intermediate step to sort of decide these things and I I I would guess that it would perform actually really well thanks um we we have another question from the audience so pascar go ahead yes thanks uh yeah to go a bit further on the structure data so if I understood correctly your base chunking is a fixed size uh did you try to use CTIC chunking or other kind of Base chunking before applying your uh clustering algorithm so we we use the simplest thing so we wanted to show that you know without without any without any hand tuning or choosing any sort of very fancy chunking algorithm that our our system performs well but obviously you know improving chunking will obviously lead to better results so you know we wanted to show a more General Paradigm that in the most basic case of just chunking every 100 tokens and just not splitting sentences across chunks so very simple thing uh to do you can get good performance but obviously at like it it's the the better the better the chunking the better performance so we didn't we didn't test it with our um with our uh with our codebase because we wanted to sort of you know because 100 chunk tokens is is sort of also standard in the literature so we wanted to keep it comparable with previous methods as well but you can definitely improve the chunking and it'll lead to better performance so we didn't test it but I like if you if you do that it's very easy to sort of implement it in the Raptor code Bas or I assume with l x and you can you can build on top yeah okay we just have four minute left and I do have one question I want to get in and I'm sure this is something you've been asked right so what are your thoughts about long context LM and how it's going to impact rag this is a question we get as well so you know I have my own answers would love to hear what are your thoughts um I think that's probably going to be a future that we go towards right as the sort of context length gets quote unquote commoditized and like I think the way we think about information retrieval as well as summarization syis will have to change a little bit especially on interface between data and sort of reconing engine um I think like the setting you're working with is like at 100 token limit for each of the chunks as well as like you know like uh Gathering up to like 2K or 4K tokens I don't remember exactly but in the world where we have you know like 1 million 10 million or even 100 million token size you know how do you think about this what are the task in which we should think more about retrieval and what are ones that we should just kind of offload it to just the reasoning itself yeah I mean that's a very pertinent question I think uh my thoughts on this are for now I think you know obviously your retrieval Corpus can be trillions of Doc trillions of tokens right it's it's still I feel if you look at the Retro Retro moral for example by fine it retrieves from a corpus of trillions of tokens so I'm not sure if it makes sort of sense to pass in a trillion tokens every time you want to you know answer a simplest thing or do so so I think retrieval is here to stay in my opinion I think the way the best way to is sort of to incorporate you know Best of Both Worlds so I think um if you if you have so you know raptor is obviously chunk siiz agnostics you can choose as long as you can embed it and I know they're better embedding models and I think grit Alm for example you can sort of pass in it it can do both generation and Ed at the same time so if you have a language model uh that can sort of embed these documents you might you can sort of think about scaling up Raptor such that you're not you're not restricting yourself to 100 100 chunk tokens but you can embed entire documents if if if you want to you know create this high tree so then let's say you know I have the entirety of Wikipedia what I can do is you know embed a one page per per chunk and then sort of create a big tree based on thetio Wikipedia for example so I think enabling long context I I mean enables us to do think much more and think at a much bigger GE of retrieval so not just retrieving 100K tokens but retrieving entire documents but I think retrieval you still sort of need retrieval for that absolutely yeah well thank you so much par and Anna and everyone else on the paper uh thanks for really pushing the forward pushing forward like the boundaries of knowledge on rag specifically and I'm sure we have a lot of different future directions to explore um I guess you get either in academic or startup um so and hopefully this session was informative for the audience and also it give you some signal in term of what future Direction you can explore that are sort of top of mind for the practitioners and um yeah thank you guys so much and thanks all the audience for attending um I think we'll close it off here thank you for having us it's great uh you know to get to get um out to so many people and we're excited to see what people build with raptor and what it enables thank you thanks everyone""