<transcript><audio_149>different ways uh including pointing to the primary index what's the protocol I use to protect the right objects right as versions are getting created I still need to protect that new version is getting created and you can do two-phase locking or optimistic concery control</transcript><transcript><audio_150>post uses some a combination as 2pl and also a time order protocol so a lot of these things are feasible as combination if you're curious about the types of things that are feasible there are two papers that you may want to read at One is Andy had a beautiful paper on</transcript><transcript><audio_151>inmemory mvcc so even a smaller version than on disk based mvcc which is even more choices for you but even in that the explosion in number of choices is huge and he does an excellent job he and his students of cataloging the different</transcript><transcript><audio_152>mechanisms giving them proper names and categories it's a beautiful paper to read if you're interested in that and in memory mvcc is super inter because a lot of transactional databases fit in memory because today you can get a 4 terab server and very few transactional</transcript><transcript><audio_153>databases need a lot more than eight or 10 of those to really come do all of their workload including with replication and a little bit of fall tolerance uh there's another paper if you wanted to go into a depth of one specific protocol I'd recommend the</transcript><transcript><audio_154>hecaton uh paper which is very clean uh was developed by these guys at Microsoft including Paul Larsson who's a giant in that field he's since retired uh but one of my students uh worked on this it's a very clean protocol as simple as uh Paul</transcript> different ways uh including pointing to the primary index what's the protocol I use to protect the right objects right as versions are getting created I still need to protect that new version is getting created and you can do two-phase locking or optimistic concery control post uses some a combination as 2pl and also a time order protocol so a lot of these things are feasible as combination if you're curious about the types of things that are feasible there are two papers that you may want to read at One is Andy had a beautiful paper on inmemory mvcc so even a smaller version than on disk based mvcc which is even more choices for you but even in that the explosion in number of choices is huge and he does an excellent job he and his students of cataloging the different mechanisms giving them proper names and categories it's a beautiful paper to read if you're interested in that and in memory mvcc is super inter because a lot of transactional databases fit in memory because today you can get a 4 terab server and very few transactional databases need a lot more than eight or 10 of those to really come do all of their workload including with replication and a little bit of fall tolerance uh there's another paper if you wanted to go into a depth of one specific protocol I'd recommend the hecaton uh paper which is very clean uh was developed by these guys at Microsoft including Paul Larsson who's a giant in that field he's since retired uh but one of my students uh worked on this it's a very clean protocol as simple as uh Paul


<transcript><audio_155>could make it with help from others uh and it also has an interesting aspect because they had SQL Server which is an Onis stuff and they wanted to add this mvcc inmemory stuff and they very interesting ways in which they could cleanly put that together so hecaton is</transcript><transcript><audio_156>the inmemory mvcc which can be uh put as an extension to SQL Server which is an on desk system so very clever engineering and a very clean protocol for the inmemory case so those of you who have been asking a million questions</transcript><transcript><audio_157>I love that uh please go take a look at that these papers won't be covered in The Graduate database class but if you're interested stop by my office hours and be happy to walk you through the paper and take questions okay and I'm sure Andy would too all</transcript><transcript><audio_158>right you guys are doing a good job of making sure the exam is not going to have a lot of material moving all right and let me just make</transcript><caption><image_159>The image is a screenshot of a lecture titled "Lecture #19: Database Logging". The background of the image is black, and the title is written in white text on the top left corner. Below the title, there is a subtitle in red text that reads "Intro to Database Systems (15-45-4545) Lecture 19". On the right side of the screen, there are two illustrations of two men, one with dark hair and the other with light-colored hair. The men are facing each other and appear to be in a conversation. The man on the left is wearing a suit and tie and has a serious expression on his face. The image appears to be taken in a lecture hall or lecture hall.</caption><transcript><audio_160>great all right so uh we're going to talk about logging uh next and it's part of a two piece component we'll figure out how to log things so that we can make changes but if something</transcript> could make it with help from others uh and it also has an interesting aspect because they had SQL Server which is an Onis stuff and they wanted to add this mvcc inmemory stuff and they very interesting ways in which they could cleanly put that together so hecaton is the inmemory mvcc which can be uh put as an extension to SQL Server which is an on desk system so very clever engineering and a very clean protocol for the inmemory case so those of you who have been asking a million questions I love that uh please go take a look at that these papers won't be covered in The Graduate database class but if you're interested stop by my office hours and be happy to walk you through the paper and take questions okay and I'm sure Andy would too all right you guys are doing a good job of making sure the exam is not going to have a lot of material moving all right and let me just make great all right so uh we're going to talk about logging uh next and it's part of a two piece component we'll figure out how to log things so that we can make changes but if something

Creative Commons Attribution license (reuse allowed)


<transcript><audio_161>crashes like some of the questions that were asked it's like what happens if I'm adjusting this chain but I haven't finished fixing the chain or I fix the chain and something bad happens types stuff like that though largely today we are only going to focus on single version protocol but the concepts will</transcript><transcript><audio_162>apply in the other cases okay and again it's like if I can if you can get you the single version foundational pieces in there you'll be able to read papers by yourself and figure things out I was just telling uh a couple students last week and earlier today too is that the</transcript> crashes like some of the questions that were asked it's like what happens if I'm adjusting this chain but I haven't finished fixing the chain or I fix the chain and something bad happens types stuff like that though largely today we are only going to focus on single version protocol but the concepts will apply in the other cases okay and again it's like if I can if you can get you the single version foundational pieces in there you'll be able to read papers by yourself and figure things out I was just telling uh a couple students last week and earlier today too is that the


<transcript><audio_163>best we can do as teachers is to teach you how to learn things so hopefully we can get the foundation material that you need and this is a never-ending game of how you can invent better and better techniques but before that a couple announcements project 3 is due really</transcript><transcript><audio_164>soon there's special office hours on Saturday uh 3: to5 PM in GSC 4303 if you still need that help some of you are already done congratulations but for those of you who need help uh uh that's available project four is on concurrency</transcript><transcript><audio_165>control it's going to be out today and uh that's you December 10th so you have a little bit of time for it but there's a bunch in there so you know don't wait again till the last minute for going through that and the write up unfortunately is going to be a little bit uh uh uh detailed because you're getting into concurrency control like</transcript><transcript><audio_166>optimization it's hard the optimization piece that you did was very light but here you're going to have to get you into uh mbcc and stuff like that so buckle up it's going to be fun but you're going to get to do uh uh some</transcript><transcript><audio_167>interesting things with mvcc and o2n type of version chains a bunch of database talks there are three more left in this and I think there may be I think this is the last three in the semester uh I strongly recommend not missing uh</transcript><transcript><audio_168>especially the Alibaba talk because they're doing something at a massive scale and uh it'll be super interesting to see what they talk about the PG Vector stuff and chroma is coming after that all right so let's get to recovery protocols today really fun stuff so you</transcript> best we can do as teachers is to teach you how to learn things so hopefully we can get the foundation material that you need and this is a never-ending game of how you can invent better and better techniques but before that a couple announcements project 3 is due really soon there's special office hours on Saturday uh 3: to5 PM in GSC 4303 if you still need that help some of you are already done congratulations but for those of you who need help uh uh that's available project four is on concurrency control it's going to be out today and uh that's you December 10th so you have a little bit of time for it but there's a bunch in there so you know don't wait again till the last minute for going through that and the write up unfortunately is going to be a little bit uh uh uh detailed because you're getting into concurrency control like optimization it's hard the optimization piece that you did was very light but here you're going to have to get you into uh mbcc and stuff like that so buckle up it's going to be fun but you're going to get to do uh uh some interesting things with mvcc and o2n type of version chains a bunch of database talks there are three more left in this and I think there may be I think this is the last three in the semester uh I strongly recommend not missing uh especially the Alibaba talk because they're doing something at a massive scale and uh it'll be super interesting to see what they talk about the PG Vector stuff and chroma is coming after that all right so let's get to recovery protocols today really fun stuff so you


<transcript><audio_169>remember we had the asset components so far we' only done I which is the isolation stuff okay we still have a and d to cover and C as we talked about is consistency based upon you know databases Integrity constraints and other forms of defining what semantics</transcript><transcript><audio_170>need to be enforced and we will enforce that so let's start getting into the recovery component which is the last missing piece to complete the asset components okay so we going to want transactions to be all or nothing and we</transcript><transcript><audio_171>want to AP aart and we need transactions to be durable so if a transaction is declared committed even after that if the system crashes the changes are recorded in the state of the database so we'll start simple with uh motivation</transcript><transcript><audio_172>simple transaction Reading Writing stuff and now we have a buffer pool which obviously database systems have but now we're going to make it explicit in our diagrams because it's because of the buffer pool that we going to have to worry about a lot of things so we bring</transcript> remember we had the asset components so far we' only done I which is the isolation stuff okay we still have a and d to cover and C as we talked about is consistency based upon you know databases Integrity constraints and other forms of defining what semantics need to be enforced and we will enforce that so let's start getting into the recovery component which is the last missing piece to complete the asset components okay so we going to want transactions to be all or nothing and we want to AP aart and we need transactions to be durable so if a transaction is declared committed even after that if the system crashes the changes are recorded in the state of the database so we'll start simple with uh motivation simple transaction Reading Writing stuff and now we have a buffer pool which obviously database systems have but now we're going to make it explicit in our diagrams because it's because of the buffer pool that we going to have to worry about a lot of things so we bring


<transcript><audio_173>in a page the page has a lot of stuff lot of Records lot of columns in each record but the uh one of them is that value a that we are trying to write the column A and we'll go bring that in and then when we write it we will go and</transcript><transcript><audio_174>update that in the buffer pole okay that's what we do right the updates happen in the buffer pool now at the time that we get to do a comit what do we do to get</transcript><transcript><audio_175>durability we could insist on taking any change that has been made and push it out to disc and if we do that we will get durability but it'll be very slow but let's say we have other protocols that don't require us to flush everything to disk at coming time</transcript><transcript><audio_176>because this transaction could have touched a billion records and a billion Pages you don't want to flush a billion pages but what happens if the power goes out or someone zaps the memory okay if the zaps the memory we've committed the transaction but the change was only in</transcript><transcript><audio_177>the buffer pool and we've lost it right so we don't have a durable transaction anymore that was Andy's picture of saying a bad guy comes and zaps it so um all right so that's what crash recovery</transcript> in a page the page has a lot of stuff lot of Records lot of columns in each record but the uh one of them is that value a that we are trying to write the column A and we'll go bring that in and then when we write it we will go and update that in the buffer pole okay that's what we do right the updates happen in the buffer pool now at the time that we get to do a comit what do we do to get durability we could insist on taking any change that has been made and push it out to disc and if we do that we will get durability but it'll be very slow but let's say we have other protocols that don't require us to flush everything to disk at coming time because this transaction could have touched a billion records and a billion Pages you don't want to flush a billion pages but what happens if the power goes out or someone zaps the memory okay if the zaps the memory we've committed the transaction but the change was only in the buffer pool and we've lost it right so we don't have a durable transaction anymore that was Andy's picture of saying a bad guy comes and zaps it so um all right so that's what crash recovery


<transcript><audio_178>is all about and we have to go and get this atomicity and durability components and we're going to do this in two parts so there are two lectures today we'll talk about the first part is what actions do we need to take during regular transaction processing to create</transcript> is all about and we have to go and get this atomicity and durability components and we're going to do this in two parts so there are two lectures today we'll talk about the first part is what actions do we need to take during regular transaction processing to create


<transcript><audio_179>stuff that we need to recover from the second part which is once you have kept stuff that you need to uh keep around how do you recover that's the uh second lecture on Wednesday so today what do we need to do</transcript><transcript><audio_180>so here are the six things that we need to worry about first we'll talk about why failures happen then we'll talk about buffer poool policies because that is at the heart of why we need to do these fancy new things and then talk about two different mechanisms Shadow</transcript><transcript><audio_181>paging and right ahead logging as a way of keeping track of changes that we make and I'll tell you Shadow paging is a bad idea it's what old system used to use it has all kinds of issues I would only touch upon it and then move to right ahead logging which is the main part that we need uh and then we'll talk</transcript><transcript><audio_182>about some of the logging schemes what do you put in these logs and then checkpointing so first is we're going to have to concern ourselves with these different storage types and we've talked about this I'm going to go to the next slide and come back to to it you</transcript><transcript><audio_183>remember this slide on the right that we had which is which talked about data is in different places and there's a reason why we are processor caches and dram and ssds and slow ssds and fast ssds because as you go down this hierarchy you get</transcript> stuff that we need to recover from the second part which is once you have kept stuff that you need to uh keep around how do you recover that's the uh second lecture on Wednesday so today what do we need to do so here are the six things that we need to worry about first we'll talk about why failures happen then we'll talk about buffer poool policies because that is at the heart of why we need to do these fancy new things and then talk about two different mechanisms Shadow paging and right ahead logging as a way of keeping track of changes that we make and I'll tell you Shadow paging is a bad idea it's what old system used to use it has all kinds of issues I would only touch upon it and then move to right ahead logging which is the main part that we need uh and then we'll talk about some of the logging schemes what do you put in these logs and then checkpointing so first is we're going to have to concern ourselves with these different storage types and we've talked about this I'm going to go to the next slide and come back to to it you remember this slide on the right that we had which is which talked about data is in different places and there's a reason why we are processor caches and dram and ssds and slow ssds and fast ssds because as you go down this hierarchy you get


<transcript><audio_184>more capacity but it's also slower and you don't want to have all the data be in the lowest capacity let's say SSD or a spinning disc if you didn't have a buffer pool the database system would be very slow so we want to use the highest level which is going to be volatile</transcript><transcript><audio_185>storage which is Dam because that's where you want to make all of your changes that's fast uh but it's volatile the non-volatile storage the SST layer is where you want to make sure you make put your changes in there so</transcript><transcript><audio_186>that if the bad guy comes in and zaps the dam the volatile storage you still can provide this durability property that you're that you have to guarantee okay now there's a third type of storage described in the textbook which is real</transcript><transcript><audio_187>called stable storage and that basically is something that survives all possible failure scenarios such a theoretical storage doesn't exist but you can come close by making replicas of everything that you do you can replicate the disk and synchronize the replicas using</transcript><transcript><audio_188>distributed transaction protocols we'll talk about that in the last three lectures of the class briefly okay so stable storage we won't uh touch at all today so we want dirty pages to sit right now we'll just concern ourselves with the two- tier scenario for the</transcript><transcript><audio_189>remainder of today's lecture and tomorrow which is buffer pool in dram and that gets spilled out to some stable storage like SST or dis which can survive a power failure okay so let's</transcript> more capacity but it's also slower and you don't want to have all the data be in the lowest capacity let's say SSD or a spinning disc if you didn't have a buffer pool the database system would be very slow so we want to use the highest level which is going to be volatile storage which is Dam because that's where you want to make all of your changes that's fast uh but it's volatile the non-volatile storage the SST layer is where you want to make sure you make put your changes in there so that if the bad guy comes in and zaps the dam the volatile storage you still can provide this durability property that you're that you have to guarantee okay now there's a third type of storage described in the textbook which is real called stable storage and that basically is something that survives all possible failure scenarios such a theoretical storage doesn't exist but you can come close by making replicas of everything that you do you can replicate the disk and synchronize the replicas using distributed transaction protocols we'll talk about that in the last three lectures of the class briefly okay so stable storage we won't uh touch at all today so we want dirty pages to sit right now we'll just concern ourselves with the two- tier scenario for the remainder of today's lecture and tomorrow which is buffer pool in dram and that gets spilled out to some stable storage like SST or dis which can survive a power failure okay so let's


<transcript><audio_190>begin why do transactions fail there are multiple reasons first transactions can fail because there are logical errors in the transaction I updated a record and oops</transcript><transcript><audio_191>the Integrity constraint fail or a constrainted on the database fail the C part so now I need to abort this transaction okay or it could be the transaction failed because everything that it's doing is fine but it's deadlocking with another transaction and</transcript> begin why do transactions fail there are multiple reasons first transactions can fail because there are logical errors in the transaction I updated a record and oops the Integrity constraint fail or a constrainted on the database fail the C part so now I need to abort this transaction okay or it could be the transaction failed because everything that it's doing is fine but it's deadlocking with another transaction and


<transcript><audio_192>sorry but you got picked as a transaction that needs to be killed so transactions can fail for a variety of reasons and that will all get part of the atomicity and durability that we doing because a transaction that has failed may have already started to make</transcript><transcript><audio_193>some changes we'll have to undo all of those changes there can also be system failures such as surprise surprise software sometimes have bugs operating systems sometimes has bugs right so your system could</transcript><transcript><audio_194>crash for a software failure and part of the work may have been done transaction has not committed or it has committed the changes were in buffer pool and we still need to make sure the right thing happens there could be Hardware failures like in the early days of data centers</transcript> sorry but you got picked as a transaction that needs to be killed so transactions can fail for a variety of reasons and that will all get part of the atomicity and durability that we doing because a transaction that has failed may have already started to make some changes we'll have to undo all of those changes there can also be system failures such as surprise surprise software sometimes have bugs operating systems sometimes has bugs right so your system could crash for a software failure and part of the work may have been done transaction has not committed or it has committed the changes were in buffer pool and we still need to make sure the right thing happens there could be Hardware failures like in the early days of data centers


<transcript><audio_195>there is to be a failure that uh the metag guys then called Facebook had written papers about saying the D Rams that were put on the servers because all of them were so close by and they hadn't really figured it out as to what happens at that scale not just them but everyone the dam chips would come off and one of</transcript><transcript><audio_196>the common failures was after a little while even though everything was fine the dam chips would just come off their slots now they glue it most of the time don't want that to happen uh so Hardware could fail for a variety of reasons it could be something like that or it could be the part actually failed uh the</transcript> there is to be a failure that uh the metag guys then called Facebook had written papers about saying the D Rams that were put on the servers because all of them were so close by and they hadn't really figured it out as to what happens at that scale not just them but everyone the dam chips would come off and one of the common failures was after a little while even though everything was fine the dam chips would just come off their slots now they glue it most of the time don't want that to happen uh so Hardware could fail for a variety of reasons it could be something like that or it could be the part actually failed uh the


<transcript><audio_197>system crashes all kinds of things that happen one thing they're going to assume today is that the nonvolatile storage the SSD or the disk does come back and has the contents because we don't have this ideal storage with the replication stuff so just reiterating that</transcript><transcript><audio_198>component the storage media could fail uh where you know you thought you wrote a page to disk but it actually didn't get written the disc driver came back and said yeah it's written but the bits got corrupted on that we've talked about bit rotting and other kinds of things you will need other types of mechanisms</transcript> system crashes all kinds of things that happen one thing they're going to assume today is that the nonvolatile storage the SSD or the disk does come back and has the contents because we don't have this ideal storage with the replication stuff so just reiterating that component the storage media could fail uh where you know you thought you wrote a page to disk but it actually didn't get written the disc driver came back and said yeah it's written but the bits got corrupted on that we've talked about bit rotting and other kinds of things you will need other types of mechanisms


<transcript><audio_199>with the replication to deal with that and again we won't cover that we will assume that something else takes care of that we'll concern ourselves with just this two tier stuff that we talked about okay where the primary storage is in this nonvolatile dram storage and the</transcript><transcript><audio_200>main storage is in this volatile storage now what we need to do as a database system is to make sure committed transactions changes make it to the stable storage and no partial changes are left around even if they made it to durable storage we can unwind ourselves</transcript> with the replication to deal with that and again we won't cover that we will assume that something else takes care of that we'll concern ourselves with just this two tier stuff that we talked about okay where the primary storage is in this nonvolatile dram storage and the main storage is in this volatile storage now what we need to do as a database system is to make sure committed transactions changes make it to the stable storage and no partial changes are left around even if they made it to durable storage we can unwind ourselves


<transcript><audio_201>from that we need two key mechanisms to do this one is called undo which is whoops I put something into stable storage into</transcript> from that we need two key mechanisms to do this one is called undo which is whoops I put something into stable storage into


<transcript><audio_202>the non-volatile storage that was changes made by a transaction that got aborted because it was a transaction failure or one of these other failures we talked about and now I need to undo that and the other one is redo which is oh the committed transaction made changes that were just in the buffer</transcript><transcript><audio_203>pool but the transaction is committed we told the world that the transaction has committed but now we need to uh go and reapply those changes so those are the two mechanisms that we need to build into our</transcript><transcript><audio_204>system so another example you start with a buffer pool and you read of a you get the page which has a bunch of things in it including the a value you write it you have a new value then stuff happens where B is changed by a different</transcript> the non-volatile storage that was changes made by a transaction that got aborted because it was a transaction failure or one of these other failures we talked about and now I need to undo that and the other one is redo which is oh the committed transaction made changes that were just in the buffer pool but the transaction is committed we told the world that the transaction has committed but now we need to uh go and reapply those changes so those are the two mechanisms that we need to build into our system so another example you start with a buffer pool and you read of a you get the page which has a bunch of things in it including the a value you write it you have a new value then stuff happens where B is changed by a different


<transcript><audio_205>transaction on the same page as where a is that was updated by transaction T1 so now two transactions have made changes to the same page and remember we move stuff from the buffer pool to the disk in pages right so now this page has two different things from two different</transcript><transcript><audio_206>transactions T2 commits what do we do at this point if we said we can flush that page to dis to ensure that p2's changes are durable uh we will also carry along with</transcript><transcript><audio_207>us A's changes and A's outcome is uh T's outcome to a is not yet known we don't know what's going to happen because if we flush all of that to disk ultimately T2 could abort and we need to go unwind the changes that made it to disk okay so</transcript><transcript><audio_208>there'll be all kinds of schemes you can come up with we'll categorize this into a nice quad chart shortly but basically saying because things are on the same page and the buffer po can move things around in pages in a given page might be changes for multiple transactions in</transcript><transcript><audio_209>different states and we have to make everything work with that scenario okay all right so if T2 needs to be rolled back I needed to know that the previous value was not A3 now that may be sitting in the version chain if</transcript> transaction on the same page as where a is that was updated by transaction T1 so now two transactions have made changes to the same page and remember we move stuff from the buffer pool to the disk in pages right so now this page has two different things from two different transactions T2 commits what do we do at this point if we said we can flush that page to dis to ensure that p2's changes are durable uh we will also carry along with us A's changes and A's outcome is uh T's outcome to a is not yet known we don't know what's going to happen because if we flush all of that to disk ultimately T2 could abort and we need to go unwind the changes that made it to disk okay so there'll be all kinds of schemes you can come up with we'll categorize this into a nice quad chart shortly but basically saying because things are on the same page and the buffer po can move things around in pages in a given page might be changes for multiple transactions in different states and we have to make everything work with that scenario okay all right so if T2 needs to be rolled back I needed to know that the previous value was not A3 now that may be sitting in the version chain if


<transcript><audio_210>that's the storage uh technique that I'm using but I have to go chase it down and I have to go figure it out okay I still have to keep track of the changes that I made which is the logging stuff which works with all of these techniques that we've discussed before it's orthogonal</transcript><transcript><audio_211>to all of that okay but as I said today we'll just concern oursel with a single version uh component of this you can do all kinds of a little bit more interesting thing with multiversion but let's get the single version foundation in first</transcript><transcript><audio_212>right okay the other complication arises from the buffer pool's replacement policy so you guys implemented the LR2 policy when you wrote the buffer pool and that gave a lot of freedom to the buffer manager simple piece of code that can decide I'll keep track of the</transcript><transcript><audio_213>recency of a page based on this LR2 counters and I'll decide when to kick something out when an eviction needs to happen the only thing we said is that if a page is pinned someone's actually using it can't kick it out but if a page is unpinned it could be dirty and I can</transcript><transcript><audio_214>kick it out so the buffer pool to get maximum performance maximum use of the space it's a cache of that cashing efficiency is saying I'm allowed to kick things out even if there are dirty changes if there are changes that are</transcript><transcript><audio_215>made to that page so that we'll call as the steel policy there's a second component uh the the that's a uh one dimension to this problem and stealing is saying the buffer manager can take a page that is unpinned and flush it to disk even if it's dirty and the</transcript><transcript><audio_216>transaction that's dirty it hasn't committed yet so we'll write uncommitted changes to the stable storage and that's okay all right no steel is saying no no no I'm going to take away this power of replacement policy from you certain</transcript> that's the storage uh technique that I'm using but I have to go chase it down and I have to go figure it out okay I still have to keep track of the changes that I made which is the logging stuff which works with all of these techniques that we've discussed before it's orthogonal to all of that okay but as I said today we'll just concern oursel with a single version uh component of this you can do all kinds of a little bit more interesting thing with multiversion but let's get the single version foundation in first right okay the other complication arises from the buffer pool's replacement policy so you guys implemented the LR2 policy when you wrote the buffer pool and that gave a lot of freedom to the buffer manager simple piece of code that can decide I'll keep track of the recency of a page based on this LR2 counters and I'll decide when to kick something out when an eviction needs to happen the only thing we said is that if a page is pinned someone's actually using it can't kick it out but if a page is unpinned it could be dirty and I can kick it out so the buffer pool to get maximum performance maximum use of the space it's a cache of that cashing efficiency is saying I'm allowed to kick things out even if there are dirty changes if there are changes that are made to that page so that we'll call as the steel policy there's a second component uh the the that's a uh one dimension to this problem and stealing is saying the buffer manager can take a page that is unpinned and flush it to disk even if it's dirty and the transaction that's dirty it hasn't committed yet so we'll write uncommitted changes to the stable storage and that's okay all right no steel is saying no no no I'm going to take away this power of replacement policy from you certain


<transcript><audio_217>things besides pin Pages you can't uh steal Pages for transactions that are running and that's obviously going to give you lot less flexibility it'll be a poorer performing system the second dimension to Steel no steel is the force policy which is at the commit time what</transcript><transcript><audio_218>do I do at the commit time if I say all the changes that were made by the transaction that is committing must be forced into disk before committing then I'll get durability but it'll be very expensive because imagine I'm a transactions</transcript><transcript><audio_219>that's touching one bite in 100 by records for a billion records I'm just changing one billion bytes but I've touched everything and I have to bring everything into memory update all of that stuff and then write all of that</transcript><transcript><audio_220>stuff at commit time okay so that'll be a lot of changes that have to be flushed at commit time and you'll have a slow system but but that could be how you how you could work in that case so for says at commit time I will force the committed transaction changes to dis no</transcript><transcript><audio_221>Force says no no no don't force that find a better way to do the commit that is more efficient and then deal with bad things can happen in a different way which is where logging is going to come in okay so let's look at one of the simplest policies of these combinations</transcript><transcript><audio_222>of Steel no steel Force no Force the easiest one is no steal buffer pool can do much it can steal a page uh and forces as you write stuff so it's a combination that works but it's obviously pretty slow but let's see in</transcript> things besides pin Pages you can't uh steal Pages for transactions that are running and that's obviously going to give you lot less flexibility it'll be a poorer performing system the second dimension to Steel no steel is the force policy which is at the commit time what do I do at the commit time if I say all the changes that were made by the transaction that is committing must be forced into disk before committing then I'll get durability but it'll be very expensive because imagine I'm a transactions that's touching one bite in 100 by records for a billion records I'm just changing one billion bytes but I've touched everything and I have to bring everything into memory update all of that stuff and then write all of that stuff at commit time okay so that'll be a lot of changes that have to be flushed at commit time and you'll have a slow system but but that could be how you how you could work in that case so for says at commit time I will force the committed transaction changes to dis no Force says no no no don't force that find a better way to do the commit that is more efficient and then deal with bad things can happen in a different way which is where logging is going to come in okay so let's look at one of the simplest policies of these combinations of Steel no steel Force no Force the easiest one is no steal buffer pool can do much it can steal a page uh and forces as you write stuff so it's a combination that works but it's obviously pretty slow but let's see in


<transcript><audio_223>that just an example as to what happens with this simple scheme which is not realistic right we want to do better and we will so you have a read of a bring that page in you've seen this page multiple times now you go right to that second transaction comes in makes its</transcript> that just an example as to what happens with this simple scheme which is not realistic right we want to do better and we will so you have a read of a bring that page in you've seen this page multiple times now you go right to that second transaction comes in makes its


<transcript><audio_224>right and force means I will now Force the changes of this page to disk but because I have changes to a that was made by T1 which is not done if I'm doing a force policy I'm still going to need to figure out what's the older</transcript><transcript><audio_225>version of this page I need some mechanism to keep track of that and then only flush that out to disk okay so just want you to know even if it's simplest scheme that you can think of you're going to have a little</transcript><transcript><audio_226>complication but we'll do much better than that this is just an example no one implements it like that because this is still be a very slow system okay but there's no free lunch even with the simplest combination you still have</transcript><transcript><audio_227>complications all right so easiest scheme to implement uh it has a couple other problems with it one is if the number of objects that I'm writing to imagine you have a buffer pool with a million pages and you want</transcript><transcript><audio_228>to update in a transaction a table that then you want to update a field for all the records in a table and that table has a million plus one Pages you can't do this because</transcript><transcript><audio_229>according to no steel Force policy every page has to be brought into memory and can't be pushed out till because of the no Steel part can be pushed out to desk till the transaction is done so this just to say that the no</transcript><transcript><audio_230>Steel part can be really problematic the force part is not trivial part too because you have to go figure out which part of that pay changed so we need to ultimately live in a world where we have to try to make the opposite of this</transcript><transcript><audio_231>happens which is to have the most high performance scheme which is to do steel no Force now as we start to make changes we'll have to keep track of what we have changed and uh work with that so one of</transcript><transcript><audio_232>the options that we will have to do and this is running out of battery for some reason even though I've got this plugged in so hold on before everything dies out</transcript> right and force means I will now Force the changes of this page to disk but because I have changes to a that was made by T1 which is not done if I'm doing a force policy I'm still going to need to figure out what's the older version of this page I need some mechanism to keep track of that and then only flush that out to disk okay so just want you to know even if it's simplest scheme that you can think of you're going to have a little complication but we'll do much better than that this is just an example no one implements it like that because this is still be a very slow system okay but there's no free lunch even with the simplest combination you still have complications all right so easiest scheme to implement uh it has a couple other problems with it one is if the number of objects that I'm writing to imagine you have a buffer pool with a million pages and you want to update in a transaction a table that then you want to update a field for all the records in a table and that table has a million plus one Pages you can't do this because according to no steel Force policy every page has to be brought into memory and can't be pushed out till because of the no Steel part can be pushed out to desk till the transaction is done so this just to say that the no Steel part can be really problematic the force part is not trivial part too because you have to go figure out which part of that pay changed so we need to ultimately live in a world where we have to try to make the opposite of this happens which is to have the most high performance scheme which is to do steel no Force now as we start to make changes we'll have to keep track of what we have changed and uh work with that so one of the options that we will have to do and this is running out of battery for some reason even though I've got this plugged in so hold on before everything dies out


<transcript><audio_233>jeez not good it's saying I'm going to shut you down in a minute but excuse me yeah it should power through it otherwise I'm going to have to take a</transcript> jeez not good it's saying I'm going to shut you down in a minute but excuse me yeah it should power through it otherwise I'm going to have to take a


<transcript><audio_234>little time and this is serious so hold on there we go now I have to switch this guy out here and then set everything back up again give me a fun that gives you a little bit of</transcript><caption><image_235>The image is a screenshot of a computer screen with a dark grey background. On the top left corner of the screen, there is a menu bar with various options such as File, Edit, View, Tools, and Help. Below the menu bar, there are two buttons - "Save" and "Cancel". On the right side of the image, there appears to be a text field with the date "Nov 13, 2020" written on it. The text field is blank, indicating that the user is in the process of editing or editing the image.</caption> little time and this is serious so hold on there we go now I have to switch this guy out here and then set everything back up again give me a fun that gives you a little bit of


<transcript><audio_236>downtime to think about all this stuff so less stuff for the final I have half a mind to record a 20 minute uh lecture and make it part of it but you guys will kill me so I won't do that I won't do that but I was thinking ways to get out</transcript><transcript><audio_237>this sorry no no no I'm nearly there I I'm fighting here guys for every minute help</transcript><transcript><audio_238>okay I think we are back let's make sure uh oh good now this thing today this connector to the external monitor is very</transcript><transcript><audio_239>finicky it seems to not want to do that okay there we go I think we are back in business and hopefully the lecture is still recording I'll record it at home and add extra material for the exam I'm just kidding won't do that that all</transcript> downtime to think about all this stuff so less stuff for the final I have half a mind to record a 20 minute uh lecture and make it part of it but you guys will kill me so I won't do that I won't do that but I was thinking ways to get out this sorry no no no I'm nearly there I I'm fighting here guys for every minute help okay I think we are back let's make sure uh oh good now this thing today this connector to the external monitor is very finicky it seems to not want to do that okay there we go I think we are back in business and hopefully the lecture is still recording I'll record it at home and add extra material for the exam I'm just kidding won't do that that all


<transcript><audio_240>right so we still have to record what we need to unwind from or make sure it made it to disk there are two techniques one is Shadow paging it's a bad idea no one does it this was the first thing that people did when they realized they need</transcript> right so we still have to record what we need to unwind from or make sure it made it to disk there are two techniques one is Shadow paging it's a bad idea no one does it this was the first thing that people did when they realized they need


<transcript><audio_241>recovery protocol so I'll describe it but I'm going to skip through the slides that are in there over here and I will refer you to uh the deck if you needed to go look at what shadow pages so remember we ran into this trouble where</transcript> recovery protocol so I'll describe it but I'm going to skip through the slides that are in there over here and I will refer you to uh the deck if you needed to go look at what shadow pages so remember we ran into this trouble where


<transcript><audio_242>two transactions that changes were on the same page and we to worry about that there's a bigger version of this problem where I've got bunch of changes that I'm making and Shadow paging effectively what we need is we need to keep track of</transcript><transcript><audio_243>a before and after version of the changes we are making okay one way to do that is to keep track of uh a scheme in which I have all the pages that I have on disk and I'll keep track in memory</transcript><transcript><audio_244>something called a master page page table kind of like your operating system those of youve taken an operating system class know that there's a page mapping table from your virtual address space to a physical address space right kind of like that but not that complicated here</transcript><transcript><audio_245>is just saying uh page one is this page this disk position uh in my stable storage okay it's just a pointer it's just a uh list of where the pages are and when the transaction comes in it will make a copy of that page table</transcript><transcript><audio_246>effectively think of it as getting a snapshot by virtue of copying this and now when it makes changes it's going to make changes if page one has to be updated it'll make a new copy of that page a shadow make a full new copy of</transcript> two transactions that changes were on the same page and we to worry about that there's a bigger version of this problem where I've got bunch of changes that I'm making and Shadow paging effectively what we need is we need to keep track of a before and after version of the changes we are making okay one way to do that is to keep track of uh a scheme in which I have all the pages that I have on disk and I'll keep track in memory something called a master page page table kind of like your operating system those of youve taken an operating system class know that there's a page mapping table from your virtual address space to a physical address space right kind of like that but not that complicated here is just saying uh page one is this page this disk position uh in my stable storage okay it's just a pointer it's just a uh list of where the pages are and when the transaction comes in it will make a copy of that page table effectively think of it as getting a snapshot by virtue of copying this and now when it makes changes it's going to make changes if page one has to be updated it'll make a new copy of that page a shadow make a full new copy of


<transcript><audio_247>itself of that page make changes just there so if some of the transaction is making changes to page one it'll make another full copy of page one uh and there ways to merge it and combine that but assume it just as one at a time and then effect L all the changes will</transcript><transcript><audio_248>create new pages those are only pointed to by my shadow page table copy and then ultimately when I'm ready to commit I will do the following which is I'll take the there's a point R memory to the root of the master page table which is pointing to the old page</transcript><transcript><audio_249>table a copy of that is also kept on disk right because that's the stable point when the dis when the database system starts from scratch it'll read that stuff because it knows kind of where everything is and then I will go FL that out over there then switch the</transcript><transcript><audio_250>pointer to the new Shadow page and now that becomes the master and eventually some background stuff goes cleans things up so this was the easiest implementation as you can imagine someone who wanted to get this type of recovery Protocol no one does that I</transcript> itself of that page make changes just there so if some of the transaction is making changes to page one it'll make another full copy of page one uh and there ways to merge it and combine that but assume it just as one at a time and then effect L all the changes will create new pages those are only pointed to by my shadow page table copy and then ultimately when I'm ready to commit I will do the following which is I'll take the there's a point R memory to the root of the master page table which is pointing to the old page table a copy of that is also kept on disk right because that's the stable point when the dis when the database system starts from scratch it'll read that stuff because it knows kind of where everything is and then I will go FL that out over there then switch the pointer to the new Shadow page and now that becomes the master and eventually some background stuff goes cleans things up so this was the easiest implementation as you can imagine someone who wanted to get this type of recovery Protocol no one does that I


<transcript><audio_251>guess there are some systems who do that where it's a bad idea the better approach is what we going to talk about with the WR ahead logging protocol obviously massive problems right huge fragmentation now you're doing this garbage collection at the page level and you'll make copies and copies of pages</transcript><transcript><audio_252>and stuff like that so lots and lots of problems we won't talk about that but as I said here you can look at last year's lecture on this which spends about 10 15 minutes on the next two slides including this I would recommend just for learning</transcript> guess there are some systems who do that where it's a bad idea the better approach is what we going to talk about with the WR ahead logging protocol obviously massive problems right huge fragmentation now you're doing this garbage collection at the page level and you'll make copies and copies of pages and stuff like that so lots and lots of problems we won't talk about that but as I said here you can look at last year's lecture on this which spends about 10 15 minutes on the next two slides including this I would recommend just for learning


<transcript><audio_253>purposes to go back and look at what sqlite used to do they had the old scheme with Shadow paging because it started in 2000 uh people hadn't advanced in all these protocols as much at that time but then they switched over to rad logging which we're going to talk</transcript><transcript><audio_254>about about next okay and so it makes copies and copies but it does copies in a slightly different way but effectively the same kind all right this is what we need to pay attention to material for exam starts back again we won't ask you questions on Shadow paging okay uh right</transcript><transcript><audio_255>ahead log is the way you implement the Bay Foundation for the recovery protocol the idea is we want to make the steel no Force the opposite of the easiest scheme the hardest scheme is steel no Force work and for that we'll</transcript> purposes to go back and look at what sqlite used to do they had the old scheme with Shadow paging because it started in 2000 uh people hadn't advanced in all these protocols as much at that time but then they switched over to rad logging which we're going to talk about about next okay and so it makes copies and copies but it does copies in a slightly different way but effectively the same kind all right this is what we need to pay attention to material for exam starts back again we won't ask you questions on Shadow paging okay uh right ahead log is the way you implement the Bay Foundation for the recovery protocol the idea is we want to make the steel no Force the opposite of the easiest scheme the hardest scheme is steel no Force work and for that we'll


<transcript><audio_256>have to keep track of what changes are being made and use those changes in two different ways so we'll create something called a log file don't confuse this with a log structured file system I'll make a comment about that in a little</transcript><transcript><audio_257>bit quick uh uh uh preview of that is even a log structured file system for the M table will do logging like this the right ahead logging okay so this is database logs and we'll keep that in a database log file which is a different file and the log</transcript><transcript><audio_258>records we're going to create log records and they'll get created in the buffer pool in pages that will eventually get flushed out to the log file okay so there's a separate file called The Log file okay and it'll go through the buffer pool to and usually there's a separate place in the buffer</transcript> have to keep track of what changes are being made and use those changes in two different ways so we'll create something called a log file don't confuse this with a log structured file system I'll make a comment about that in a little bit quick uh uh uh preview of that is even a log structured file system for the M table will do logging like this the right ahead logging okay so this is database logs and we'll keep that in a database log file which is a different file and the log records we're going to create log records and they'll get created in the buffer pool in pages that will eventually get flushed out to the log file okay so there's a separate file called The Log file okay and it'll go through the buffer pool to and usually there's a separate place in the buffer


<transcript><audio_259>pool for buffering log pages so you can think of a design space as having these two Dimensions Force yes no at transaction commit time do I for Force the changes to disk or how</transcript><transcript><audio_260>flexible is the buffer pool does is it allowed to steal Pages we want that because that makes it more efficient or not allowed to steal and we already talked about the no steel Force policy as being trivial and what we want is that desired space so no Force steel</transcript><transcript><audio_261>policy on the force aspect basically says for every update flush the updated page to disk and this means transactions are durable committed transactions changes are on dis so for committed transaction you can say I've met the D property for you okay but it'll be poor</transcript><transcript><audio_262>response time because you have to flush a lot of pages to disk for transactions that update a lot the steel policy and the force plus no steel is the easiest combination right but all no steel policies what they will say is the</transcript><transcript><audio_263>buffer manager cannot take a page away for a transaction that is still working and that works for aborted transaction because their changes never will make it to disk right but uh it'll be a low throughput because very soon the buffer</transcript> pool for buffering log pages so you can think of a design space as having these two Dimensions Force yes no at transaction commit time do I for Force the changes to disk or how flexible is the buffer pool does is it allowed to steal Pages we want that because that makes it more efficient or not allowed to steal and we already talked about the no steel Force policy as being trivial and what we want is that desired space so no Force steel policy on the force aspect basically says for every update flush the updated page to disk and this means transactions are durable committed transactions changes are on dis so for committed transaction you can say I've met the D property for you okay but it'll be poor response time because you have to flush a lot of pages to disk for transactions that update a lot the steel policy and the force plus no steel is the easiest combination right but all no steel policies what they will say is the buffer manager cannot take a page away for a transaction that is still working and that works for aborted transaction because their changes never will make it to disk right but uh it'll be a low throughput because very soon the buffer


<transcript><audio_264>manager has very few degrees of freedom to take pages and uh do the replacement policy so we really want the no force and steel policy what's the complication with the no Force so remember in no force a concern is</transcript><transcript><audio_265>that at commit time I'm not requiring the dirty Pages be flushed to this we let the lru time dictate that when a page is page is written out but what happens if a page crashes before changes make it to disk the example we started</transcript><transcript><audio_266>out with Putin zapping up the page like we have to go deal with that so what we'll do is we'll write these things called logs which are essentially diffs of changes that we made and we'll use that at recovery</transcript> manager has very few degrees of freedom to take pages and uh do the replacement policy so we really want the no force and steel policy what's the complication with the no Force so remember in no force a concern is that at commit time I'm not requiring the dirty Pages be flushed to this we let the lru time dictate that when a page is page is written out but what happens if a page crashes before changes make it to disk the example we started out with Putin zapping up the page like we have to go deal with that so what we'll do is we'll write these things called logs which are essentially diffs of changes that we made and we'll use that at recovery


<transcript><audio_267>protocol time next lecture to redo the changes that should have been in the stable storage in the first place so we need to we need logs to redo changes that we should do the steel policy see we want stealing</transcript><transcript><audio_268>allowed and this is like 2023 so probably if all of these techniques were invented now we'd probably not call it steel and force but you know those are non-pc times when these terms were invented so so pardon that but we'll stay with the terms that are in the literature so with a steel policy Our</transcript><transcript><audio_269>concern is that a page that was stolen and flush to disk may have changes that were made by an uncommitted transaction now we need to undo that so we're going to need logging to allow us to undo that so log loing is going to be used logging</transcript><transcript><audio_270>must have enough information to allow us to redo and undo and the recovery protocol that we'll talk about in the next class we decide whether when it needs a log record whether it has to redo or undo but along with that there's another</transcript> protocol time next lecture to redo the changes that should have been in the stable storage in the first place so we need to we need logs to redo changes that we should do the steel policy see we want stealing allowed and this is like 2023 so probably if all of these techniques were invented now we'd probably not call it steel and force but you know those are non-pc times when these terms were invented so so pardon that but we'll stay with the terms that are in the literature so with a steel policy Our concern is that a page that was stolen and flush to disk may have changes that were made by an uncommitted transaction now we need to undo that so we're going to need logging to allow us to undo that so log loing is going to be used logging must have enough information to allow us to redo and undo and the recovery protocol that we'll talk about in the next class we decide whether when it needs a log record whether it has to redo or undo but along with that there's another


<transcript><audio_271>fundamental protocol that we need and that is called WR ahead logging the two very foundational pieces that you need in databases one is this notion of two-phase locking because from that you got the dependence graph of which everything is based for you to</transcript><transcript><audio_272>understand how to get isolation the other piece is right ahead logging which is the protocol that says what at what point can I declare a transaction status to be committed or aborted uh there's a</transcript><transcript><audio_273>magical moment in the protocol where you say at the point where X happens that X is going to be when the commit log Proto commit log hits dis uh the transaction changes its status from active to committed it has to be one very finite</transcript> fundamental protocol that we need and that is called WR ahead logging the two very foundational pieces that you need in databases one is this notion of two-phase locking because from that you got the dependence graph of which everything is based for you to understand how to get isolation the other piece is right ahead logging which is the protocol that says what at what point can I declare a transaction status to be committed or aborted uh there's a magical moment in the protocol where you say at the point where X happens that X is going to be when the commit log Proto commit log hits dis uh the transaction changes its status from active to committed it has to be one very finite


<transcript><audio_274>boundary in that magical moment is determined by this right ahead logging protocol and what that says is that the database system is going uh to log records this stays in volatile storage and all records updated to a page are</transcript><transcript><audio_275>stay in this non-volatile buffer pool log buffer pool storage but before I write a page to stable storage I must make sure its log is written before I can write the page that's it's a uh</transcript><transcript><audio_276>right login protocol says that way I have the log information to undo and redo right so before I can write the page and overwrite it in stable storage all of its log must hit the disk and</transcript><transcript><audio_277>without that all of the stuff that we'll talk about will fail so rad login protocol says this is the way you're going to get that log information and intuitively it makes sense right I if I wrote the page and then crashed before the log hit it I don't know how to redo</transcript><transcript><audio_278>and un do the things that I may need to okay so record your changes record take that record of changes put it in someplace safe that is stable before you can make the page the the changes uh uh push out to stable</transcript> boundary in that magical moment is determined by this right ahead logging protocol and what that says is that the database system is going uh to log records this stays in volatile storage and all records updated to a page are stay in this non-volatile buffer pool log buffer pool storage but before I write a page to stable storage I must make sure its log is written before I can write the page that's it's a uh right login protocol says that way I have the log information to undo and redo right so before I can write the page and overwrite it in stable storage all of its log must hit the disk and without that all of the stuff that we'll talk about will fail so rad login protocol says this is the way you're going to get that log information and intuitively it makes sense right I if I wrote the page and then crashed before the log hit it I don't know how to redo and un do the things that I may need to okay so record your changes record take that record of changes put it in someplace safe that is stable before you can make the page the the changes uh uh push out to stable


<transcript><audio_279>storage table storage yes log so there are two things happening I've got a table in which I'm making changes a bunch of tables in the database all of those changes we're going to record in a log that is sitting in the buffer pole because we also don't want the log to be written to disc every time that's slow</transcript><transcript><audio_280>but in that buffer pool we'll flush the buffer pool the logs portion of the buffer pool to dis so if I'm page 13 and the buffer manager says I want to EV page 13 first it will say to the log</transcript><transcript><audio_281>manager which is managing the log buffer pool different buffer pool right may come from the same memory space but it's a different manager and say hey log manager flush all your logs for page 13 and tell me when you're done the log</transcript><transcript><audio_282>manager will flush it to the log file it'll all the rights to this log file are going to be sequential and then basically come back and now the buffer manager can go and evic page 13 okay</transcript> storage table storage yes log so there are two things happening I've got a table in which I'm making changes a bunch of tables in the database all of those changes we're going to record in a log that is sitting in the buffer pole because we also don't want the log to be written to disc every time that's slow but in that buffer pool we'll flush the buffer pool the logs portion of the buffer pool to dis so if I'm page 13 and the buffer manager says I want to EV page 13 first it will say to the log manager which is managing the log buffer pool different buffer pool right may come from the same memory space but it's a different manager and say hey log manager flush all your logs for page 13 and tell me when you're done the log manager will flush it to the log file it'll all the rights to this log file are going to be sequential and then basically come back and now the buffer manager can go and evic page 13 okay


<transcript><audio_283>so yeah we'll see the logs the structure of that log so that's coming what's in the logs and what that looks like is coming okay but basically just think about it that way when a buffer poool a dirty page only on a dirty page if it's a clean page you don't have to talk to</transcript><transcript><audio_284>the log manager you as the buffer man buffer pool manager but the buffer pool manager on evicting a dirty page has to First tell the log manager please flush everything because the rad protocol requires you to do that before you can</transcript><transcript><audio_285>say all records pering to an updated page are written to nonvolatile storage nonv nonvolatile is stable yeah the dis and ssds yeah I know the textbook is nonvolatile previously people used to call it stable before that they used to</transcript><transcript><audio_286>call it dis so terms are changing like every few years because the storage hierarchy is changing yeah so nonvolatile is a good stuff that will not get zaap okay all right so the slide is</transcript> so yeah we'll see the logs the structure of that log so that's coming what's in the logs and what that looks like is coming okay but basically just think about it that way when a buffer poool a dirty page only on a dirty page if it's a clean page you don't have to talk to the log manager you as the buffer man buffer pool manager but the buffer pool manager on evicting a dirty page has to First tell the log manager please flush everything because the rad protocol requires you to do that before you can say all records pering to an updated page are written to nonvolatile storage nonv nonvolatile is stable yeah the dis and ssds yeah I know the textbook is nonvolatile previously people used to call it stable before that they used to call it dis so terms are changing like every few years because the storage hierarchy is changing yeah so nonvolatile is a good stuff that will not get zaap okay all right so the slide is


<transcript><audio_287>correct great so besides this we're going to do a couple more things we're going to write a special type of a log record called a begin log record and when a transaction finishes with a commit status we'll write a commit log record there's also an abot log record we'll see that in the next class and</transcript><transcript><audio_288>then again this says before I commit the transaction this magical moment for a commit right protocol has two components one is eviction of a dirty page please flush your logs before you flush the page and the other thing it says when</transcript><transcript><audio_289>you commit a transaction that commit log record must be created written in the log uh buffer pool and that log page page must be flushed to dis into this</transcript><transcript><audio_290>non-volatile storage when you get the signal back that the page has been written correctly that's the magical moment when the commit happens and now you can tell the world the transactions come in so right here logging protocol will have those two components dirty</transcript> correct great so besides this we're going to do a couple more things we're going to write a special type of a log record called a begin log record and when a transaction finishes with a commit status we'll write a commit log record there's also an abot log record we'll see that in the next class and then again this says before I commit the transaction this magical moment for a commit right protocol has two components one is eviction of a dirty page please flush your logs before you flush the page and the other thing it says when you commit a transaction that commit log record must be created written in the log uh buffer pool and that log page page must be flushed to dis into this non-volatile storage when you get the signal back that the page has been written correctly that's the magical moment when the commit happens and now you can tell the world the transactions come in so right here logging protocol will have those two components dirty


<transcript><audio_291>Pages essentially it says the when a log of the change you're trying to make hits stable storage at that point we can declare that we now have a mechanism to unwind ourselves out from any changes that we need to move out to or reapply changes as we need to because logs will</transcript><transcript><audio_292>have the redo and undo information okay all right questions I want to make sure I get this part because this is the foundation for understanding everything today and the rest of the next lecture I I why</transcript> Pages essentially it says the when a log of the change you're trying to make hits stable storage at that point we can declare that we now have a mechanism to unwind ourselves out from any changes that we need to move out to or reapply changes as we need to because logs will have the redo and undo information okay all right questions I want to make sure I get this part because this is the foundation for understanding everything today and the rest of the next lecture I I why


<transcript><audio_293>we do we the B record is is just to say this is the mark of it and you'll see when we get to checkpointing it will help us figure out what's the boundary of that otherwise if you have to scan a big log file the log file can have millions of billions of Records you'll say where did</transcript><transcript><audio_294>transaction T1 start you have to scan everything this tells you oh I don't need to go any further if I'm worried about transaction T so you want that all right so what's in this log</transcript><transcript><audio_295>records I'm going to simplify it at a big uh dramatic level there's a whole set of papers on what to put in the log records but they'll all pretty much have this in the different type of log records too besides B commit and the the ones I'm telling you again we're going to ignore that there's a big fat</transcript> we do we the B record is is just to say this is the mark of it and you'll see when we get to checkpointing it will help us figure out what's the boundary of that otherwise if you have to scan a big log file the log file can have millions of billions of Records you'll say where did transaction T1 start you have to scan everything this tells you oh I don't need to go any further if I'm worried about transaction T so you want that all right so what's in this log records I'm going to simplify it at a big uh dramatic level there's a whole set of papers on what to put in the log records but they'll all pretty much have this in the different type of log records too besides B commit and the the ones I'm telling you again we're going to ignore that there's a big fat


<transcript><audio_296>textbook that can come that Jim gry wrote uh on all the details you'd ever want to know about transactions and logging borrow it from me if you're interested in that for this course we're going to uh assume logs have this common</transcript><transcript><audio_297>structure they'll record which transaction do I belong to what object am I logging record and we're just going to assume it's record level for this class and then what type of information am I logging is it something that I use</transcript> textbook that can come that Jim gry wrote uh on all the details you'd ever want to know about transactions and logging borrow it from me if you're interested in that for this course we're going to uh assume logs have this common structure they'll record which transaction do I belong to what object am I logging record and we're just going to assume it's record level for this class and then what type of information am I logging is it something that I use


<transcript><audio_298>for undo or for redo and obviously the undo stuff which means give me my old value is something you wouldn't do if you're working with an mbcc based storage system okay so little connection across that but rest of it again we're going to work with a single version</transcript><transcript><audio_299>system to keep all the Powerpoints uh sensible so right ahead logging example start with a transaction now as you can see the buffer pool has a buffer pool for your pages and a little additional thing called the right ahead uh buffer</transcript><transcript><audio_300>so it's a buffer pool like your page oriented buffer pool probably much smaller U and it's things from that get evicted from top to end it's a the log as you'll see is a linear sequence of log records they're ordered and so</transcript> for undo or for redo and obviously the undo stuff which means give me my old value is something you wouldn't do if you're working with an mbcc based storage system okay so little connection across that but rest of it again we're going to work with a single version system to keep all the Powerpoints uh sensible so right ahead logging example start with a transaction now as you can see the buffer pool has a buffer pool for your pages and a little additional thing called the right ahead uh buffer so it's a buffer pool like your page oriented buffer pool probably much smaller U and it's things from that get evicted from top to end it's a the log as you'll see is a linear sequence of log records they're ordered and so


<transcript><audio_301>effectively the right ahead buffer pool is typically pretty small it might just have a small number of pages so you need at least two but it might have a few hundred and everything's going to get flushed from top to bottom and that'll become more clear when we talk about the</transcript><transcript><audio_302>recovery protocol in the next class okay so it's a sequential file little bit different than the other files but it has a buffer pool so that when log records are created they get created in the buffer pool so transaction T1 started has a begin now T1 is getting</transcript> effectively the right ahead buffer pool is typically pretty small it might just have a small number of pages so you need at least two but it might have a few hundred and everything's going to get flushed from top to bottom and that'll become more clear when we talk about the recovery protocol in the next class okay so it's a sequential file little bit different than the other files but it has a buffer pool so that when log records are created they get created in the buffer pool so transaction T1 started has a begin now T1 is getting


<transcript><audio_303>written so that transaction is recording your old value was one new value is eight uh sorry old value is eight new value is one and so that's why that record has t a one and eight because eight was the old value so now in the log record I'm keeping track of my old</transcript><transcript><audio_304>and new value okay and you can as we just talked about you can optimize that stuff if you're in mbcc uh and now B got written again I've got old and new value when I write when the transaction is ready to commit even</transcript> written so that transaction is recording your old value was one new value is eight uh sorry old value is eight new value is one and so that's why that record has t a one and eight because eight was the old value so now in the log record I'm keeping track of my old and new value okay and you can as we just talked about you can optimize that stuff if you're in mbcc uh and now B got written again I've got old and new value when I write when the transaction is ready to commit even


<transcript><audio_305>though that log buffer pool page is not full for that right ahead blog file I will flush that half full page to disk before I can declare commit when that page comes back with a okay the page was written the</transcript> though that log buffer pool page is not full for that right ahead blog file I will flush that half full page to disk before I can declare commit when that page comes back with a okay the page was written the


<transcript><audio_306>transaction is declared committed okay now obviously the scheme at commit time we've optimized a lot of things if this transaction were touching a million objects then it may have just created if it were touching just one bite in the million objects it may have just created</transcript><transcript><audio_307>a few log Pages as opposed to having to write million data pages to this that's why you can see why this is faster but it's still slow because at commit time I still have to wait for that disco to</transcript><transcript><audio_308>come back and those are as you know many tens of milliseconds right uh and so slow and if the now at that comit time</transcript><transcript><audio_309>if the buffer pool Get Zapped out the me Dam Gets Zapped out which means I've lost both buffer pool I can reconstruct from that log file but it's still slow because for each commit I have to wait for the disc to finish now imagine I've got 100 transactions running they all</transcript><transcript><audio_310>are ready to commit I the first transaction will come it wait for that page to come back the other transaction cannot come till the page comes back so now you're blocking transactions from tens of milliseconds which is an eternity you still have a pretty slow system much</transcript> transaction is declared committed okay now obviously the scheme at commit time we've optimized a lot of things if this transaction were touching a million objects then it may have just created if it were touching just one bite in the million objects it may have just created a few log Pages as opposed to having to write million data pages to this that's why you can see why this is faster but it's still slow because at commit time I still have to wait for that disco to come back and those are as you know many tens of milliseconds right uh and so slow and if the now at that comit time if the buffer pool Get Zapped out the me Dam Gets Zapped out which means I've lost both buffer pool I can reconstruct from that log file but it's still slow because for each commit I have to wait for the disc to finish now imagine I've got 100 transactions running they all are ready to commit I the first transaction will come it wait for that page to come back the other transaction cannot come till the page comes back so now you're blocking transactions from tens of milliseconds which is an eternity you still have a pretty slow system much


<transcript><audio_311>better than the no steel force uh policy that we had but can we make it better so most systems for high performance will do something called group commit which is a very simple idea is that I'm going</transcript><transcript><audio_312>to batch up the Comets that are coming together I'm going to pick an interval like 5 milliseconds or something like that is pretty common and even though a transaction is ready to commit like T1 comes in creates its stuff in the uh buffer pool here it's in the right ahead</transcript><transcript><audio_313>buffer pool now I'm not showing the data buffer pool right remainder of this uh class will only care about the right ahead log buffer pole okay or the log manager buffer pole this comes in starts to go and that</transcript><transcript><audio_314>page gets flushed to disk and that's fine because maybe we need that page we start writing to that new page and one of the transactions is ready to commit as you can see the other other transaction is going to commit very shortly we don't know that but with the</transcript><transcript><audio_315>group commit what you'll do is you'll say I is a transaction and ready to commit but you know what I'm going to hold off for five milliseconds whatever is the timeout and wait for anyone else who wants to write a commit log record to everything that's in the log buffer</transcript> better than the no steel force uh policy that we had but can we make it better so most systems for high performance will do something called group commit which is a very simple idea is that I'm going to batch up the Comets that are coming together I'm going to pick an interval like 5 milliseconds or something like that is pretty common and even though a transaction is ready to commit like T1 comes in creates its stuff in the uh buffer pool here it's in the right ahead buffer pool now I'm not showing the data buffer pool right remainder of this uh class will only care about the right ahead log buffer pole okay or the log manager buffer pole this comes in starts to go and that page gets flushed to disk and that's fine because maybe we need that page we start writing to that new page and one of the transactions is ready to commit as you can see the other other transaction is going to commit very shortly we don't know that but with the group commit what you'll do is you'll say I is a transaction and ready to commit but you know what I'm going to hold off for five milliseconds whatever is the timeout and wait for anyone else who wants to write a commit log record to everything that's in the log buffer


<transcript><audio_316>pole okay and then every 5 milliseconds I'm going to take everyone that is waiting to be committed because they've declared I've thrown my commit flag tell me when I'm done you collect all of their log records and then you'll flush that out to disk and so effectively when</transcript><transcript><audio_317>you write that page everything that you've written all the transactions that were ready to come with they'll wait an average of if 5 milliseconds is your timeout interval they'll wait an average of 2.5 milliseconds but what you'll get</transcript><transcript><audio_318>is a very a much higher throughput system though you've added late a little bit of latency to each transaction on average 2.5 in this case but the throughput of the system will be much higher because one disio will commit a whole bunch of transactions if lots of transaction systems are active maybe you</transcript><transcript><audio_319>got a 100 transactions that were ready to commit in that 5 millisecond period so you'll get a much higher throughput system than you have so pretty much everyone does some form of group commit to do that some might even play around with a little bit of tricks like oh that</transcript><transcript><audio_320>uh I won't even wait they will go even further they'll say I will write this out to disk I've initiated the dis IO I won't even wait for the io to come back and tell you it's committed and you can read the manuals and sometimes you'll see that in all the database fers like</transcript><transcript><audio_321>oh you could set it up that way which means it's like what's your tolerance to that failure in that short amount of time while that IO happens okay there are options like that you will see a couple more slides and then I</transcript> pole okay and then every 5 milliseconds I'm going to take everyone that is waiting to be committed because they've declared I've thrown my commit flag tell me when I'm done you collect all of their log records and then you'll flush that out to disk and so effectively when you write that page everything that you've written all the transactions that were ready to come with they'll wait an average of if 5 milliseconds is your timeout interval they'll wait an average of 2.5 milliseconds but what you'll get is a very a much higher throughput system though you've added late a little bit of latency to each transaction on average 2.5 in this case but the throughput of the system will be much higher because one disio will commit a whole bunch of transactions if lots of transaction systems are active maybe you got a 100 transactions that were ready to commit in that 5 millisecond period so you'll get a much higher throughput system than you have so pretty much everyone does some form of group commit to do that some might even play around with a little bit of tricks like oh that uh I won't even wait they will go even further they'll say I will write this out to disk I've initiated the dis IO I won't even wait for the io to come back and tell you it's committed and you can read the manuals and sometimes you'll see that in all the database fers like oh you could set it up that way which means it's like what's your tolerance to that failure in that short amount of time while that IO happens okay there are options like that you will see a couple more slides and then I


<transcript><audio_322>promise I will stop the logging schemes are I want to finish the logging schemes and then we'll stop uh there are three different types of logging schemes let's just go through that the first one is physical logging which is to say at the what I'm going to record is effectively</transcript><transcript><audio_323>like a g diff I'm going to record in the log the before and after image of what was changed now usually that is done uh at the value level but as you can imagine this diff can start to become</transcript><transcript><audio_324>really large now the challenge with this scheme is that if I'm doing this imagine I've got a page and that page is a slotted page structure and Records have records can move around in a slotted page structure while still keeping the</transcript><transcript><audio_325>slot ID right you might have compaction happening inside the page for example this before this physical logging where I'm keeping before and after images might record changes at the page level of stuff that is not even Changed by this transaction because it was just the</transcript> promise I will stop the logging schemes are I want to finish the logging schemes and then we'll stop uh there are three different types of logging schemes let's just go through that the first one is physical logging which is to say at the what I'm going to record is effectively like a g diff I'm going to record in the log the before and after image of what was changed now usually that is done uh at the value level but as you can imagine this diff can start to become really large now the challenge with this scheme is that if I'm doing this imagine I've got a page and that page is a slotted page structure and Records have records can move around in a slotted page structure while still keeping the slot ID right you might have compaction happening inside the page for example this before this physical logging where I'm keeping before and after images might record changes at the page level of stuff that is not even Changed by this transaction because it was just the


<transcript><audio_326>before and after images uh uh might have physical it's just physically looking at the bites that have been changed on the page before and after and might get a whole bunch of stuff that is irrelevant to the actual changes which is a cue to the physiological logging that is coming next there's the Other Extreme which is</transcript><transcript><audio_327>to say logical locking I'm just going to record the data function that caused this value to change in a very simple form it is recording the query more often it's going to be I incremented the value by 10 right if that was an update</transcript><transcript><audio_328>like that if the update query said A+ 10 this would actually record it's I'm an increment function and my parameter was 10 to this record so it logically record what was what was changed uh uh or it might say I updated 10 for everything</transcript><transcript><audio_329>that has a predicate of B greater than 10 effectively take whatever is in the query find some representation for that and represent that in The Logical structure in the log</transcript><transcript><audio_330>now you can get a much more compact log uh record but when you have to go and apply these changes you have to actually go run that query again to redo it and if that query took an hour to run while applying the log you're going to have to</transcript> before and after images uh uh might have physical it's just physically looking at the bites that have been changed on the page before and after and might get a whole bunch of stuff that is irrelevant to the actual changes which is a cue to the physiological logging that is coming next there's the Other Extreme which is to say logical locking I'm just going to record the data function that caused this value to change in a very simple form it is recording the query more often it's going to be I incremented the value by 10 right if that was an update like that if the update query said A+ 10 this would actually record it's I'm an increment function and my parameter was 10 to this record so it logically record what was what was changed uh uh or it might say I updated 10 for everything that has a predicate of B greater than 10 effectively take whatever is in the query find some representation for that and represent that in The Logical structure in the log now you can get a much more compact log uh record but when you have to go and apply these changes you have to actually go run that query again to redo it and if that query took an hour to run while applying the log you're going to have to