In the last video we left off, scoring our machine learning model that we fitted to the data after we just filled all the numerical values, filled all the categorical values and turned all our data into numbers. Model worked, our data had no missing values and it was all numeric. But we left off with the question, why isn't this metric reliable? Let's put that here, because that's worth highlighting. Because remember, always remember, fitting a machine learning model or evaluating a machine learning model is just as important as fitting one. And that's where we're up to now because we just fit a machine learning model. We have to make sure that our results hold water, right? Because we don't want to be promising things that our model can't do. So question, why doesn't the above metric hold water? Hold water is an expression for why isn't the metric reliable? That's just a way of speaking. So why doesn't it hold water? Why isn't it reliable? Have you had a think about it? I'll show you a diagram. That may refresh your memory. If we come back to our keynote, you remember from right back at the beginning, the most important concept of machine learning, OK, the three sets. So what we've done is we've trained a machine learning model on one data set. But then what have we done? We come back to the code. We've fit the model on the data set on some data. But then we've evaluated it on the exact same data. OK, so if we come back here, essentially what we've done is if we were thinking about this has been a university course, we've learned the course materials. But what we've done instead of testing our model on the equivalent of like a final exam, AKA a test data set, we've just evaluated our machine learning model on the exact same materials that it's learned on. You were in one class and you got given a book to read and then at the exact same time, it's just like you're going to be tested with the questions directly from that book you're reading. Hmm, OK, but we are after our model's ability to generalize and OK, the ability for a machine learning model to perform well on data. It hasn't seen before. That's what we're after. If we go back to Kaggle, there's a training data set, which contains data through the end of 2011. There's a validation set and there's a test set. So that's what we're after. We need to evaluate our model not on the data we've trained it on, which is this. We need to evaluate it on the test data. But before we do that, if you remember, right back up the top, let's go right back up the top to we imported our first data frame. I said we'd revisit this and now the time has come. So we go here. We imported train and valid.csv. Hmm, wonder why did we do that? Well, it's all going to be come clear in this video. So we got train and valid csv. We got valid csv and we got train csv. Why don't we just import them separately? Well, I wanted to demonstrate what it's like creating your own validation set, rather than someone else creating it for you, with a time series data set, which is what we're working on. So this is a perfect playground for that. So if we read here, train csv is the training set. So basically what we've been working on and valid csv is a validation set. The key point here is because it's a time series, the training data set has data up to the end of 2011, whereas the validation set has data between January 1, 2012 to April 30, 2012. So what we will do to better evaluate our model, we need to split our data up into training and validation sets. So let's do that. Let's go here, splitting data into train validation sets. Because that's what we're after now. We've got a model we can build models now, but we need to rather than just build better models, we need to make sure that what we're evaluating with makes sense. So let's check our data again, as we always do. So maybe we can use the year. What was it called sale year? So do you have temp sale year? There we go. Okay, so maybe because our data frame is in order, yes, this is going to help us. When we import our data frame, we ordered it in the order of the sale dates. So now reading this to create our own validation set, we want to split our data. So all of the rows up to 2011. So the sale year is 2011, that can be in the training set. And then all of the rows in 2012 can be the validation set. We won't worry about the test set for now because that's in a separate data set. The reason why we're only working on train and valid CSV is because those, we imported the train and valid.csv at the start of a notebook, this file here. So we have to create our own validation data set. So let's do that. So we go, do you have template? Check the sale year.value counts. There we go. 2012. So there's 11,573 samples in 2012. All right. So to split our data into training and validation, it should be as easy as going, okay, let's introspect the sale year column. And every column that's equal to 2012 will be a validation set for sale year. And example, and every row that's not equal of sale year, that's not equal to 2012 will be in the training set. So let's stop talking about it. And you'll let's see the code. So split data into training and validation. This is something that you might have to do with your own time series data, right? Because you're not always going to be given in a Kaggle format, right? When you're working with a client or working on a project, you're not always going to automatically have your data in train, valid and test. These are things you're going to have to create for yourself. Looking at the sale column or looking at the time column, looking at the date column and figuring out how you can make your own training and validation set. So that is exactly why we imported them as one set to begin with. So we could practice making our own training and validation sets. And I feel like I'm saying the word set a lot. But that's important because we go back to our keynote. This is the most important concept in machine learning because whatever we train our model on, we want to make sure we're evaluating it on something else. So come here. So the validation set is every row in DF temp where the sale year column equals 2012. Yes, that's correct. And the the DF training set is every row in DF temp where the sale year column is not equal to 2012. Wonderful. And then we might go learn DF valve and learn DF train. So all this is going to tell us is the length of these two data frames. But beautiful. So now we have a validation set, which contains 11,573 rows and a training data set, which contains 401,125 rows. Or samples and they're split on date. Beautiful. We're taking boxes here. We are taking boxes here. So now what we might do is split data into X and Y. We've seen this before. And so that way we have an X training set and a Y training set and a X valid set and a Y valid set, which is our data and labels. So we'll have X train Y train equals DF. Now we're working with the training set here. Drop. We want to drop the sale price column on the first axis. Okay, I just dropped the column. And we're going to set it to DF train Y train is going to be equal to sale price. So see how I'm doing it with a comma here. I'm being a bit tricky. If I just start doing it in one line, beautiful. And we're going here valid Y valid. Not sure. And DF valve. Drop sale price just the same again, but this time with the validation set, DF valve dot. This is actually going to be sale price. There we go. And now we'll have a little geese. It's back to our data just to make sure we haven't made a little error somewhere. These should all be comparable shapes to each other. Y valid dot shape. So we're just taking these datasets that we're creating here and we're finding out the shapes of them. Beautiful. So our training set is 401,000 rows or about that with 102 features, 102 columns. Y training is 401,000 rows. And then validation is about 11 and a half thousand with the same amount of columns for X and no columns for Y because Y is just one column. Let's have a look. Now look, this is what we do. Make sure all of our data. Okay, so these are all sale price. Beautiful. Woo. Okay. So we've got our data into train evaluation sets. We've taken care of two of the three most important things or most important sets. We'll have a look at the test set later, but this is what we've created. Training set and a validation set. It's time to keep building some more models. So we might end this video here. We've got to figure out a way to evaluate our machine learning model. So we'll probably have a look at that in the next video.