We're probably missing an activation function for our output layer. However, I hope you didn't take my word for it


Learn TensorFlow and Deep Learning fundamentals with Python (code-first introduction) Part 2/2

I hope you tried to write the code yourself, but if not, let's do it


Let's model this circle once and for all i'm getting sick of seeing these uh, these straight yellow lines. So let's um, set the random seed


As usual, you can probably tell that my uh coding hands are eager to write this neural network and get this


Circle modelled so let's do step one create a model just as we have before I believe we're up to model seven


Fingers crossed this is lucky model seven, you know


And we're going to go sequential


Go there


tf


Caras layers, it's going to be the exact same hidden layers as what we've been using before


the activation equals


Relative to the input layer


As what we've been using before the activation equals relu


wonderful


Come back tf caras layers dense four as well activation equals


value


tf caras


dot layers


Dot dense we want one for the output layer because we're dealing with a binary classification one thing or another


And here is where we're going to introduce the magical we could do tf caras and activations


dot sigmoid


or


If we wanted to keep in line with the string notation that we've been using we could just turn this into sigmoid


Now again, where does this come from? We could ask google or we could check out the typical architecture of a classification model


So binary classification


Hidden activation usually relu or the output activation is sigmoid for binary


When we deal with multi-class, we'll have to deal with softmax. So keep that in mind going forward now, we'll come back


All right. Our model architecture is looking great. Let's compile it


compile the model


model 7


dot compile


Now we want to go loss equals tf caras losses dot. No, we're using string notation. Come on daniel


Let's go binary cross entropy


Wonderful, and then optimizer can be tf caras


optimizers


Atom we'll set the learning rate to be 0.01


Again, that's the default. We don't necessarily need to do that. But just keeping in line with


staying true to our tensorflow playground setup


And we will go metrics equals


accuracy


Wonderful now


Let's fit the model


Oh, i'm eagerly awaiting


To see if the results for this one are better than our previous model. This is the fun part of uh,


Neural networks in deep learning is running so many different modeling experiments


Epochs equals one. Oh, let's watch the training


I was about to set verbose equal to zero


But let's watch it if this neural network is going to work. I want to see those metrics going where they should go


So accuracy it starts are we going up? Yes. Oh my gosh


Look at that


We finished with an accuracy


Before we we added that output activation function


We were getting about a 50 accuracy and now we're borderline 99


But again, let's not trust just the metrics. Let's try number four is evaluate


our model


So model lucky number seven, that's where we're at


Evaluate x and y are we getting the same?


Wow, we are loss


below 0.3 and accuracy is 99


How does that line up with our tensorflow playground?


So that loss is getting almost 10 times lower, but this is fit for


2000 epochs maybe ours would somewhat approach that if we kept training


But again, let's not just trust the metrics. Let's visualize let's


Visualize our incredible


Metrics, so again, we created this beautiful function a few videos ago


Because we want to use it multiple times plot decision boundary


Model number seven x y are you ready?


three


two


one


Oh


my


goodness


How much better is that?


So it looks like our model has basically perfectly found the decision boundary between the red and blue dots


Except for maybe a couple of points. Oh that one that one got caught


And that one got caught there. So that's why we're not getting a perfect result, but we're very close. That's


99 accuracy between two evenly spread classes or two evenly spread labels


That's pretty darn good


But I have a question for you. I want to put this down here


And we've we've discussed this


Previously, but i'm going to put the question emoji


It's a little challenge. So question


equals


Or equals i'm too used to saying equals what's wrong with the predictions we've made?


Are we really


Evaluating so if we're looking at this our evaluation metric and our plotting of predictions


Are we really?


Evaluating our model correctly


Here's a little hint


What data?


Did the model learn on and what data did we predict on?


Now have a think about that you probably know the answer already if not perfectly fine, but before we we answer that


I want to emphasize


What we've just covered


Which is back to the question we had before


If we go to our non linearity slide


I posed this question a couple of videos ago


What could you draw if you had an unlimited amount of straight linear and non straight non-linear lines?


And then we looked at our linear data


and we looked at


our non-linear data


so the combination of linear straight lines and non-straight lines if we go back here


So non-linear functions remember we haven't even discussed what relu is or what sigmoid is


We just know that it's not linear


So we come back


Linear and these three are not linear. That's all we've discussed for now


But this is a i'm going to even write this down


So let me put a key here


If you want to take away


Anything from the few videos we've just gone through


Is that the combination of


linear which is straight lines


and


non-linear


Which is non straight lines. That's all you need to know for now


functions


Is one of the key fundamentals of neural networks


And notice key is on purpose because i've got key there


Now back to our question


Just think of it like this if I gave you an unlimited amount of straight lines and


non-straight or non-linear lines


You could essentially draw any pattern that you wanted to


Now that is essentially what our neural networks are doing


This data set is relatively easy


But if you imagine if we're working on a whole bunch of different other data sets such as building a neural network to understand


What's in a picture?


There's almost an unlimited amount of things look up if we go food


Images


If we want to build a neural network to identify different patterns in pictures of food. Look how many different patterns there are here


We need a whole bunch of non-straight and


straight lines


So that's the essence of what a neural network is doing when it looks at different


Examples of data it's drawing patterns with straight and non-straight lines


With straight and non-straight lines through it


So if this doesn't really make sense for now, you might even be thinking hey daniel


I've never actually seen a linear function or a non-linear function before


Well, you kind of have we've been using the whole time


They're what power


The layers we've just built


So with that being said


In the next video, let's take a look at applying these linear and non-linear functions


That we've been using in our neural networks, but just applying them on their own


So i'll see you there


In the previous video we modeled our


Non-linear classification data once and for all I mean look at that. That's a beautiful sight, isn't it?


Our decision boundary is basically splitting them perfectly


now


We've also discussed the concept of linear straight lines and non-linear non-straight lines or functions


But we haven't really built up an intuition about them. We've just we've just got familiar with the names of some of them


We've looked at relu and we've looked at sigmoid, but let's start to build up an intuition of them


so


right here now we've


discussed


the concept of linear and non-linear


functions


lines


Let's see them in action


So to do so, how about we create a toy tensor?


Which is very similar to the data we pass into our models


because


We're dealing with tensorflow all of the data we use gets encoded into a tensor


We pass that tensor to a neural network. It figures out patterns and outputs another tensor. So let's go a equals


tfcast


I'm going to go tfrange. I want just a tensor a nice one from negative 10 to positive 10


And I want it to be offload 32


Let's have a look


Okay


Nice and simple just negative 10 all the way up to 9 because it includes 0


But it's 20 20 long


Now let's have a look at what it looks like


visualize our


toy tensor


plt.plot a


Wonderful nice and straight line. What would that be called? What's a straight line? It's a linear line, right?


now


What activation function did we just try?


in our output layer


The sigmoid


Okay, so how might we apply this sigmoid activation function directly to tensor a


I've got an idea. Why don't we look that up tensorflow?


sigmoid


activation function


tfkeras activation sigmoid, okay


wonderful


Oh the sigmoid activation function sigmoid x equals 1 divided by 1 plus exponent


To negative x. Okay. What if we just did sigmoid activation function? What does that look like?


We're going to get some fancy math notation. I'm guessing. Yeah, there we go. Okay, so sigmoid z equals 1 on


1 plus e to the power of negative z and we get a line like that. Oh, so see how that's non-linear


All right enough looking at pictures let's try to replicate this we could do it tfkeras activation sigmoid, but how about


We start off by just replicating this


So what I would do


Uh in this in a case scenario like this where i'm trying to replicate something we go


let's start by


replicating sigmoid


And I put this here


So i'm trying to build a function that replicates this here


I was just trying to point at the screen, but I realized you can't see that


And so let's go def sigmoid and it's going to take an input x


And then how about we we just it's relatively it's only


this one divided by


um one plus


Now to get the exponent


We let's look that up. Can we do it in tensorflow tensorflow?


exponent


tf math exp


Wonderful, does it have an alias? Oh tf


x


All right exp


So if we go


tf dot exp


Pass in negative x


Is that our sigmoid function?


Really? That's all it is


So let's try it out use the sigmoid


function on our


toy tensor


So we created a up here before remember this is what a looks like right now. It's linear


Now if we use our sigmoid


sigmoid on a


All right, we get a whole bunch of what looks like gibberish numbers


But let's plot them. That's a much better way to look at them. And how do we do that?


plot our a


our toy tensor


transformed by sigmoid


So plt dot plot


We want to go sigmoid


a


And what do you think this will look like give you a second to think about it


Let's check


Ho ho


That's what's up


Now notice here that the values are between zero and one. Hmm


That might be something we look at later on but the most important point here is that this line


was originally straight


And has now been modified to be non-straight


Now where is the intuition there if we come back


Remember how we couldn't draw a curved line around our data, but as soon as we added


this sigmoid activation function to our neural network it was like


Now that we've seen what it does


To a straight line. It was like we gave our neural network a tool to go


Hey, you've been trying to draw patterns with just straight lines before but now you've got this non-straight line


So use that as best you can to find better patterns


And as it turns out it was able to find better patterns


So let's not stop there. Let's keep going. What was the other activation function that we've used so far?


We'll come up here


We've used relu activation. All right. So let's see. How can we uh replicate this?


Because that looks pretty cool. I wonder if we can do the same for relu. Let's go. Um another tab


tensorflow


relu function


And what do we get?


And what do we get tfkeras activations relu. All right


Well, it looks like we can do very similar notation to our sigmoid function


But does it have a definition of what relu is? Oh, it's just the max between x and zero


Oh


So does that mean it's going to make all negative numbers zero?


Hmm or we could even just go what does relu do?


In a neural network the activation function is responsible for transforming the sum


Weighted input from the node into the activation of the node or the output for that input


hmm


We could dive into that


relu is the max function x zero


What if we go images? Is there a function of it?


Equals zero for x less than zero or x for x


Equal or greater than zero. Okay. Why don't we just replicate that max zero or z?


Okay, let's give that a go let's


recreate


The relu function and see what happens. Oh


one thing before we do


Notice what this line looks like


How does it look compared to our original tensor?


Our toy tensor that we created above


So maybe we go def relu


And


We want to return


How do we get the maximum in tensor flow? We can just do max. Is it maximum or just max maximum?


Wonderful, we want to return the maximum between zero and x


That's all we've done for relu


If we come here we want to go or


Is there a wikipedia page wikipedia is usually pretty good


There we go rectifier neural network


This is all we've written


relu equals max zero x


That's it. That's all we've done


Okay. Now let's pass


our toy tensor


to our


custom relu function


relu


a and what's going to happen? What do you think's going to happen?


Well, let's check it out. All right. How is this different to our original tensor?


Just by looking at it. This one's a bit easier to look at than the sigmoid


But see how these are all zero. It seems like relu all it's done


Is it's turned the negative numbers?


to zero


And that we've done that with the maximum function by just going hey, look at the input you're getting


And give me the maximum of of zero


Or or the number itself. So is zero larger than negative one?


Yes, it is. So we set it to zero. All right now better still let's see how this looks so


plot relu


modified


Tensor let me go here plot plot


a


And let's see


Ha ha


Now what is this?


Ha ha now, what is that?


We come back. Well, let's just look at bring our straight line tensor down here


So this it started out straight


And now this is you could still argue that this is straight but


It's got a kink in it. So now we have a bunch of tools. We've given our neural network with hey, here's this curvy line


And here's this bendy line


Now you've got these two tools


Let's start drawing patterns


in our data


I mean I could draw some pretty cool shapes if I had this curvy line and this bendy line


Now let's not stop there. We've got one more activation function that we've tried remember


We're now in our neural network playground. The first thing we tried was linear


And then we tried non-linear. So let's let's do that. Let's uh for completeness. Let's see


how


Let's try the linear


activation function


so


Might just do what we did before tensor flow linear


Tf carers activations linear linear activation function pass through what does it do?


Arguments x the input tensor returns the input


unmodified


Are you serious?


We just put a tensor into the linear activation function and it just returns the same tensor


So this is a linear function and this is a non-linear function


So let's go ahead and try it out


And we'll go tf carers


activation dot linear it's not even fun to replicate this one


Has no has no model activation or activations. That's what we need. We need the s


Come in here shift and enter


Wow, are you serious?


That's all that does just our exact same tensor. Well for completeness, let's uh


does the linear


activation


function


change


anything


so plot


plot tf carers activation dot linear


a


We'll see


Oh again, I'm making the same error


Hold on this can't be real life. Let's go does a


Does a even change a equals tf carers dot activations


dot linear


All of the elements are still the same I guess well the linear activation function lives up to its


To its our documentation returns the un the input unmodified


But this is a fundamental concept of what we've what we've just covered now. Let's come back to our slide


I've just prettified all the code we've just written but the important thing is that we've written this code and we've seen it happen


But it makes sense that the model didn't really learn anything using only linear activation functions


Because the linear activation function doesn't even change our input data in any way


So it's just basically passing the same input data through the entire neural network and the outputs no wonder they're


Basically as good as guessing because it hasn't changed a single thing


whereas


with our non-linear functions


Such as sigmoid for the output layer. We give our neural network this this tool here of using this curved line


And the same thing here


For the relu function


When we give our model non-linear functions, it's able to deduce patterns in non-linear data


That's a fairly important concept now


We've only covered two activation functions here, but the the premise of what we've just seen


Is the concept of non-linearity is we've covered that so that's the main takeaway you need to


Take away from this series of videos is that neural networks use a combination of linear


activations


And non-linear activations to find patterns in data


Now if you want a resource for learning more about activation functions, there's the machine learning


cheat sheet


Activation functions i'll leave this in the resources section


But ml cheat sheet dot read the docs.io actually has a whole bunch of different stuff


But here are some of the most popular and useful activation functions. We've seen linear relu


Leaky relu is a is a different form of relu and sigmoid. I'll let you go through here


Maybe some of your extra curriculum could be to reproduce these in tensorflow code


however


We're going to push on


And in the next video, we're going to see how we can evaluate and improve


Our classification model this one that we're going to use


Our classification model this one that we've built here


So with that being said


I'll see you in the next video


In the last few videos we tackled the important concept of non-linearity


and we learned


That the combination of linear straight lines and non-linear non-straight lines functions is one of the key


Fundamentals of neural networks or other words how they find patterns in data


And we even saw a few different examples of linear functions


We even rebuilt our own non-linear functions in the sigmoid activation function and the relu activation function


but now it's time


to


evaluating and improving our


classification model


Alrighty


So do you recall that in a previous video I posed the question of what's wrong with the predictions that we've made so far?


We scroll back up again jumping all over the place here, but I did pose a question before


What's wrong with the predictions we've made?


Are we really evaluating our model correctly hint?


What data did the model learn on and what data did we predict on?


So so far


In our toy example, we've been training and evaluating or training and making predictions on the same data set. But why is that wrong?


We'll come down here


What should we do what type of data or what data set should our model learn on and what data set should we evaluate our model?


on


Now if you answered that question with we should train our models on the training data set and test our models on the test data


Set you'd be a hundred percent correct. But at the moment we don't have a training or a test data set


So let's remind ourselves of


the three data sets


Possibly the most important concept in machine learning and I know I say that about a lot of the concepts we've talked about but


This is probably the number one thing to do with data


So we've got the course materials as if you were studying at a university course is the training set the validation set is a practice exam


And the test set the test data set is the final exam and the goal here


Is for our machine learning model or deep learning model to generalize in other words the ability for our model


To perform well on data. It hasn't seen before


so


What do you think we should do to properly set up our machine learning model training and testing?


Well, if you guessed we should create


Actually, let's write ourselves a little note here


So far we've been training


and testing


on the same


data set


however


in machine learning


This is basically


a sin


so let's create a


training and test set. All right, so how many examples do we have let's uh


check


how many


examples we have


we can get


length of x that'll tell us


Beautiful it's a thousand because we use the make circles function all those videos ago to create our data


Now we could because our data is in random order


order


Right, if we look at that and we look at why?


We could randomly split this using scikit-learn's train test split


This one here


Or we could create a train and test data set by indexing


I'll let you choose how you do yours, but i'm going to create mine using indexing


so split into train


and


Test sets and then i'm going to go here


i'm going to set x train


And y train equal to we're going to do an 80 20 split


So 80 of this uh of our samples is going to be training data and 20 is going to be testing data


So the first hundred samples will be or 800. Sorry of x and y will be training samples


and then


We'll do the same for x test and y test. However, these will be the last 200 samples. So from index 800 onwards


Wonderful. Now, let's check the shape of x train


and x test and then


Y train shape and y test shape


What have we got at the output? Okay x train


And x test so there's 800 examples in x train


And it's of a shape two. That's excellent and 200 examples in x test and then y train has 800 labels


And then y test has 200 labels


Beautiful. So now we've got a training and test set. How about we recreate a model


Fit on the training data and then we can evaluate it on the the testing data


This one here. So let's write that there. Let's recreate


a model to fit on the training data


And evaluate on the testing data how we should have been doing things right from the start


So first of all, we'll set the random seed tf random set seed 42


And then we'll go number one


Is create the model now again?


We're retyping the same code because we're making the same model as lucky model 7, but we're getting ourselves or


We're getting a lot of practice writing tensorflow model code, which is very important because we are doing a tensorflow dp


learning course, so if we go over here


And what was it tf keras? Well, we could find


Model 7 dot summary. What's this going to tell us?


Okay, so we had two hidden layers with four hidden neurons each


So let's recreate that layers dense four. What was our activation function? It was non-linear


relu


Come back here


Then we'll create another layer tf keras layers


Dense four


Activation equals relu and then what was our output activation?


We recreated it again if you need to refer to the slide which which has the


Activation for the output layer of the binary classification model, but it was sigmoid. So now


Let's compile the model. So we want model activation


Compile the model. So we want model eight dot compile


We want to set the loss function to


binary cross entropy because we're working with a binary


problem


And then we want to set the optimizer equal to tf keras optimizers


Dot adam. However


We're going to make a change here. We're going to change the learning rate


So adam's default learning rate is zero point zero zero one


But I have a feeling that if we increase it


To zero point. Oh one our model might be able to discover the patterns it found in our data


Faster than what it did previously


So with model seven


We fit for a hundred epochs. We come back up here


We fit for a hundred epochs, but it looks like our model was a little bit slow out of the gates


So it really didn't start learning much


So it's still in the 50s here


and then


It's not until about halfway that it starts to really increase up past the 60s


in


About five epochs and then the 70s and about another


10 or so epochs and then it gets really close to 100 accuracy


so


The reason why do you remember how we discussed what the learning rate is? It's okay if you don't


But we mentioned it briefly in a previous video


I'll set the metrics here, but the learning rate


dictates because


Let's actually let's take a step back the optimizer


Tells our model how it should improve or how it should update its internal patterns that it's learned


The loss function says how wrong these those patterns are and then the optimizer says hey, you should improve them in this way


And the learning rate is how much our model should improve


Those patterns so if we set the learning rate to be a lower value say adam's default such as 0.001


This might be the equivalent of saying hey every time you take an epoch improve your weights by 0.001


so if we were to


Change it to 0.01. We've increased it by 10 percent. So basically every epoch we've given our model the potential


To improve its weights by 10 times as much now. That's not exactly how it works


But that's how I intuitively understand the learning rate and in practice


You'd be surprised how much that simple definition of going hey the higher the learning rate the more our model will update


works in practice and by the way


Adam's default learning rate of 0.001 and these other default parameters here are actually very good


For the majority of problems that you'll you'll work on of course as you start to get more and more into the the deep learning world


You can start to tune these to your problems


But you'll find that a lot of the values that we use in tensorflow all of these parameters that are hard coded already


Have been experimentally found as very very very good


So now let's go here. We're going to fit the model


We're going to fit the model model8.fit


We're going to fit on the training data. Oh, what have we been getting into the habit of the last few videos?


That's right setting the history variable


ytrain epochs


So i've increased the learning rate by 10


But decrease the amount of epochs our model is going to look at or in other words the amount of times


The model is going to go through the training data by four times


So let's see if it can still get just as good results with only 25 steps versus 100 steps


What have we done wrong here, oh we need to set this as model8


All right, where do we get to oh do you notice what's happened here straight away


So we have epoch one out of 25. We start at 54 percent


And then we're into the 70s after only eight epochs


And then we're into the 90s after only 15. I mean if we go back up


To our previous model. We didn't reach the 90s


until epoch 77


Wow, okay


So if we come down here


Look at that by the end where we're borderline 98 accuracy after only 25 epochs now


the real test here, of course is for


evaluate the model on


The test data set so let's go model8


Dot evaluate now that we have a test data set x test y test. How does it perform?


Well


So on the test data set in other words data rail model has never seen before


It gets an accuracy value of 100 that means every single test value. It's it got correct


Out of 200 it predicted them all correct. That is


It predicted them all correct. That is amazing now we can really evaluate this how how can we do so?


By plotting the decision boundaries. We've done this for our complete data set before but now let's do it both for the training data set


and the testing data set


Plot the decision boundaries for the training and test sets


plot dot figure


fig size


equals 12


beautiful plot dot subplot


and 121


Plot dot title


Let's go train data. So here we're just saying well, hey, we want to create a subplot


We want it to have I believe it's rows columns. So we want it to have one row two columns and the first value


Is going to be the training plot


plot


Decision boundary and then we're going to pass at model eight our trained model x


Is going to be x


train


our training data and then y is going to be


y train our training labels and then we'll set up the second subplot


This is going to be one two and the second plot


so


One row two columns. The first one is training. The second one is going to be test


And then again plot decision boundary


Model eight


X equals the test data set so x underscore test y equals the test labels


Beautiful and let's show our plot


Doing binary classification. Oh, would you look at that?


So our model's decision boundary the yellow line


Goes through the training data. It misses a few examples here. I think it might have missed that one the red dot and I think this


blue dot


So that's why our model doesn't get 100 results on the training data set, but that's okay


We want our model to generalize the data it hasn't seen before


So when we look at the test data set


It gets about 100 percent or it does get 100 percent not about 100 percent


It gets 100 accuracy


Excellent, and that was after 25 epochs


So the main takeaway is that all we did was we we took our model 7 the model that was working


But we increased the learning rate by 10 times


Now again, this is one of those hyper parameter things that won't always work


Our adam's default learning rate is actually very good for the majority of problems you're working on the only reason I set this


To be a little bit higher is because I had an inkling that our model might learn a bit faster because it performs so well


on our data set


so again


In practice you may take a little bit of a may take a little bit of tweaking for the learning rate


But we'll see in an upcoming video how we can design a function to find the ideal learning rate value for us


but in the meantime, let's uh


Since we've been saving our data


Since we've been saving our model's training history to this history variable in the next video


Let's see how we might visualize that training history


You may have noticed in a few of the previous videos


And whenever we called the fit function and created our different models


We've been getting into the habit of setting history equals model 8 dot fit something or model whatever number dot fit


Now I think we have seen this before


Plotting the history variable, but we haven't discussed it very recently. So let's go through that now


Let's see


How we can plot the loss


Or also referred to as training but probably most often referred to as loss curves


Let's go here. Now the reason why we're doing this if we come back to our keynote


Let's find our


Where's our data explorer's motto? First of all, what is our machine learning explorer's motto?


It is visualize visualize visualize


And we've had a little bit of experience visualizing our data and our models


And our predictions, but not so much training. So that's what we're going to cover here


And of course, it's a good idea to visualize these as often as possible


So let's see


What the history variable is but first to understand it. How about we look up the doc string of tensorflow?


fit


Function


What does this give us


Here we go tfkeras dot model


So this is the model class which our sequential model is built off and then when we call fit off this


so if we look for fit and


Does it have here once the model is created?


You can config the model with losses and metrics with model dot compile. We've seen that in practice


Train the model with model dot fit. That's not what you want. We want the fit function


We want the fit function to discuss what it returns. Here we go fit


So we can actually pass a fair few things to fit


We look here


If we go


X y batch size a whole bunch of things


We'll see a fair few of these throughout the course


But if you want to skip ahead and read them, you definitely can


But I want to see what it returns


Where do we go here returns? Here we go a history object


History dot history attribute is a record of training loss values and metric values


At successive epochs as well as the validation loss values and validation metrics if applicable


Beautiful, so that's why we've been setting up this history variable


now


History the documentation just said that history dot history. So what happens if we go history dot history? What does that give us?


Ah accuracy


Okay


So it looks like there's about 25 values there. So I think it tracks at every epoch which is beautiful


So really what history tracks for us is this output here?


And it's good to look at these in numerical form, but we can also look at them


In visual form. So let's um convert


the history object


Into a data frame make it nice and tabular so we can structure it up pd dot data frame


And we want to go history dot history


Let's see what it looks like


Beautiful, so we can see our models loss started at around 0.7


And decreased right down to about 0.14


And the accuracy


started at


About half 50 percent and then increased right up to 97 percent and that was on the training data set


So now how about we look at the the loss curves? We just plot this so


plot the loss curves


pd


data frame


history dot history


And then we're going to just go simple plot


I believe by default it's going to be a line plot


So this is going to be model


Eight training curves or we'll call them loss curves because that's what we've been calling them


loss curves


And let's have a look


Wonderful now that is


Probably the ideal loss curve scenario. We want to see for a binary classification problem


or actually most classification problems because accuracy


is going up and loss is going down


So actually for many problems, I think this is a key we can add in here


Let's write


I'll write in here note the loss function because remember what is the loss function? It's how wrong our model is


so


for many problems


The loss function going down means the model is improving


The predictions it's making


Are getting closer to the ground truth labels


Now you might be wondering


Okay, I understand that it's good to see the loss curve going down


But what other values or what other value can I generate from looking at plots like this? Well


In future videos, we're going to see how we can compare multiple different models and check out their loss curves


So the value in this is if we are running say we ran 10 experiments at the same time


And we plotted all of our models loss curves together and say model 8 had a learning rate of 0.01


Model 10 had a learning rate of 0.001


And we noticed model 8's loss decreases far quicker than model 10's but after about 100 epochs


Model 10 starts to catch up


So that's where we'd sort of be able to use this visual knowledge to guide our future experiments


So just keep in mind that whenever we set the history variable we can inspect our models training curves


By plotting them like this and we'll see in the future another way to do this with tensorboard


But for now, let's just leave it there and in the next video


We're going to check out how we can use loss curves to find the best learning rate


So i'll see you there


In the previous lectures we've seen how much the learning rate hyperparameter


Can influence our models training


So wouldn't it be great if we had a method that we could use to find the ideal learning rate?


I mean a value which


When our model started training meant that its loss decreased as fast as possible because remember the loss is a measurement of how wrong


Our model is so if we want to decrease the loss as much as possible


What value could we set the learning rate to be?


Hmm well to do this we're going to have to visualize our loss decreasing


And potentially decrease our learning rate during training


Now have we visualized the loss decreasing?


I think we have


There we go. Okay, so we visualize the loss decreasing now


How might we decrease the learning rate during training?


Because so far we've only hard set the learning rate


So using something like lr equals 0.01, but I haven't actually introduced you the answer to this question that I just asked


So don't worry if you're not sure but that's what we're going to do in this video. It's


finding the best


Learning rate and to do so i'm going to introduce a new concept which is called to find


the ideal learning rate in other words, it's uh, the learning rate


where


the loss decreases the most during training


Because remember that's the loss metric. We want that to decrease during training


We're going to


Use the following steps


So the first one is a learning rate


callback


Now callback is the new concept that we're going to talk through this lecture or we're going to code it out actually


Now a callback if you're wondering what it is is you can think


of a callback


as an extra piece of functionality


You can add to your model


while


It's training


So when our model trained before it went through the data


And found patterns, but it'd be great if we could execute some other kind of functionality while this training is going on


That is where callback comes into play. So that's the first one a learning rate callback. We'll see that in a second


And we're going to need another model. So we could use


The same one as above but we're practicing building models here


And finally


We're going to need a modified


loss curves


plot


So very similar to the previous one


So we're going to need a modified loss curves plot


So very similar to what we've got here this loss curve


But we're going to have to modify it because what we're going to set up is a learning rate callback


to start at a certain value of a learning rate and gradually decrease or increase that learning rate during training


And then we'll make another plot to plot the loss versus the learning rate


The learning rate value that the loss decreases the most now if that didn't make sense remember what's our motto?


If in doubt code it out


So let's get started. We're going to create a new model. So set random seed tf random set seed


Beautiful now we're going to create a model


Now it's just going to be the same as model eight remember how I said we're going to have heaps of practice coding our models


Well, we're almost up to 10 models so far in this one little section


So tip model nine equals tf carers sequential and that's really the only way to


To get better at programming or any kind of machine learning or or data science is to just just to keep writing more code


Keep working on different things. There's no secret here


Relu we could go up and see what it is


But or you could just follow along with what we're doing. So looks like it's got oh if you've got the layers


It's got two dense layers with four hidden units with the relu non-linear activation


beautiful, and then we have


An output layer with one


Hidden unit and we're using the sigmoid activation beautiful


So now what's next up we create the model. So we're going to compile the model just the same as before except now


We're using model nine


model nine dot compile


Model nine dot compile last we're working with binary


So we're going to do binary cross entropy


And then we're going to put in here the optimizer


Let's use adam


We use the text-based version of adam metrics equals


accuracy


beautiful and now


Here's the step that's going to be different


So we're familiar with these steps here, but we're going to introduce as I said


The learning rate callback. There are many different types of callbacks. But for this one we're using a learning rate callback


and so


a callback


Works during model training. So to get it to run during model training. It has to exist before model training. So before we call


model


Nine dot fit. Yeah. Yeah. Yeah


Our callback has to exist. So let's create that create a learning rate callback


So we're going to call it lr scheduler


Because as you'll see in a second if we go tf keras


callbacks


There's a callback called learning rate scheduler beautiful


And now how do we find out the doctoring of this we can press command shift enter or command shift space


Sorry, it came up automatically


So this is learning rate scheduler at the beginning of every epoch


This callback gets the updated learning rate value from schedule. So schedules this parameter here


A function provided at init with the current epoch and current learning rate rate and applies the updated learning rate on the optimizer


Okay


So this is kind of a way of saying every epoch if we put in some functionality here to change the learning rate


It's going to this callback if it's running during the fit function. It's going to


Give our optimizer in our case adam the updated learning rate. So let's see what we might do


What we're going to do is do lambda


epoch


Oh lambda needs a b


epoch and then


We're going to go one e


Negative four. So this is just 10 to the power of negative four. Yeah, and then times 10


Epoch divided by 20


wonderful


so


Essentially what this is saying is to or for the learning rate scheduler every epoch to traverse a set of learning rate values


starting from one e negative four and increasing


by


10 to the power of the epoch divided by 20


every epoch


Let's see what that looks like


Now we go here fit the model


So this time we're going to pass lr scheduler


callback


Model oh, we're getting into the habit of saving history


history


nine equals


Model nine dot fit now, let's pass it our training data


And training labels epochs. We'll run it for 100 epochs and then


Callbacks now callbacks come as a list


So we're going to pass it in lr scheduler because you can put us you can pass multiple


Callbacks here. So say you want to call back two


Three i'm not sure why i'm putting an underscore between call and back. It's actually just one word, but that's all right


We don't have callbacks two and three


We only have one. So let's just see what's going on here


We run this


beautiful


Now our training appears to be just working as normal


This is we're very familiar with this. We've seen this before


But I want you to have a think about it. What might be different now


If we passed it our learning rate scheduler here, what's it going to do every epoch


It's okay if you're not sure


Because remember our motto if and doubt code it out now what we can do is check out the history


Remember we saved it to history nine. So we'll turn it into a data frame so that we can plot it history


Nine dot history and then we're going to go plot


Fig size only my favorite two numbers in poker 10 7


Also just a handy size to put out in this this this kind of window


x label equals epochs


Beautiful. Let's see what this looks like


Oh


Okay, so this looks a little bit different to the loss curves that we plotted before


It's the exact same sort of plot that we created up here. So this is our model eight loss curves


So you see here history dot history


Now this is model nine


History nine dot history plot. We've just given the x label epochs


so


If we look at this axis the y axis our learning rate starts very low basically zero


And then as the epochs go on and on and on it starts to increase


Okay, so that's essentially what this code up here is doing


So it starts at a low number one e negative four and every epoch it increases by 10 to the power of epoch divided by 20


So that's why towards the end we get this exponential curve. It starts to increase really fast


But what happens here with our?


accuracy


our accuracy


It seems to go up slightly but then goes down and the loss


Goes down


Fairly significantly here and then it stays low but then goes back up


Hmm. So looking at this


What did we want? What did we want before? We wanted the learning rate where our model's loss decreases the fastest


so potentially


This value here, whatever it is the learning rate at say let's say 45 epochs


Maybe this is where our learning our loss seems to be decreasing the fastest


I've got an idea. Let's plot


the learning rate values during training versus the loss


So how might we do that?


Let's go plot


Because we can't really see what's going on here. Let's get accuracy out of there and let's just compare learning rate to loss


So plot the learning rate


versus the loss


so our lrs


is


What did it start at? It started at one e negative four times ten to the power of now


We're going to have to do we need a range so tf


range


100 because that was how many epochs we did divided by 20. How does that look?


That should give us there we go shape 100


Beautiful. So if we go len lrs


Lrs is short for learning rate. There we go. We have a hundred different values of learning rate


So see how it starts here


At one e negative four and then it slowly increases as we go along


All we've done with this line of code here is we've just replicated the same thing we passed


To the learning rate scheduler


Except that we had to substitute in 100 as an integer for epoch because we set 100 here. That's all we've done


So these are our different learning rates that our model tried out now. What can we do?


How about we create a plot?


plt dot figure big size


Equals


10 7


Wonderful. Now we're going to do a semi log x plot. So that means this just means we want log on the x scale


Make a plot with log scaling on the x axis. You'll see what I mean by that in a second. So log x


Lrs, so we're just passing at this that we created which is just a tensor of this shape


May have to be numpy. I'm not entirely sure we'll find out


If in doubt run the code


Now we're going to go history. We're going to get the loss. So this is what we want


So on the x axis we want lrs


And on the y axis we want the loss values from our model history


Beautiful. Now we're just going to


decorate our plot with x label can be learning rate and the y label


Can be loss


And finally, let's give it a title


plot.title


learning rate verse


loss


Beautiful. What does this look like?


Aha


Okay


Now before we even discuss this I want you to maybe pause the video for like 10 seconds and just have a look at it


and have a think about


Where do you think our ideal learning rate would be?


Totally. Okay, if you're not sure, but just have a look remember what we're trying to do here


Is we're trying to pick a learning rate value where the loss decreases


the fastest or the most


So what what part of this graph is the loss decreasing the most?


so have a think about that and


Press play when you're ready to go


How'd you go?


Well, I'll tell you the methodology


To figure out the ideal value of the learning rate or at least the ideal value to begin training our model with


The rule of thumb here is to take the learning rate value where the loss is still decreasing


So maybe here but not quite flattened out like it like it looks here


and it's usually


About 10 times smaller than the bottom of the curve. So in our case our ideal learning rate would be somewhere


in this section here


So it ends up being between 0.01. So 10 to the negative 2 so this value


And 0.02. So it'd be about this value here


Now I actually did prepare something that highlights this a bit better earlier finding the ideal learning rate. Here's the code that we ran


There's just our plotting code


boom


This is what I was talking about. So this is the lowest point on the curve somewhere down here


But remember the ideal learning rate is somewhere about 10 times smaller than that. So we have to go back here


So it's somewhere in there


now if we go back if we said that the ideal learning rate is


0.01 or 10 to the power of negative 2. What was the learning rate? We said above here


That our model got really good performance


So we can go back and look at our model


Where's model 8?


Wow


We set our learning rate to be 0.01


How phenomenal is that you could call it a lucky guess


But there's another little heuristic we can go here. We can either find the best learning rate through this methodology here


or


We could use the default value learning rate or just take a guess


So the example of other typical learning rate values


are


10 to the power of 0


10 to the power of negative 1


10 to the power of negative 2 10 to the power of negative 3


And 1e negative 4


If we have a look at these


So you see here what do these have in common?


They're all multiples of 10 now


So you could have a learning rate that's 0.03 or you could have a learning rate. That's


Realistically, there's a whole bunch of different range of learning rates that you could use but why?


Daniel you've given me these values, but you said your learning rate can be almost anything


The reason why i'm highlighting this is because whenever you use an optimizer say


Let's go


tensorflow


atom


If you use a pre-built optimizer as we've discussed before


Their default parameters are generally pretty good as in for most of the time


They'll work pretty well, but for when they don't


Well, you've got some other learning rates that you can try here. You could just try hard coding these typically


You probably won't ever use one. You'll start to use below one. So starting from here and keep going down


Or you can use this methodology to find the ideal learning rate


So up here by using a learning rate scheduler


Have it decrease during training and then plot the log learning rate versus your model's loss and find the value


Where or the learning rate where the loss curve decreases the fastest or in other words?


Just like this graphic here


So see here loss decreasing very fast. The ideal learning rate is going to be somewhere between the lowest point in the curve


And about 10 times smaller than that point


So have a practice with that potentially you could train another model on some other data that we've worked with and find the ideal learning


Rate but now that we've found the ideal learning rate, let's fit another model


Actually, you can probably try that before we go there. So pick a learning rate in this little section here


Create a new model and fit it to our data and i'll meet you in the next video


In the last video we saw how we might be able to find our model's ideal learning rate


by the learning rate scheduler callback


And starting with a learning rate of a fairly low value and slowly increasing it every epoch and then


plotting the different learning rate values versus our model's loss


and we also discussed that


The ideal learning rate is somewhere between the lowest point on the curve


And if we jump back to a region where the loss is still decreasing


So how about we try building a model?


with a learning rate of


So if we look at this point on the curve


The loss is still decreasing sharply. So that's the value we actually set before if we come back up to model eight


Model eight, there we go


So this is the learning rate we used before 0.01


And then if we come down here


Which is the same as 10 to the power of negative two. So if we go 10 to the power of negative two


But we can also see that if we go jump up one little notch here because this is a log scale remember


That the loss is still fairly sharply decreasing here. So that would be a value of


zero


So how about we try build a model?


With that learning rate and see if it can achieve similar results to model eight that we use this learning rate with


In less epochs because remember what does the learning rate code for?


It codes for how fast our model should try and update its patterns the higher the value


The more our model is going to update its internal patterns every epoch. So let's see it in action


Let's try using a higher


ideal


learning rate


With the same model as before so we're going to


set random seed


tf


Random dot set seed 42 beautiful and then we're going to


Create the model. We're up to model 10 now. How good is that double digits?


sequential


Wonderful. Now you could probably just jump ahead here if you really wanted to


We're going to have two layers that are dense with four hidden units and a non-linear activation


relu


And then we'll do the same for the other layer tf keras layers dense and it's going to have four


Activation it's going to equal relu


Or relu depending on how you want to pronounce it


And then the final layer just the same as model a dense one


activation equals


sigmoid


And now what do we have to do next? This is where we're going to compile the model with the ideal


Learning rate so the value we've just picked off the curve. So this one here


And then we'll go and put this here


Learning rate we used before model 8


Come here, we're going to go model 10 dot compile


Loss is going to be what loss are we using binary cross entropy and then we're going to go


optimizer


equals tf keras we have to use the


The class version of atom here


Lr equals what are we going to set it to?


Beautiful because we said we're picking this learning rate value here


Let's come back down


metrics


equals we'll just keep everything else the same except with


increasing our learning rate by


And now let's go fit the model for 20 epochs because model 8 we fit for


25 epochs so five less than before let's see what kind of results it achieves with less


epochs


model 10


Dot fit actually, I might say this as history 10. That's probably better


history 10.fit


x train


y train


And epochs equals 20. Let's see what happens


And come down how did it go? All right loss equals


0.878 and accuracy is 98.24


so


What was our model 8 results?


Let's go


Let's go back. So let's save this zero


Zero point nine eight two four and if we come back so point nine eight two four you remember that too


Where's model eight?


Are we going too far?


model eight


There we go


and it's


Ah, so 9824


model 10 has performed


Slightly better than model eight on less epochs so less chances to look at the data or because we increase the learning rate slightly


So it gave our model 10 a chance


To learn patterns in the data faster because they were updating


With larger steps now, of course as with every hyper parameter with our deep learning models that might not always happen


But that's just another example of how powerful


Tuning the learning rate for your models can be


So how about we evaluate it?


Evaluate


model 10 on the test


Data set actually we could have just done this to begin with model 10 evaluate


X test y test wonder how they perform differently on the test data


99 beautiful and then we go


Evaluate model eight on the test data


How did this perform model eight dot evaluate?


Model eight dot evaluate x test y test


Hmm


so our model 10 gets a lower loss value on the test data set than uh


Than our model eight, but model eight gets a higher accuracy value


Hmm now that depends on which one of these metrics you want to optimize for


So again, remember the metrics you get from the training data set aren't always important as the metrics you get from the testing data set


So this is something you'd want to investigate further depending on what your needs are and depending so potentially our model 10


Uh, just because it's learned faster doesn't mean that its eventual performance on a test data set or unseen data will turn out to be better


So this is where it takes a little bit of a trial and error to figure out which model is ideal for your use case


Now how about we finalize this model 10 with the ideal learning rate and just see how the predictions look


So i'm going to go plot


the decision boundaries


for the


training and test sets


so plt figure


fig size


equals 12 6


Wonderful, and then we'll do we'll get a subplot going


That's one two one


then


We'll do the training data first. We'll set up the title there


Then we'll bring in our trusty function plot decision boundary


And that takes in model 10


It'll also take in x train first. We need an underscore there


And then it'll also take in y train


And then we want to subplot again


one two two


So again, this is row one


Column two or two columns and this is section one. So the first one and this is


One row two columns and the second section. So we'll see what that looks like in a second


plt dot title


It's going to be test and we want to bring in our


Fancy function from above model 10


x equals x test


And then y equals y test


Beautiful and then plt dot show


Binary classification wonderful


So again our model 10 using a ideal learning rate that we picked off the loss curve


gets


Basically perfect predictions on the training data set and the test data set and see if we can get a perfect


Data set and see this is what i was talking about before with the subplot function. We want one row


two columns and this first plot train is the first element here and this test


Plot is the second element in the subplot


So with that being said we've explored a few ways to evaluate our


Classification models we visualize them, but there are a few more classification evaluation methods that we should really be looking at


So let's have a look at those in the upcoming videos


So far we've seen a few visually rich ways to evaluate our classification models


But let's have a look at a few more evaluation methods that we can use


Now these are some of the most common


Evaluation metrics that you should have in your machine learning and deep learning toolbox to evaluate your classification models


So let's have a look here. They've got a key tp equals true positive not toilet paper tn equals true negative


Fp equals false positive and fn equals false negative. So let's have a look. We've got metric name accuracy. We've seen that one


There's the formula there true positives plus true negatives


So all of the true predictions on top of divided by all of the other predictions


If you wanted to do this in code the accuracy metric you can do tf dot keras dot metrics accuracy


Or if you want to use scikit learn it has a accuracy score function in there


Now when should you use accuracy? Well, it's the default metric for many classification problems. However, it's not the best for imbalanced classes


So for example if you had 10 000 examples of one class and only 10 examples of another class


And you've got a classifier to score 99.999


99.999 or something like that. It could just predict that


Everything is all one class and get that sort of result


So in that case, you probably want to look into using metrics like precision


Now there's a formula for precision here true positives over the total positive predictions including false positives there


If you wanted to use it in code you can use precision


Higher precision leads to less false positives


Let's go to google and go what is a false positive?


Oh coronavirus testing what is a false positive that's a topic of the moment


What is a false positive? There we go


A false positive is when someone who does not have coronavirus tests positive for it. So you can see


where a false positive may have


Altercations. Oh, it's not a good thing because if someone tested positive for coronavirus that could go on to have a whole bunch


Of adverse effects that you didn't want


So if you're training a machine learning model deep learning model as well


And you wanted it to predict less false positives


In other words predicting if someone had coronavirus when they actually didn't you probably want to optimize for the precision metric


Now there's also recall


Which is true positives over true positives plus false negatives


If you wanted to use a code you could use one of these two functions here


Now higher recall leads to less false negatives


So if a false positive is someone being predicted as having coronavirus when they don't actually have it


What do you think a false negative is?


And what would what would be the implications there?


So the false negative in this case would be if someone was to predict we did a coronavirus test and I actually had coronavirus


But my test came back and said that I didn't have it so you could imagine


Again that would have consequences as well


If I had a test that said that I didn't have coronavirus and I go about my life and do whatever I want to do


And then I start giving it to other people


Well, then that's not an ideal scenario. Is it?


So for problems where false negatives are not good for your use case


You want to train your deep learning models to have higher recall


However, you might be thinking why don't we just increase precision and recall?


There's often a trade-off between the two


And I'll show you here. What is the precision?


recall


trade-off


Precision recall trade-off beautiful


Let's understand precision recall


Where is it? Have we got a trade-off curve?


There we go, this is probably it unfortunately, you can't have both precision and recall high


If you increase precision, it will reduce recall and vice versa. This is called the precision recall trade-off. So keep that in mind


that


In an ideal case your model would have high precision and high recall, but you don't have high precision


And high recall, but usually when you try to improve one versus the other


The other one goes down. So say we wanted high precision


You would have lower recall


And the inverse of that is also true


Another option is to try and improve the f1 score which is like a combination between precision and recall


So the f1 score is one of the scores that I like. It's usually a good overall metric for your classification models


But again me just reading these evaluation metrics to you out loud


Probably doesn't make as much sense as to when you start to code them up and start exploring them yourself


So just keep that in mind. We're just talking about these here. We're just naming these different metrics


It's not until you get to start to use them in practice that you'll really start to understand when to use which


And then finally another great one is a confusion matrix


So this is particularly helpful when you're dealing with all the way from binary to multi-class classification


You can create your own custom function or scikit-learn has a built-in confusion matrix function


So when to use?


So when comparing predictions to truth labels to see where the model gets most confused


It can be hard though to use with very large numbers of classes as we'll see in a future project


So keep these in mind again


This is just a slide in the upcoming video. We're going to have a practice implementing a confusion matrix


But for these ones here the accuracy we've already we've already done this during our model training


So for precision recall and f1 score your homework for this video is to dive into the tensorflow or scikit-learn


Documentation for each of these three and then i'll see in the next video


We'll start applying some of these metrics or evaluation methods to the problems. We've been working on


In the last video we introduced some more classification evaluation methods


Now again, these are some of the most common and the ones you definitely want to keep in your toolbox


So make sure you you screenshot this or take a note down of these metrics


Because they're probably going to be the ones you most commonly see with classification deep learning models


We didn't write much code last video. So let's make up for that. We'll go back to our notebook here


And all right another heading here more


classification


evaluation


methods


and so alongside


visualising


Our models results as much as possible


There are a handful of other


evaluation methods and metrics


You should be familiar with


And because this is not a markdown cell it's going to keep going that is all right


We'll just put a little few dot points here. We'll just replicate that slide that we had we had accuracy


Which is probably the most common


and then precision


A higher precision leads to less false positives


And then we had recall


A higher recall leads to less false negatives


However, there is the precision recall trade-off which is an important concept to be aware of


There's the f1 score, which is a combination of precision and recall and usually a good overall classification metric


And then of course, there's the confusion matrix, which is another visual way of looking at things


And then finally, this is not tensorflow specific, but it is


Also another way that you can see everything classification report from scikit-learn


Is that on the slide before? I don't think so. We can put this one here


Okay here classification


classification report scikit-learn the good news is a lot of these


Classification metrics all have a very similar principle. They take in some true values and they compare them to our models predictions


That is the crust of of whenever we're evaluating a model


What were the true values the model should have predicted and what were the values that it did predict?


Let's compare the two. So i'll just put this little link in here


For the rest the previous slide that we looked at shows the code examples that you can use


So let's start off with accuracy. I mean our model has already used it because we passed in


Accuracy up here as the metrics


Let's write some code to make that look a little bit better


So we can go here


We've got check the accuracy of our model


and we got loss


accuracy


equals


model 10


dot evaluate


And we're going to evaluate it on the test data set


And then we're going to print we'll make this a little bit prettier


so model has


Or just model loss on the test set that sounds a bit better doesn't it?


And then we're going to do because it's an f-string we can pass that beautiful and then


model


accuracy on the test set


wonderful and then we can just


Because we're going to set accuracy up here. Let's times that by


100 and then


we'll just put


We'll shorten it to two decimal places will this work


I hope so


There we go. Okay. So see what we did there was we just evaluated it on the test data set and then we found out the


Loss because that's what our model is going to return with the evaluate function


So the loss is there and then the model accuracy it comes out in this dot point


Notation we've just adjusted it here to be more visually appealing


So what should we work on next?


How about a confusion matrix?


So what we do is


We'll end this video here and we'll come back to the next video and we'll see how we can make a confusion matrix with our model


Last video we left off with the question. How about a confusion matrix?


We checked out the accuracy again


If you want a little bit of homework, you can check out our models precision recall and f1 score


Before we create a confusion matrix


Let's see


What the anatomy of a confusion matrix is so this is what we're going to be working towards creating


So it's a matrix


So it's got rows and columns


And on the y-axis of a confusion matrix are usually the truth labels


So this is what ideally our model did predict but on the x-axis is what our model actually did predict


so


Inherently what happens when you lay it out like this is that on the diagonal?


Are the correct predictions in other words the true positives and the true negatives?


So you might be wondering well, what's a positive and what's a negative?


Well in the case of binary classification a true positive is when the model was supposed to


Predict one when the truth is one and a true negative is that the model predicts zero when the truth is zero


So knowing this you might


Have guessed that the ones outside of this diagonal are the false positives


And the false negatives in other words a false positive is when a model predicts one


When the truth is zero and a false negative is a model predicts zero when the truth is one


So again, this is just seeing the name of things. Let's code this up and see what it looks like. Let's go back to the previous slide


And see where the code is at confusion matrix. What is a code custom function or


Sklearn metrics confusion matrix. All right, let's not reinvent the wheel to begin with let's bring in scikit-learns confusion matrix


scikit-learn confusion


matrix


Metrics dot confusion matrix beautiful


Okay


Is there an example?


All right. So example from sklearn metrics import confusion matrix. We need true labels


We need some predictions and then we just pass it to the function. Wow


Let's try it out. So


Create a confusion matrix


So from sklearn dot metrics


Import confusion


Matrix wonderful, then we need to make some predictions. How do we do that with our train model?


So we'll save into ypreds


equals model 10


Dot how do we make predictions with a train model?


If you guess predict


You'd be correct. So we'll predict on the test data set and then we can create


our confusion matrix


How good is that I love scikit-learn I love tensorflow


Some beautiful beautiful code libraries out there


So that's all it is. We've got the y test which are our test labels and


Our model's predictions. Let's yeah, see what the confusion matrix looks like. Oh, no


What's happening?


No module named sklearn. Oh typo


standard


Sklearn dot metrics import confusion matrix. This should work


Oh, no


Classification metrics can't handle a mix of binary and continuous targets


Hmm


What is happening here?


What is happening here? Well, what do we do when we face value errors like this?


We have to inspect what we're trying to predict on. So what does y test look like? Maybe we view the first 10 values


Okay, so that's what y test looks like what do our preds look like do they look the same


Ah


I see where the trouble is


Look, this is why we're getting a value error classification


Metrics can't handle a mix of binary and continuous targets. So our test values are in binary form


So it's zero or one whereas our predictions are in continuous form. So they're not zero or one


So what should we do here? What do we have to do to compare?


our


test array and predictions array


Well, we're going to have to convert


Our predictions array into zeros and ones but what even are these values here?


Well, these are called let's write this down


oops


looks like


our predictions


array


has come out in


prediction probability


form


So this is the output


We'll make that in bold


so


The standard output from the sigmoid


Or softmax as we'll see later on


Activation functions


So what we're going to have to do


Is convert them so they're in prediction probability. So this is a value


That the model has output. So the closer the value is to one


The more the model thinks that it's a one label and the closer the value is to zero


The more the model thinks it's zero value


Out of the knowledge that we've learned with tensorflow so far


Is there a function that we can use to potentially round these values? There was a little hint there


to their closest integer value


so for example if one of these was


Let's see if this actually outputs 9.8.5.2. I'm not going to type out the whole thing because that's going to take way too long


What does this actually look like?


There we go. So is that closer to


Zero or one


Now you might be wondering


Of course, it's closer to one. So this one would go to one, right?


And then the same would be for this one and the same would be for this one


And then we might have to go on for a while before we find zero


You might be wondering what's the cutoff?


Well, the cutoff is 0.5


So anything higher than 0.5


Will go to one and anything lower than 0.5 will go to zero. However


This is a value again. You can tune but for simplicity's sake we're not going to use that for now


We're going to just convert


our prediction


probabilities


to binary format


and view the first


Now again, I want you to try and figure out this yourself. How could we round this?


To zero or one using tensorflow?


If you guessed we use tf round


Or perhaps you already knew that you're like daniel. Come on. I'm about 50 videos into using tensorflow. I know this stuff already


We go boom, what does this look like? Ah, that's what we want. That's looking more like what our test labels are


So remember whenever we're comparing things


Again, one of the biggest issues that you'll run into is your test labels


And the other issues that you'll run into is your tensors or your data types being of the wrong format


So all it is is about thinking about how you can get them into the right format. So that's what we want


So this is what you'll often have to do with a classification problem


The outputs of your model will come in prediction probability form and you'll have to convert them to


human readable form in other words integer form


now


Let's see what our confusion matrix looks like. So let's


create


confusion matrix


Y test and then we want to go tf round y preds


And we're off


Hmm. Okay


So we got down the diagonal if we go back to the anatomy of our confusion matrix


let's


See


All right, so we've got the same values here


But this isn't as pretty this is just an array


What we might do in the next video is prettify our confusion matrix to look something like this


We know what the y-axis is. We know what each row is. We know what each column is. We know what the x-axis is


So we can look at this one, but it's not the ideal type of confusion matrix


I mean if you imagine you were trying to share that with a colleague what is even going on here?


so


In the next video, we'll see how we can prettify our confusion matrix


So in the last video we made our first confusion matrix, however, we said that it's nowhere near


As pretty as a confusion matrix we have here. So how about we write this down?


How about we prettify?


our confusion matrix because there's to be more beauty in the world


And excuse me, i'm getting a little bit poetic here a code can be poetic too


So the function we're going to use or the code we're going to write


Is i'm going to put a little note down here. So the confusion


confusion matrix code


We're about to write is a remix of


scikit learns plot confusion


Matrix function now we've got this up here


If you look into this we can click on the source code


And we could follow through with that fairly extensive source code scikit learn is beautifully documented and you can go through that


I tried out this plot confusion matrix function and I found out it only works with estimators. So


A scikit learn model is referred to as an estimator. Whereas we want to use tensorflow


So we want to adapt it to our tensorflow code. So you'll often run into this right?


You'll often run into it's like where you want some sort of functionality


But it exists somewhere else


But to get it working for your use case, you kind of have to tailor it. So that's what we're going to do here


So just follow along


And we're going to make a pretty confusion matrix with this here and with our tensorflow model


So let's set up a fig size


Just so we can use this again


Now because we're writing we're going to be writing a fairly extensive amount of code here


What's our principle?


Whenever we're writing something if you get stuck or you're not sure what's going on and my explanation isn't as great as it could be


Always pause and rewrite it yourself and see what happens


That's what I do whenever I don't understand something. So we want to go


We can just create our confusion matrix just like we've done up here. So nothing new so far


YPREDs


Beautiful. You might have noticed I've imported iter tools. We'll see where that comes into play in a second


We're also going to create a little normalized feature


Because if we come back here, this is how we're going to get percentages here


Right. So this label 98 percent of them are correct and two percent are incorrect


This confusion matrix works out well because we have a hundred examples


Might be different depending on the amount of samples you have for each label, but


Let's not get distracted Daniel. Let's code. So we're going to go cm


as type


float


Now because remember our confusion matrix so far is just an array


A very non pretty array that is we can just divide it by the sum to normalize it


Axis equals one


And then we're going to go here np


New axis. So this is going to be this will normalize


Our confusion matrix


Again if we wanted to see what that looks like we could go cm norm


Wonderful, there we go


And now let's go in here


We want to set up number of classes


So we can do this by getting the shape of


Our confusion matrix again to see what something does cm dot show


See what something does cm dot shape


This will be helpful for if we had multiple classes


So right now we're only dealing with binary classification. But what if we had 10 classes?


We may see that later on spoiler alert


So let's prettify it


Got


My dyslexia kicked in there. I've got my f's and t's mixed up. So we're going to create a figure


and an axe using


Plt


Dot subplots and we're going to set our fig size to equal fig size


We hardcoded this up above here. You don't have to do that, but I just decided to


and then we're going to


create


a matrix plot


so cx


equals


Ax dot match show. So this is a matrix plot


We'll go here matplotlib


Mat show. What does this do?


Display an array as a matrix in a new figure window beautiful


Again, we're remixing some code from scikit-learn here


so


Mat show. What do we want to pass it?


We want to pass it our confusion matrix, right? So cm and then we'll set the color map to equal plt dot cm


blues a lot of cm's here don't confuse our confusion matrix cm with


plt dot color map blues


Wonderful, and then we'll go fig dot color bar


cx


wonderful


And finally we can create


classes


We need to set up a bool


So if we do have multi-class we want it to do something and if we do only have binary class


Which is what we're working with now. We want it to do something else. So that's why we're creating a conditional


if classes


labels equal classes


else


labels equal np a range


cm dot shape


zero


Wonderful


So if we have a list of classes


If it exists, we'll set the labels to equal classes


But if it doesn't which is our current scenario


We'll set the labels to be just a range of our confusion matrix shape on the zeroth axis, which is just two


So it'll be zero to one


Wonderful and now let's label the axes


Ax dot set because we're going to be writing a bunch of labels here


We're just going to use a shortcut axe dot set for matplotlib confusion matrix


That's a title


And remember we're prettifying it. So it looks like this. That's our title. This is our x label. This is our y label


So these are the ones that we need


x is the predicted label


And then y label equals true


Label


Wonderful. We're also going to set up the x ticks to


be np a range and classes


And then the y ticks is going to be the same thing np


A range and classes. That's the number of little dashes that we want to have on our figure


We'll see what that looks like in a second


We can also set the x tick labels to be our labels variable which in the case of binary classification


It's just going to be the range if we don't have a list of class names


the labels are just going to be the class integer values and


Same thing for the y tick labels equals labels


We need to set the threshold


For different colors. This will make sense in a second. So the threshold


equals


cm dot max so the max value in our confusion matrix


plus


the minimum


And then we want to


Divide that by two. So that's the threshold. So this is going to give our confusion matrix different shades of squares


Depending on how many values are in there. So a typical confusion matrix uses you want the diagonal access to be really dark where


The correct predictions are and all of the other ones where there's not many prediction values to be light


It doesn't make a lot more sense when you visualize our pretty confusion matrix


So plot we're going to plot some text on each cell


So we can go for i j in


iter tools


So iter tools is going to iterate through whatever we pass it. So we want range cm shape


zero


And same thing for range cm shape


one


I'm going to go boom plt now. This is we're going to set the text for each square


So j i this is giving ourselves coordinates


And then we can set up an f string


for our confusion matrix for


i j coordinates i j index that is


We want that


And then it's also going to be and we want in brackets


cm norm


Again the same locale


index


times 100


We want one decimal point here f


and then


squiggly bracket fix up the f string


percentage


Or our brackets correct here fingers crossed


There's going to be an error here somewhere, isn't there?


Let's traverse back through where have we got an extra because we want this to be the same color


Okay, wonderful. Yep, that should be like that. Oh, our f string has ended too early


That's what we want wonderful and now we can have a little comma after that


We want horizontal


alignment


equals


center


We want to go color equals white. This is going to be the color of the text if our confusion matrix


At a particular index


Is greater than the threshold else we want the text to be black


Wonderful and the size can be 15


There's a fair bit going on here, but again


If you're not sure what's happening in any of these lines remember break it apart. We've got a lot going on here


This took me a while to remix this function here


So I didn't just pull this out of the hat


Let's see what it looks like. Anyway, see where our errors appear wonderful


Text has no property horizontal alignment. Oh, I have a note n


There we go


So we've basically just replicated our pretty confusion matrix see what I meant with the color threshold here


We want our squares with lots of values in them to be darker and the squares with not many values in them to be lighter


And the more values is you can't really see it with this confusion matrix, but we will see it later on


The higher the values the darker the shade of the square will get maybe if we go confusion


matrix


Images


There we go. This is a good example


So this is without normalization


So we see here all these squares are light and all these squares are darker because they have more values


So with that being said we probably could increase the text size here


So that's a bit more visual. How about we do that?


What do we want to do? We want to set the x axis labels


To the bottom


That's what we'll get that to the bottom. You see those zeros and ones we want to get those down there


So we can go ax dot


x axis


access the


x axis parameter set label position


And then we're going to just type in


bottom


Then we go x axis


Dot


Tick we want to have ticks down there as well, correct? And then we want to


Adjust the label size we can do that by going ax


y axis


label set size


And then we're going to ax x axis label set size


And then we can do the same for the title ax dot title


dot set size 20


now, of course


Beautiful that's looking way better. Have a go at this


We've just prettified


Our basic confusion matrix we come back up here


Albeit it took a fair bit of code. We have this array here


It now looks like this


Now, of course we could functionize this maybe that's a little task for you could functionize this to work with any y test values


And y pred values


So maybe we look at that doing that later on or maybe you could try that now but as you could see


This is a really great and visual way that we could quickly show someone how our model is performing


And see how it does on different classes where it gets confused now our model is doing pretty well right now on our data


Because as you can see the diagonal is very dark


But going forward it'll make more sense once we work with a multi-class classification problem and make a bigger confusion matrix


To see where our model messes up when comparing say if we had 10 different classes


So that's where we'll finish this video


We'll get rid of these two cells here


And then in the next video we're going to start tackling a larger example more specifically multi-class classification


So go back through see if you can turn this into a function. Maybe you want to turn it into


A version of plot confusion matrix so you could do


def plot confusion


matrix


Blah blah blah and see if you can pass it something like true labels and predicted labels


And have it come out with something like this


So give that a shot


And i'll see you in the next video


Alrighty welcome back. I'm super excited for this series of videos because now we're going to start


Let me write this down actually


working with a larger example


In other words or more specifically multi-class classification


So we'll just make this little heading here actually it's probably worth making it


A size one heading so we'll turn that into markdown


Oh, we've spelled multi-class classification wrong


That's all right


Now what we're going to do in this series of videos is so far we've created our own data set


Now we created a blue and red circle data set which is a fairly simple data set


And then we went through a whole bunch of different steps to model it. We learned how we can improve our models


We learned about non-linearity, which is important if we want to model data that is non-linear so has different shapes


We learned how we could evaluate and improve our classification models. We learned how to plot the loss curves


Find the best learning rate and a whole bunch more classification evaluation methods


So now to really drive all of these concepts home. Let's start with a new problem


a multi-class classification problem


So let's say you were


A fashion company and you wanted to build a neural network to predict whether


A piece of clothing was a shoe a shirt or a jacket. So in that case you have three different options


So let's write that down


When you have more than two classes


As an option


It's


known as multi-class classification


Multi-class classification


So two classes on their own is binary classification, but more than two is multi-class classification


so that means


this means


If you have three different classes


It's multi-class classification


It also means


If you have a hundred different classes


now the good news is


with a few tweaks


Everything we've worked on so far in our binary classification problem. We can apply to a multi-class classification problem


We could just look at here to see the steps that we've gone through


or we could just jump back into our presentation


And remind ourselves of the steps in modeling with tensorflow


I love that little animation


Look at this colorful little picture. So we need step one is get data ready turn it into tensors. Okay


Step two is build or pick a model. We've built a lot of models so far. So we are pretty familiar with this. Yeah


Fit the model to the data and make a prediction. Okay evaluate the model


Improve through experimentation and save and reload your trained model wonderful. Let's start with step one


Let's get the data ready


So I kind of hinted at what kind of data set we're going to be using before


We're going to pretend that we're a fashion company and we want to build a neural network to


classify different images of clothing


so


To practice


Multi-class classification


We're going to build a neural network to classify


Images of different items of clothing now


The beautiful thing about this is that we can use the fashion mnist data set which is built in to the tensorflow


Dot keras data sets module. So let's have a look if we go tensorflow


fashion


mnist


Fashion mnist beautiful fashion mnist tensorflow data sets


So the tensorflow data sets module


Is up here. It has a whole bunch of different built-in data sets that you can use to practice on your own problems


So they're great to use to sort of get familiar with how a neural network you can build with tensorflow and get it working


Before you adjust it to your own data set


That's a really important concept a lot of the time in machine learning and deep learning you'll work on problems that have existing


Outlines and then slowly adjust them to whatever you're working on


So we've got a description here fashion mnist is a data set of zelando's article images consisting of a training set of


60 000 examples so it's going to be our biggest data set yet


And a test set of 10 000 examples


Each example is a 28 by 28 grayscale image associated with a label from 10 classes


You could go through that documentation there or you could just follow along with the code here


To get started with the tensorflow data sets. We'll re-import


tensorflow as tf


And we'll also get the tensorflow


dot keras dot data sets module. We're going to import


Oh that needs to be data sets we're going to import fashion mnist


now


The good thing about the tensorflow data sets is that all of the data sets are basically


As many as you can or at least the ones i've worked with so far, which is a fair few


Have already been split into training and test sets


So we can go


The data has already been sorted


into training


And test sets for us


So to import it we can use tuples. This is going to be trained data train labels


This is going to be trained data train labels


And then another tuple for the test data


and test labels


So we'll set that up as equals fashion underscore mnist, which is this module. We just imported here


And then we'll use the load data method


So of course if you wanted to see this in the documentation you could come back here


And the split says that it's already in test and train


10 000 and test 60 000 and train


And then there should be an example somewhere


Of how to import it


I guess not


That's right. It's somewhere here. It may be in the overview section


Let's get back to focusing writing code. So we shift and enter that


And it'll download it'll be pretty quick because it's a relatively small data set and it's stored on google storage


Now to check out an example


We can go


show the first training


example


We go print f


Training sample


And i'm going to put it on a new line just so it's nice and neat


We'll go train data and then we'll just get the zeroth index from that


And then i'll get a new line after that


We need to finish that f string


There we go. And we're going to do the same


thing for the training label


We'll just do the same index here, of course, we could have done this randomly but i'm going to just


Get one out of there because that's one of our first steps right when whenever we're downloading data


We want to become one with the data. We want to visualize visualize visualize


Okay


So what have we got here training sample?


We've got some sort of


array of numbers


beautiful


And the numbers are varying


It looks like from zero


Up to about 255. Okay


And then the training label is a nine


All right. So this array of numbers


Or this matrix or this tensor of numbers


represents


Training label number nine. So if we go here


We look up zelando researchers fashion m nest


These are the images that we're going to be working with


So there's 10 different classes of image. We've got shoes. We've got dresses. We've got shirts


Come down


We can get the data from there. Okay. Here's the labels so labels


Each training and test example is assigned to one of the following labels


So number nine, this is an ankle boot


Wonderful. So this first sample is an ankle boot now


Of course, we could keep looking at numbers


What else do we want to investigate whenever we're working with a new data set? We want to look at the shape


We also want to look at what it looks like. So let's do that


Let's go


check the shape


of a single


example


So let's go train data


We need to get the first sample dot shape and let's go train


label


labels zero


dot shape wonderful, so we've got a 28 by 28


Tensor here and the train labels is just a scalar so it has no shape there now. How about we


Get visual plot a single sample


And come in here so to plot it we can import that plot lib


Plot as plt and we go plt because we're working with an image


m show


And then we can just pass it a single example


we go zero and


Because its training label is nine


What should it look like? It should look like an ankle boot. Let's see if this works


Oh, okay, so fairly simple


Oh, okay, so fairly pixelated here, but you can kind of see the outline of a boot there


And how about we try another one?


Wonderful, so that looks like a sweater and then


check our


samples label


So if we go train labels and pass it the same indexes here


What label does this get?


All right, it gets a two and if we come back here


It's a pull over. Okay? So


I said this was a sweater kind of looks like a sweater to me


But again, depending on what data set you're working with it will have different labels


So we've gotten pretty familiar with our data


We're probably going to have to set up a little list to index on our training labels


So let's start doing that in the next video


And then we'll slowly keep getting more and more familiar with the data before we start to build a multi-class classification


neural network


multi-class classification neural network


Let's keep going getting familiar with our data that we're going to be working with for the multi-class classification problem


So far it looks like that our labels are in numerical form and while this is going to be fine for our neural network


We probably want them in human readable form. So what we're going to do is create a small list of the class names that we found


on


the data sets github page


And that way we can index onto that list so that instead of just having a train label as two this actually reads as pullover


So right down here


Create a small


list


so we can index


onto our


training labels


so


they're human


readable


Again, this is all part of becoming one with the data that we're working with understanding what kind of problem that we're working on


So we go here. All i'm going to do


is just


copy these


In the same order. So it'll be t-shirt top trouser pullover dress


So let's just write that down might speed this little section up


Wonderful now we've got a list of class names that reflects


The actual human readable name and we've got their labels in integer form, which will be great for our neural network


So let's have a look. How many classes are we dealing with?


10 so again anything over two is classified as multi-class classification


So now we've got our class label names. Let's plot another example


So we go here plot an example image


and it's


label


And go plt.mshow


And which number or which index should we pick this time?


How about we go 17?


and then


we're going to set the color map to be


plt.cm for color map


To be binary because this is going to be grayscale


We'll set the title plt.title now. This is where we can index onto our class names list


class names list with train label


And the same index here


Or actually we can probably just create an index


of choice


variable


So that we can just update that index of choice again


You can probably just set this up to be just a random number generator


If we really wanted to


But this is how we're going to plot different examples. Wonderful. So there's a t-shirt slash top and we can do it again


20 there's a dress


T-shirt slash top


T-shirt top 100


2000 there's a bag


And there's a code wonderful. Now again, we could keep going through these


And visualize different examples so that we know the data that we're working with but what's next?


How about we set up?


Rather than just running this cell dozens of times


We set up just some code to plot a bunch of random examples so that we can get familiar with them


Now with a data set like this, you could probably just look at what it looks like here and start to understand


Okay, this is pretty straightforward what it is. It's 10 different types of pictures and they're all grayscale


But if you had a larger data set with a hundred different categories and your images weren't grayscale


They had a whole bunch of other details in them


You probably want to have a look at multiple or hundreds if not thousands of samples before you start building a model


And when I say that number again, that's an arbitrary number


It's just the number of samples that you start to feel familiar personally with the data that you're working with. That's the most important part


So let's write some code to plot multiple random images


of fashion MNIST


import random


plt.figure


And right now we're working with an image data set, but the same would go for any type of multi-class classification


data set that you're working with is to visualize, visualize, visualize


As many samples as you can. So we'll probably loop through four samples at a time


We'll set up an axe with plt.plot


Actually, we need a subplot don't we because we want to plot multiple different things


We'll give it two rows two columns


And the index can be i plus one because we're looping through


We'll select a random index using random.choice


And we'll set up a range for the length of how many training samples we have which is 60 000


So all this is going to do is just pick a random number in the length of our training data


So it'll be from zero to fifty nine thousand nine hundred and ninety nine


And then plt.mshow


We're going to show the training data


training data


Sample that appears at the random index and the cmap


Can equal plt.cm.binary


So it comes out in grayscale


And then the title what we want the title to be we want class names


And then again, we'll index on our training labels


Where the random index occurs


And we don't want any of the ticks because we don't need them. We just want to see the images