So hopefully this works. We should be able to run this cell


Learn TensorFlow and Deep Learning fundamentals with Python (code-first introduction) Part 2/2

And visualize


Okay shirt bag sneaker trouser wonderful


T-shirt shirt dress bag


So you can imagine if we went through this this is a bit easier than what we just did here


so this is what I like to do whenever i'm working with a fairly large data set is


Never underestimate the power of randomness


Is I like to look at samples randomly and just keep going through the data set start to


Build an image of what these look like in my own mind before I start to build a neural network


That's going to distinguish patterns in them


Now I want you to have a careful look at the data that we're working with


What kind of shape does it take on?


Are there many straight lines? Are there any curved lines?


How does that relate to what we've learned with linearity and non-linearity?


So do you think with this type of data that we're working with are we going to need a neural network that uses just linear?


Like straight lines or will we need a neural network that's going to have some kind of non-linearity in it?


So that's something to think about


But have a play around with this little section here


Become one with the data run it about 25 more times. So you've seen at least 100 different samples and then next video


And then next video


If we come back to our modeling


We've got our data ready. Luckily. It's already in tensors for us because we've downloaded it from the keras data set module


However, oftentimes your data won't be ready turned into tensors. So we'll take this step off


Now we're up to building or picking a pre-trained model to suit our problem. So we'll get onto that in the next video


Okay, so we've started to become familiar with the multi-class data that we're working with now


Let's build a model. We're building a multi-class


Classification model now I said earlier


Well, we've already built a fair few binary classification models and I did mention earlier that the multi-class classification model


We're only going to have to tweak a few things


To get it to work with our multi-class classification data


So if we go back to the keynote


We look at the typical architecture of a classification model


So we've got multi-class classification over here is what we're working on now


So the input layer shape same as binary classification


Depends on the number of features you have


Number of hidden layers again same as binary classification


Neurons per hidden layer, it's the same as binary classification


Neurons per hidden layer. It's the same


We've got another difference here. It's the output layer shape


so


For binary classification is one one class or the other


Multi-class classification has one per class


For the hidden activation we can use the same


In binary classification, we use the non-linear function relu and I asked you in the last video to think about


Whether or not we might need non-linear activation functions


The output activation. Okay, that's different too for multi-class. We're going to need the softmax function rather than sigmoid


We also need a different loss function


So for binary classification, we use binary cross entropy


But for multi-class classification we use categorical cross entropy in tensorflow


And then finally the optimizer can stay the same so we can use our trusty atom. All right, let's write some of these down


We go here


For our multi-class


Classification model


We can use a similar architecture to our binary classifiers


however


We're going to have to tweak a few things


Namely first of all is the input shape. So


What is our input shape? What what shape is our data? This is what we explored before. So the trained data sample zero


dot shape


Okay, so our input shape is 28 by 28 so let's write that down 28 by 28


The shape of one image


wonderful


And we're also going to have to modify the output shape


What was the output shape?


If we come back to our typical architecture of a classification model the output layer shape is one per class


So how many classes did we have again?


length class names


Beautiful. So the output layer shape is 10 one per class of clothing


That's what we want


And what else was there? Ah the loss function


loss function equals has to be tf carers dot losses


dot categorical


cross entropy


instead of binary cross entropy


I'll put that there and then


I think that's all we'll have to do


Beautiful


Because we can keep the other things the same


Ah the output activation, that's what we need to write down


output layer activation


Equals softmax not sigmoid. All right. So with this being known


How about we put together our first multi-class classification neural network model?


So we're going to go through the exact same steps that we've been through before so set random seed tf random dot set seed


We'll give it 42 wonderful


Let's just try replicate the exact same model we've been using in previous videos


So we're going to create the model


I think we're up to model 11 now


tf carers sequential


Wonderful, and then what did we have we had two


dense hidden layers tf carers layers dense with four hidden


units and


a relu activation


So we'll just make two of those


make two of those


tf carers dot layers dot dense


four and a relu activation


Wonderful non-linear because this is the answer to the question I posed before


If you guess that we're going to need non-linear activation functions for our data


You'd be correct because it is composed of straight lines and non-straight lines


So that's why we need non-linear activation functions


For our output layer. Oh, this is going to be different. Isn't it? We can't use the exact same one


How many output shapes do we need? We need 10


Wonderful, we can do that 10 and we're going to change our activation from sigmoid to softmax


So we can do it like this


softmax or


We can also do it like this tf carers activations dot softmax now. I'm not sure if it's a capital s


Let's look it up tensorflow


softmax


activation


There we go, so I believe we might just be able to put it like that


Let's see if autocomplete helps us out softmax


There we go. Wonderful. Now. What else do we have to do? We have to compile the model as always


Compile the model. We're up to model 11


dot compile


Tf carers loss equals what do we have to change here? We need to change it to


categorical cross entropy tf carers losses


categorical cross entropy


Wonderful, we can do that


And then we can go optimizer. We don't have to change that we can do our trusty atom optimizer


We'll leave all the default parameters there and finally let's set up metrics we'll just set as accuracy


Our trusty accuracy beautiful so let's fit the model and we're going to save this as non


norm history


And you might be asking why is that? Well, I will reveal all in an upcoming video


Now we can fit directly on the train data and the train labels


And we might go just for 10 epochs. So 10 passes through all of the data


And we're going to introduce a new parameter here to the fit function, which is validation data


So this is where you can put in we don't have a validation data set, but we already do have a test data set


So what what we can do here is at the same time the model is fitting and trying to find patterns in the training data


So the relationships between the training data and the training labels it can evaluate how well those patterns


Are on the validation data, but in our case, we don't have a dedicated validation set


So we'll just use our test data here


So this data will remain unseen


And that way we can evaluate how good our models patterns are that it's learning in the training data


are


When we use them on unseen data


So, let's see how this goes


Oh, no, what have we got wrong here value error shapes


Ah the classic shape error. Hmm


the classic shape error, hmm, what have we got shapes 31 32 and


32 28 10 are incompatible. Where are we getting these shape errors from?


Ah four


So what's 28 plus 4 that's 32


You know what? I'm gonna have to introduce a new layer here. And this is a layer that you're gonna often need


so


It's worth


Exploring it a little bit. We need to flatten


Our data


And what does flatten mean i'm going to give it here input. What's our input shape of our data?


Input shape is 28 by 28. So the input shape we're going to pass it here as 28 28


This is telling our neural network that hey


We're passing you some images that are 28 by 28


And you might be wondering what does flatten do?


Well, let's explore it


We'll get a new cell here and go flatten model


equals


We're just going to create a model with a single layer


tfkeras sequential


And tfkeras


Of course if you want to skip ahead and just look up the documentation rather than listen to me talk about what the flatten layer


Does you can 100 do that, but I like exploring things by writing code


Because i'm not sure about you but I tend to read like


Documentations three different times and still don't understand it when I write the code. I start to understand it. Let's check the shape of this


What is happening here?


Ah


784 where did 784 come from i'll give you a little hint. What's 28 times 28


784 all right. Well


Now that we've seen the flattened layer in action


Now let's look it up in the documentation


So tensorflow flatten layer


Flatten layer


Flatten z input. Okay does not affect the batch size


Do we have a demonstration here?


Okay, so this is the original shape 1 times 10 times 64


And if we use flatten on that it's going to turn it into 1 times 10 is 10 10 times 64 is 640


Ah, I see


So what it turns it into?


Instead of being a 28 by 28 array it flattens it all so that it's now


of shape none


Because what you'll often find is a neural network likes everything to be


In one long vector and then we pass that through these other layers here and we get to the outcome that we like


so


Our data needs to be flattened


And we'll go here from 28 by 28


To


So if you ever run into a shape error


In your neural networks and you find that you haven't flattened your data into one long vector


Could be because you're not using a flattened layer as a very first layer of going into your neural network


some layers can flatten your data automatically, but


Typically, you'll need to tell your neural network that hey, here's the data. Here's my input shape


I want you to compress that flatten that into one long vector and then pass it through your other layers


So now that we've flattened our data, let's see if it gets rid of this value error the shapes that are incompatible


And again, as I said the shape error is one of the most common errors you'll come across. Oh, no


What have we got here value error shapes 32 1 and 10 are incompatible


incompatible


Hmm, you know what?


I think it's our loss function


You might be thinking daniel that loss function is the exact same that we have


In the multi-class classification in the architecture of a classification model what's going on?


well


There are two types


of loss function now one is four if your data is one hot encoded so if we go


What's our training labels look like train labels zero?


So our training labels are in the form of integers, so maybe we'll have a look at the first 10


And there we go, so nine zero zero three zero two


seven


two five five


Let's have a look at the documentation for categorical cross entropy


I'll just copy and paste that in there Computes across entropy loss between the labels and predictions


Use this cross entropy loss function when there are two or more label classes


We expect the labels to be provided in a one hot representation


If you want to provide the labels as integers, please use sparse categorical cross entropy loss


So that is where we're getting our value error from now


The shape error is coming because the loss function categorical cross entropy


Now I get confused with this all the time


But this loss function expects our labels in one hot representation


But if we change it to sparse categorical cross entropy, it should work


So let's try that sparse categorical cross entropy


Let's see what happens


Oh


Oh, yes, look at that and neural network is running


beautiful


So two little tidbits to take away from this


Is that our binary class classification model that we've used before this is very similar this model 11 to all the other models


We've been building throughout this entire section


Can work for multi-class classification data with a couple of tweaks namely defining what the input shape of our data is


Changing the output layer activation function as well as how many classes we're after


And updating the loss function to reflect the problem that we're working with and also the style that our labels are in


So I wonder if we can one hot tf.onehot


One hot train labels depth equals 10. What happens if we do that?


There we go. Oh, I wonder if that'd work


categorical cross entropy change that


and then we just


tf.onehot


train labels


Depth equals 10 and then we can do the same. We're going to have to do the same for


the validation


This is our experimenting on the fly here


Depth equals 10


Man we need an extra bracket here, don't we?


Boom our neural network starts to run as well wonderful. So let's put a little note down here


If your labels are one hot encoded


Use categorical cross entropy


And if


Your labels are integer form


use


sparse


Categorical cross entropy. This one has tripped me up a whole bunch of different times. It just takes a little bit of practice


so again


Oh typo there


If you get any shape errors with your models


The three things you have to look at input shape


Output shape


And the loss shape


And the loss function that you're using so they're the three main


Value errors or shape errors that you're going to come across. Of course, it could be more but they're the three that I


Most often run into so how exciting is that? We've already built our first multi-class classification model


Let's continue on with where we were in the next video


In the last video you built your first multi-class classification model


So you should be very proud of yourself. Give yourself a little pat on the back


But there was one thing we kind of forgot to talk about


We did code it but we didn't really explain it. Is this validation data parameter?


So you might have noticed a change in the output of our model training log here


If you see loss accuracy val loss val accuracy


You might have sort of figured out what the val loss and the val accuracy is if not, that's fine


The loss here with no prefix is the loss on the training data


So how wrong the model is trying to figure out the patterns between the training data and the training labels


The accuracy here with no prefix is that the model's accuracy on the training data


But the val loss and the val accuracy here is the model's loss on the validation data


And the accuracy on the validation data now this is important because this is the data that a model has never seen before


So it trains on the training data


And then it validates itself to see how good its patterns are on the validation data


So whenever you pass the validation data parameter with some kind of data, you're going to get these extra outputs here


And so this is a way to tell remember a model's results on the training data set don't necessarily reflect how it's going to perform


In the real world you really want it to perform well on data


It hasn't seen before to sort of get an idea of how it's going to perform in say your application or something like that


So right now our model is getting a score an accuracy score of 0.5


So we're getting a score an accuracy score of about 35 percent


Now that's better than guessing because we're working with 10 classes. So if we do 10 or 100


100 accuracy divided by 10


For 10 different classes that model was just guessing it would get about 10 accuracy


So, okay, we're getting about three and a half times that but let's see if we can improve it


first let's get a


Model summary check the model summary. I want to highlight something else before we move on to try and improve this accuracy


Is the input and output shapes of our model, which is a very very important point check the model summary model 11 summary


We go here


So we can see that the flatten layer takes our 28 by 28 images flattens them into a 784 long vector


It passes through these two dense layers. So dense layer one here dense layer two there and then it gets output


into a size of 10 for 10 different classes


So now


Do you recall?


Back when we were pre-processing data


We haven't done much pre-processing data with this problem because the data we got from the keras datasets module


The fashion ms data set is already numerical


However, we spoke about in a previous video the concept of scaling or normalization if you're not sure if you can't remember it


That's okay. We're going to go back through it


So we come back to our keynote. I deliberately left this arrow here so that you would think about hmm


What's done you're missing out on there?


Well, number one is turn all data into numbers. Luckily. We've already done that


Number two is make sure all the attempts to the right shape. Well, we've already been through that too. So


Tick tick number three is scale features


Hmm. What does this mean?


Normalize or standardize neural networks tend to prefer


normalization


So let's remind ourselves of what that is better yet. Let's remind ourselves of our training data check the min


and max values of the training data


Train data dot min and train data dot max


All righty


So zero and two hundred and fifty five


now


If we said in this slide


That neural networks prefer normalization


What is normalization?


Well, it's also referred to as scaling so let's write this down neural networks prefer data


To be scaled


Or which is also referred to as normalized or depending on what circle you're from


This means


They like to have the numbers in the tensors


They try to find patterns in


between zero and one


However right now our data is between zero and two hundred and fifty five


So how might we get our training data and we're going to have to do the same for our validation data


Well, we can do that by dividing all of the data by the max number


So let's have a look. So write this down


we can


get our training and testing data between


Zero


and one by


Dividing


By the maximum so this is referred to a scaling or normalization


Again, sometimes you'll find different names for different things


But if I use the word scaling or normalization i'm referring to getting our data set between zero and one


So let's go here train data


Equals train data and we'll divide by 255 as a float because that's the max value 255


Then we do the same for the test data


equals test data


Divided by


255 or actually what we might do to save the fact that we're not overriding our original variable. We'll do train data norm


And


Test data norm wonderful and then we can check the min and max


Values of the scaled


Training data


Train data norm dot min and train data norm dot max


So let's see how that works


Boom what do you think they'll be zero and one


Beautiful now that our data is between zero and one let's see what happens when we model it


So we're going to actually just use we'll change nothing


Absolutely nothing from model 11 except for the data that we're using


so the only thing that it will change is we use train data norm and


Everything else will stay the same so you can go ahead and try to replicate that before I replicate that in the video


But I want you to think now that our data has been normalized


It's one of the things that we can tune with our neural networks to make the performance better


What do you think will happen with a different data using the exact same model?


It's all right if you're not sure let's find out


so


Now our data is normalized


is normalized


Let's build a model to find patterns in it


So we'll come here


Set random seed tf random


set seed


And then we'll go create a model


same as model


We'll call this model 12 actually


A dozen models. How good is that tf carers dot sequential?


I'm gonna know how to do this off by heart


And then we're gonna need a flatten layer


To get our data from 28


flatten


Input shape equals 28 28 telling our model. Hey, i'm passing you


images that are sized 28 by 28


And then we'll create our two hidden layers


with non-linear activations


relu


tf carers layers


Dense


So that our model can find


Non-straight line patterns in our data and then the output layer


Needs a shape of 10 because we have 10 different classes and an activation of softmax


Because we're dealing with multi-class classification


And we can compile the model


We go model 12


dot compile


We'll set the loss function. It's going to be tf carers losses


dot


sparse


categorical cross entropy


Because why because our labels are in integer form


I say integer like differently every single time I say it


But if our labels were one hot and coded we just want to get rid of sparse


The optimizer is going to be tf carers


Activations our trusty friend adam beautiful and then the metrics is going to be


Accuracy, which is a great baseline metric for all classification problems


And then finally we can fit the model


We want norm history


So we saved our models training history to non-norm history. I said i'd come back to this variable


But this time we're going to use norm history


model 12.fit


on train data


norm


And then train labels is just going to be the same thing


We use 10 epochs


Equals 10 wonderful and then the validation data is going to be


The test data norm and test labels


Look at us building multi-class classification neural networks like a boss. All right


Do you reckon this will work? What do you think's going to happen before we run it? We've normalized our data


That's the only thing we've changed


Have a think about it


And i'm going to run the code in three two one


Hopefully no errors. Oh, of course


activations


Activations did I spell that wrong? Oh, no, this has to be optimizers. Oops, honest mistake


Again what have we got wrong here expected float 32 passed a parameter y got type string


hmm


Test data norm test labels


Oh, this is an error. I haven't seen before. What are we missing out on here?


You know what I believe it's just a simple brackets


Wow, yeah, so those are the type of errors you're going to run into just simple little things like that


That was kind of a lucky guess but you can spend hours sometimes troubleshooting them


So don't worry


If you get stuck on something like that because even I myself who have built a fair few of these models


Still run into simple syntax errors like that


But look what's happening here 10 epochs. We've run this just as before


What has changed between our two models the data we've normalized it. That's it


The val accuracy has shot up from zero point. What was it before?


Come back up here has shot up from 35


To 70 to 80 percent. So it was over doubled so almost 2.5 times as good now


All we did is we normalized our data


So that's something to keep in mind


So let me write this down as a key


so


And we'll get our key emoji out and we'll write here note


Neural networks tend to prefer data in numerical form


as well as


scaled


normalized so


numbers


Between zero and one


Beautiful. So just by normalizing our data. We got a fairly dramatic increase in performance


How cool is that? So that's something I want you to keep in mind


So in the next video what we'll probably do is we've saved our


norm history our model training history from norm history as well as


Non-norm history up here. So that's a bit of a tongue-twister to say


So if you want to you can have a go at plotting those loss curves between each other from the history variables


And see how they look compared to each other


We've trained two neural networks using the exact same architecture


But except one has or one was trained on non-normalized data


And then we said neural networks tend to prefer data in numerical form. That's definitely needs to be in numerical form


But they also prefer it to be scaled normalized. In other words the numbers need to be between zero and one


So let's compare the loss curves of each model


In other words the loss curves of normalized data versus non-normalized data


So we'll grab pandas. We'll import that


And then we'll plot the non-normalized data


loss curves


so pd


data frame


Non-norm history dot history and then


We'll go dot plot


Give that a title of non


Normalized data


Beautiful and then we can plot normalized data loss curves


pd dot data frame


And we want norm history dot history


or plot


title equals


normalized data


Wonderful. Let's see what this looks like


Wow


So from these two plots we can see how much quicker the model with normalized data


improved


Versus the model with non-normalized data. So have a look at this if we go here a model with non-normalized data the loss


Decreased and then it kind of flattened out


there


But with normalized data our model's loss dropped it even started at a lower value than what this model finished with


And then it kept decreasing as it kept going


And the accuracy who knows maybe if we train this for longer it will keep going the same with this one actually


But it would probably take a fairly long time to find the same level of results


That our normalized data found


Another key point to remember


Is when making comparisons of different models?


So we'll put down here


Where's our key emoji?


When comparing the results so we'll write down note


The same model with even


slightly


different data


So this is the same data set. All we've done is we've just turned it from non-normalized to normalized


so it's the same data that we're working with but


The same model with even slightly different data can produce


Dramatically let's get really dramatic here dramatically different results. So when you're comparing models


It's important to make sure you're comparing


them on the same


criteria


Eg same architecture. This is the comparison we're making here but different data or


same data


but different architecture


So that's something to keep in mind here when you're comparing


Comparing the results of different models keep your comparisons to only comparing as small variables as possible


What I mean by that is don't change like 10 things and then compare one model to the other


Change one thing and then compare the results from one to the other so, you know, what change is making the difference in performance


So with that in mind


When we're pre-processing data, let's come back here when we're getting our data ready for our neural networks


We need to turn it all into numbers. We already started with that with fashion MNIST


We had to make sure all of our tensors are in the right shape. We did that using the flatten layer


so the input shape to our neural network as well as the output shape of our neural network and


Number three scale your features. So normalize or standardize them. Remember neural networks tend to prefer normalization


With this being known, how about we try to tweak another one of our neural network hyper parameters


Let's see how we might find the ideal learning rate just as we have before in a previous video and see what happens to our neural


network training


We've seen how even slightly changing the data we input to our neural network can produce dramatically different results


How about we try?


Let's go in here finding the ideal learning rate and see if that changes anything


So just have we done before in a previous video


We'll set the random seed


So the ideal learning rate is the learning rate value where the loss decreases the most


So to do that, we're going to set up the exact same model that we did before


Create model. We're up to model 13 now tf. Kerr is sequential and


We had a flatten layer tf. Kerr is layers


layers


flatten


wonderful and the input shape


equals


2828 of course that will change depending on the data you're working with


Tf. Kerr is layers. We need a dense layer


I'm spelling everything wrong today. Activation is going to be relu


Again, not sure what's going on with the code editor. I want to keep our code nice and neat, right?


So if someone else had to read this they're not going holy gosh


What did you even write here Daniel? And don't forget when you're writing code that someone else could be you just later


layers dense we need a layer with 10 output units and


An activation function of softmax because we're dealing with multi-class classification. We can compile model


model 13 dot compile


Everything is going to stay the same as above loss function. We're using tf. Kerr is losses sparse


categorical cross entropy the optimizer is our trusty friend Adam


Optimizes dot Adam wonderful and the metrics we're going to use


accuracy


boom


And now we're going to fit the model. Oh


What do we have to do first? I forgot a step. I'm too used to going create compile fit. We need to


create


the learning rate


callback


so LR scheduler


Equals tf. Kerr is do you remember how to create a callback when you call back and we need learning rate scheduler?


And we're going to go lambda


the Epoch let's start at


What can we start at one a?


Negative three. Let's start at yeah negative three for now because our models already performing pretty well. So


We'll start there a park divided by 20


so that just means remember this just means start at this value here and slowly increase the learning rate every epoch by


10 to the power of epoch divided by 20 and


To fit the learning rate we can go fit our LR history. We'll create another history variable


model dot underscore 13 fit


we're going to use trained data will fit on the normalized data and


train labels


when we go epochs go about 40 this time and then


the validation data of course is going to be


test data norm and


test


Labels and then finally the callbacks is LLR


scheduler


Wonderful, so this should work fingers crossed


Who there we go?


okay, so


This is going to take about three seconds per epoch. So what's that times 40 120 seconds?


So I'll come back. I'll speed this up and I'll come back once our model has finished finding the ideal learning rate


Alrighty looks like our model has finished. Let's see what's happened here


Okay


So it looks like it ends up on a pretty not as great validation accuracy as what it once was


We get a pretty good range of values here, but let's not look at just this training output. Let's how did we do it before?


We plotted the learning rate decay curve. So let's do that again plot the learning rate


decay curve


So we want to import numpy as MP


Import matplotlib


We've already got these in our notebook as before however, if you were continuing on from this


We'll just reimport them just for completeness


LRs equals what did we do before 1e negative 3?


times 10 to the power of


NP a range


40 because we use 40 epochs divided by 20 and actually we don't even need to use numpy


Let's just use tensorflow stick in the spirit of using tensorflow


And we're going to plot we want to semi log X because we want our learning rate to be on a log scale


LS and then find LR history our history variable from up before


dot history and we only want the loss


component from the history and


Then we should make our plot pretty so we'll add X label. This is a learning rate and


the Y label is going to be


the loss and then the title can be


finding


Finding the ideal learning rate


Let's see what this looks like alrighty


So we can see when it started off at 10 to the power of negative 3 the loss decreased


Fairly sharply and then it kind of plateaued all the way up to 10 to the negative 2


Then the loss just sharply increases as it gets closer to 10 to the negative 1


Hmm now if we come back to our keynote


Where is our finding the ideal learning rate slide there we go?


So the ideal learning rate is where the loss is decreasing sharply


Find the lowest point on the curve and then go back a little bit


Alrighty, so let's do that


The lowest point on the curve is about here


And if we went back a little bit to where it's still sharply decreasing. I would say you know what 10 to the negative 3 is


10 to the negative 3 is probably our ideal learning rate which happens to be


What optimizer are we using we're using Adam?


So as I said, this is proof look at that


That the default parameters for a lot of the different optimizers and other functions in tensorflow are pretty darn good


So looks like for our problem in particular the ideal learning rate is just the default value for Adam


So with that being said for completeness. Let's refit a model with the ideal learning rate


So let's refit a model with the ideal learning rate


with the ideal learning rate


And we'll go here set random seed


tf random dot set seed


Don't want the out symbol getting ahead of myself pressing the chef key shift key


All of the keys are chef keys because what are we doing? We're cooking up neural networks. That's what we're doing


tf carers sequential wonderful


Now we come over here


tf carers we have to flatten our data from


28 by 28


arrays or tensors


into a vector of 784 in other words 28 by 28


And we go tf carers dot layers dense


Just the exact same as what we're doing before


Activation equals relu


We'll do another hidden layer


layers dense


for activation equals


Relu


wonderful


And then we can come down here tf carers output layer. What does our output layer need?


How many classes do we have?


We need a hidden unit for every class and we also need an activation function that is not sigmoid Daniel. Come on softmax


Thank you for catching me on that one


And we're going to compile model


Model dot compile


What's our loss function tf carers losses we're dealing with


Integer values so we need sparse categorical cross entropy. When would we use


Just categorical cross entropy if our labels were in one hot encoded form


Now our optimizer


We don't even we actually don't even have to change anything here with our optimizer


But we will just because we spent all that time finding the ideal learning rate. So


We might as well put it in there


Lr equals 0.001


So again the default learning rate per atom is actually 0.001 we can check that by looking at the doc string


Wonderful, and then we can set up metrics equals accuracy


And we can fit the model


We can fit the model


Let's save this to just history 14


So we're keeping track of what we're doing


model 14 dot fit


train data


train labels


Epochs this time. How about we go 20 epochs? We found kind of what the oh no


We forgot the normalized data here because remember our model performs much better on normalized data


We'll go the normalized data


We've got the ideal learning rate, which we actually didn't have to change anything for atom, but we'll fit for a bit longer this time


Because we've worked out that our model is doing pretty good


So maybe this time we'll the thing that we'll tweak is how long it looks at the data for


validation data


Equals test data. We need to do this as a tuple actually test data norm and test labels


Wonderful, let's fit that shouldn't take too long


Invalid syntax of course. Oh, did you catch that? I didn't catch that


Wonderful, so i'm going to let this fit


But this should turn out to be a fairly


Well trained model with close to the ideal learning rate and performing pretty well. So we've got a couple more options


Based on what we've done in the previous lectures


We can evaluate and improve our classification model. So we've done a bit of improving


So it's probably now time to once this is finished fitting


Is to start evaluating it with some techniques that we've used before


so


I'll let this run through and then i'll see you in the next video


We'll start to to run some more evaluation methods on our multi-class classification model


Welcome back. So now we've got a model train with a close to ideal learning rate and performing pretty well


Let's check back in with our workflow and see what we should do next


So we've got build a picker model


We've kind of done that we've definitely got our data ready turned it into tensors. We fit the model to the data


We haven't quite made any predictions yet


But making predictions is kind of synonymous with evaluating the model because again what happens during evaluation


We compare what the model should have predicted with what the model actually did predict. We've actually already improved through experimentation as well


So this is another key point to highlight here is that we've got a model that's already been tested


Another point to highlight here is that although this is kind of like a linear step through way to do things. This is


Just like a rough overall guideline to steps in modeling with tensorflow


You can always jump back and forth through different steps and suit them to whatever problem that you're working on


With that being said, let's make a little heading here


Evaluating our multi-class classification model


And we'll turn this into markdown


And let's put in here a couple of things that we could do


Or actually make this sound a little bit better to evaluate our multi-class


classification model


We could


What are some things that we could do?


hmm


evaluate its performance


using


Other classification


metrics


Such as a confusion matrix


Assess some of its predictions through


Visualizations


That's always a fun one. We could improve its results


by


training it for longer or


Changing the architecture


Now this isn't really evaluating but that's just another step that we could do


Now and then of course referring back to our steps with modeling in tensorflow


We could save our trained model so that we could use it later. So we'll put that there just so we know save and export it


for use in an application


Wonderful


How about in terms of sticking with evaluation?


We go through this and i've made a typo assess should have a double s on the end. We go through the top two


So we'll write down here


Let's go through the top two


So the first things first is create a confusion


matrix


Now i've actually got some code up here that we did with the binary classification problem


Evaluating and improving our classification


That should be


model, but let's find


Where's the confusion matrix code that we had before?


It should be somewhere up here


Far out we've written a lot of code. Here we go


I'm back again


You've reached the time of the part two video


Where i'm going to tell you about a special code that you can use


To sign up to the full course that you're watching 20 plus hours more content on zero to mastery.io for 15 off


So if you do go to the website and go to the academy page where you sign up and you enter the code


T flow for tensorflow, it's a shortened version of tensorflow t flow t f l o all capitals


You'll get 15 off but keep this a secret between you and me


You can share it with your friends


But don't leave a comment on this video because we want to keep it a surprise for people who've made it this far


But that's enough from me. Um, i'll probably only see you one more time by the end of this video. I'll say goodbye to you


Enjoy


So this is our remix of scikit-learn's plot confusion matrix what we might do instead of typing that all out again, I might just


break my rule here of writing code and i'm just going to


copy everything that's in this cell


And i'm going to come down to evaluating our multi-class classification model. This is why it's handy to make headings here


You know, so really lays out your work so you can just jump between different sections


Create a confusion matrix


And have an idea


Why don't we functionize this because I kind of issued that as a challenge in a previous video


It'd be cool if we could actually go through it together. So from sklearn metrics


Import confusion matrix just in case we don't have it


We want this to run as a a standardized cell


now


To functionize this. What should we call it? Let's call it make confusion matrix


matrix


And because the whole premise of an evaluation function or a confusion matrix in general this function here


Is comparing the test labels with predictions?


At a bare minimum our make confusion matrix should take in


test labels, but I believe


Scikit-learn calls them


Is it y true?


Yeah, let's make it y true. So we stick with theirs and is it also y pred?


Yeah, y pred y true


And then the classes we're going to set that equal to none


Because if we want a list of classes we can set it there


And then what else should we set?


We've got the fig size. We've hard coded that let's hard code that into here


10 10 wonderful


Then what else is there?


Well, we could do the label size


Label size


Why don't we just do the text size


Text size equals we'll set that to 15 by default. Okay, that should be some good parameters there


We'll have to tab all this across


Tab so it's part of the function


And then now what do we have to update here? So confusion matrix


This is now going to be


y true


And then our preds are going to be


Y pred


wonderful, and then


Fig size equals fig size that works there so we can get rid of this


Yes. Yes


wonderful classes equals we're gonna have to set that to


I don't think we need that anymore


So


Set labels to be classes


So if classes exist


Now the label should be the class names. Yes


Else they don't exist. It should just be a range of how big our confusion matrix is


wonderful


Anything else we need to change?


We could change these to text size


Text size


That way I know I set it as 15 at the top but we'll just make all the text a very similar size


Why don't we do that? That should work text size and we can adjust it


Size equals text size. Okay. Now all the text should be the exact same size


Beautiful. So now we've got a function make confusion matrix, which takes in a bunch of true labels and a bunch of predictions


And classes that's what's important


So let's run this


Make sure the syntax is correct remind ourselves of what our class names are


Okay, beautiful, but what don't we have


We have a function


make confusion matrix


And we could have really put some doc strings here to tell us what the function does, but that's all right


I'll leave that for you to explore


We have some true labels. So this is our test labels, but we don't have any predictions with our model yet


So let's make some predictions


make some


predictions with our model


with our model


And we're going to create yprobs for prediction probabilities. You might be wondering what does that mean?


We kind of covered in a previous lecture, but we're going to reiterate here


So to make predictions, we just go model14.predict and then we want to make predictions on the test data


And i'll put a little note here probs is short for prediction


probabilities because


We've got the activation function of our output layer


The outputs of our models predictions


Are going to be prediction probabilities


So view the first five predictions, we're going to have a look at yprobs


Oh, what do we get here? Why are these like that? Hmm


So has it already rounded it for us


That is interesting


so this is


Hmm did we set our model to have


Let's come back up here


train labels test labels


sparse categorical cross entropy


Yes


Hmm that means it's outputted what label it should be


I'm going to pause the video here and inspect this and then come back to see what's going on here


now


That took me a while, but I figured out what went wrong here


So I said that probs is short for prediction probabilities, but as we see here we get whole numbers here


Now you might have spotted what went wrong


Our model predicted on the wrong data


Now this is a very important point to highlight here


Is that we want our model to predict on the same kind of data that it was trained on?


Now what's the key difference here? What did we do before with test data?


And test data


norm


That should be the hint as to what we got wrong just before


So here's our test data and our test data normalized


What did we change?


Well, let's have a look at this first example here test data zero and test data norm


What did we do?


We normalized it remember?


So our test data samples are


Test data samples still have values between zero and two by five


Whereas our test data normalized have values between zero and one


So that's why we get interesting outputs from our model when we ask it to predict on test data non normalized


But if we passed in test data norm


Which is the variable we created before?


We get the outputs that we should get


So that's a very key point i'm going to write a little note here even this is a reminder to myself


Let's get a key emoji in here


note


Remember to


Make predictions on the same kind of data your model was trained on


Eg if your model was trained on normalized data


You'll


Want to make predictions on normalized data


Beautiful. So what are prediction probabilities? So these are different numbers. Let's get the first one actually yprobs


We'll get the zero


So the highest number here indicates the index so we'll get our class names back up


Indicates the index that our model thinks is most likely the value


So I believe it is this last one maybe so y


probs


We can use the arg max


Let's use tensorflow tf arg max for arg maximum


Which is going to give us the index where the maximum probability occurs. So all of these are values as to how likely


The sample zero is t-shirt or top or trouser or pullover


So the first one here is t-shirt or top. So that's a fairly low number e to the negative 11


This is even lower. So it's definitely not trouser or at least in the model's eyes. It's definitely not trouser


So it seems that the highest value is this one here, which is it should turn out to be about 0.8


So that's the last one


So our sample zero our model is predicting as ankle boot


So let's have a look here


So numpy 9 and then if we take that index tf arg max, let's get yprobs


zero


And then we index on our class names list


With that we should get ankle boot


Wonderful. Okay. So now let's turn our prediction probabilities


From this for prediction probabilities array into


integers


So we can do that by going


And get rid of this


Now convert all of the prediction


probabilities


into


So let's go ypreds


equals


yprobs arg max and we'll go it on the first axis


And then we will view


the first 10


prediction labels


ypreds


Boom so now our predictions are in the same format as


Our test labels. So what can we do now?


Well, we can compare the two


So I will leave that as a challenge for you. But in the next video


We're going to use our make confusion matrix function


To compare these two and how about you could try another metric maybe look into


Creating an accuracy score as well. See if you can reproduce the accuracy score we got from model 14 evaluate


By comparing these two


Wonderful now that we've got our models predictions in the same form as the true labels. Let's create a confusion matrix to


Evaluate our models predictions, but first we're going to create


a boring confusion matrix


using scikit-learn


Just so we can demonstrate how valuable our make confusion matrix function is


confusion matrix


y true is the test labels and


ypred


Is ypreds


So, let's see this


It's kind of hard to tell what's going on here, but with our confusion matrix, we know that down diagonal


We should have the highest numbers. So looks like our model's performing pretty good across all the classes


So the highest numbers are down the diagonal now, let's remind ourselves of what an ideal confusion matrix looks like


So the correct predictions the true positives and true negatives are down the diagonal


but right now


This is pretty hard to interpret. So if we were to send that to someone they're probably going what the hell is going on here


And this is where our pretty confusion matrix comes in. So let's make a prettier


And we're going to make confusion matrix, yes, yes, yes


y true


equals just the same as before test labels


ypred equals ypreds now we have classes


Is going to equal class names


That's our list of class names of the fashion m nest data set then the fig size


It's hard-coded as 1010


But let's make it 15 15 just to have a little practice and then tech size equals 10


Let's see what happens


Ho ho


Look at that. I think I'm gonna have to zoom out because that's a that's a big dog


Or maybe we change this to 10 10. See what happens might be the text is too big now. Yes, it is


So 15 15 was a good size


So let's go back to that


alrighty


This one looks much better than before


So this is where you can really see the power of a confusion matrix


See how visual this is like you could you could send this to someone and they'd be like, okay


They'd start to intuitively figure out so the true label here. Let's explore this for this t-shirt top. So


Again, the darker the square the more predictions the model got right for this class


However, if we come over here


This way is pretty dark. I mean 16% of the predictions of t-shirt top were shirt


Okay. Well, does it make sense that our model is getting confused between the shirt and the t-shirt top?


I mean to me, how do you decipher those, you know?


And then where's another one? What else is our model getting confused with so pull over?


Okay, again the darkest squares down the diagonal. That's good, but it's also getting confused a lot


With the coat class, so it's predicting coat when it should have been pull over


188 different times now are they similar?


Now we're gonna have to explore that in a second


But you can kind of imagine pullovers looking similar to coats and what else is getting confused on ankle boot and sneaker


Okay. Well at least for ankle boots, they're a shoe at least so a sneaker is a shoe so that kind of makes sense


This is a kind of thing that you'll you'll be doing with your own problems


I see the shirt class gets confused a lot with a t-shirt top pull over


it's to explore where your model is making errors and


What you can do with this information is start to improve your models


so maybe the t-shirt top class should actually be


incorporated with the shirt class or


Maybe we need some more data of actual just shirts so that our model really learns to differentiate it between t-shirts and tops


but anyway


Investigate this confusion matrix a little bit more. It's nice and pretty in the next video. Let's go back through


What did we say we're going to do to evaluate our model?


Let's get visual a little bit more


Assess some of its predictions through visualizations. So the confusion matrix is one way to visually explore it


But it's something else to actually look at a picture and look at the label that our model predicted versus the true label


So let's write some code to do that in the next video


We're gonna start this video off with a little key


So let's go here. Okay


note


So what's our motto or one of our mottoes is visualise visualise visualise


so often when working with images and other forms of


visual data


It's a good idea


to visualise as


Much as possible to develop a further


understanding of


of the data and the inputs and


outputs of your models


So I've also discussed the power of randomness when exploring your data. So how about we create a fun little function for?


Let's write this down so we don't forget how about we create a fun little function for


Hmm what should we do we should plot a random image


we should make a prediction on said image and


we should label the plot with the truth label and


The predicted


Label. Yeah, this is a great great way to evaluate our model


So let's do that first. We'll need random from Python


Because we want to plot a random image now we'll create a function def plot


random image I


Often create a lot of these little helper functions


If I want to do something over and over again


I make sure to functionize it so that I can use it multiple times. So we'll pass it our model


We're right now. I think we're up to model 14


We'll pass it a list of images. We want to inspect


what the true labels are and


the class names


Beautiful


So we'll make a doc string here. So it's a little bit complete picks. What does it do clicks a random image?


plots it and


labels it with a predicted


With a prediction that will do and truth label


Wonderful


so


We need to choose a random number


setup


random integer


We'll go I equals random dot


Rand int


Between zero and Len images. Well, we need a comma there


So that makes sense


So I just going to be a random number between zero and the length of images and images is going to be the images


We want it to look at


Wonderful, and now let's create


predictions and targets


The target image is going to be images. I


Will just index on our random number and then the Pred


Probs is going to be model dot predict. So it's just going to take our model here dot predict on target image


dot


reshape


One twenty eight twenty eight so we'll make sure it's in the right shape for our model


Because we're only predicting on one image here


So our model right now is trained on images in twenty eight twenty eight size, but we're telling our model


Hey, we're only passing you one image at a time this time


Then we're going to do a pred label equals


classes


Pred at probs and we want to get the arg max


Wonderful and then the true label is just


true labels I


Beautiful


What else do we need? Oh, yeah, we need to plot the image


So plot the in show


We want target image to be the plot and we'll set it to a binary color map


So it's just black and white


And beautiful and now let's do a little something a little bit fancy here


All right, we're gonna change the color of the titles. So change the color of


the titles


depending if I could type correctly depending on if the prediction is


Right or wrong, so how might we do that?


Let's just create a ball lane so if pred label


equals true label that makes sense


So if the pred label is the same as the true label


So the prediction our model has made is the same as the true label. Yes, we want to set the color


green


because green means good and


else


the color equals red


Wonderful and then let's add


X label information and we want prediction


true label


Let's go PLT dot X label and how can we do this?


Maybe we just do it with a dot format


pred


What's the pred


pred and


Then we want to change this to let's put the confidence in there. Why not?


to point F I


Always get confused when I'm typing out this little section of code


There we go. And what else do we want? We want the true


So the true is just going to be like that


And then check adult format on here. We could have done this as an F string or could we?


Because we have to have a calculation. Yeah, we probably could have oh, well, that's all right


This will do for now. We want a hundred times for the confidence. We want TF reduce max


so in other words find the maximum value in pred probes and then


We'll just finish off with true


Label and then I believe color goes down here color


Equals color. We've definitely got something wrong in this function, but so set the color to


green or red


Based on if prediction is right or wrong. So


This makes sense


Plot random image pass it a model picks up a random integer picks a target image


from images


Then uses the model to make a prediction on the target image. We have to reshape it to be because it's only one image


the pred label is going to be


classes dot pred probes


Ag max so the index of pred probes which has the highest prediction probability


The true label is going to be the true labels indexed on I


Yep, that's correct plot the image target image. See my binary. Yeah, yeah, yeah, we could keep going through that


But let's just run the code see if an error comes up. Oh


I know we're just defining the function there. We haven't actually run it yet. Let's do this


Check out a random image as well as its prediction. Oh


Yeah


plot random image


Model equals our current model model 14


images equals test data


Because that's the test images. We want to work with test or true labels


sorry equals


test labels and


then classes is


class names


How does it look? Oh


Nice. Okay. There we go. So the prediction is code 100% So that's a very high prediction probability


But the true is shirt. Does that look like a shirt to you?


Kind of let's look at some more


Ankle boot 100% true sandal. Ah


Is that an ankle boot? I mean, that's a fairly pixelated image. That's kind of hard


Come on, surely our model got something right prediction coat true is actually shirt


Wow, that's a bit see that's that one's on the edge to me as long as our functions, correct? There we go


There's one that's right t-shirt top 100% true


Wonderful. Oh, what do we also forget here? This should be norm


Come on Daniel


Remember always make predictions on the same type of data that your model has trained on and right note here always make predictions


on the same kind of


Of data your model was trained on and by the same kind I mean pre-processed in the same way


All right ankle boot yes, that's what we want green show me some more green. Ah, damn it


T-shirt top. It's a shirt. See this what I mean


I'm not sure who created this data set but to me t-shirt and top is kind of the same thing as shirt


Maybe it's not maybe it's not


Trouser yes, that's what we want shirt predicted of the true as a code


Kind of so as you see here you keep going through this for a lot


We could have really just set this up to plot more than one at a time


So we don't just have to keep sitting and to rent


Oh, maybe that's a little bit of an extension for you is to see how you could function eyes this to plot


Say four images just like we did before and we're exploring our data


So going through this I would say do another 20 or so go through them and then figure out


Does visualizing these predictions here help you to better understand this confusion matrix here?


And what I mean by that is does it start to make sense where the model gets confused in other words?


Like does the overall shape of a trouser?


Relate to pants or does the overall shape of an ankle boot is the model getting mixed up with a sneaker because an ankle boot is a similar shape to a sneaker


Does that make sense?


So we've done a bit of evaluating for our model. How about


The next video we've talked a lot throughout this entire series that our model is learning patterns


So how about in the next video we discuss what patterns?


Exactly is our model learning


So have a little bit of a play around with our visualization function here


See if you can start to get an idea of where the model gets most confused and compare that to the confusion matrix


And then I'll see you back in the next video and we'll check out the patterns


Our model is learning to make these kind of predictions. We've covered a fair bit


Well, actually fair bit is probably an understatement. We've covered a lot in this series


But this whole time we've been talking about how neural networks learn patterns in our data so that we can use those patterns later on


But what exactly do those patterns look like? Well, let's use this video to find out


so first we're going to


Crack open one of our models. So find the layers of our most


recent model


So if you're not sure by now


Take this as me stating it straightforward is that a deep learning model is constructed of layers


And each one of those layers has a specific role in finding patterns in the numbers that we feed it


So let's have a look at model 14 layers


So we can see we start off with a flattened layer and then two hidden dense layers


And then an output dense layer because we typically go from this direction


So top to bottom this is input and this is output and now we can inspect what's going on in a target layer


Using indexing. So how about we extract a particular layer?


I'm going to go with the first hidden dense layer by indexing on one


Wonderful so there we've got an individual dense layer there and now


We can find the first hidden dense layer


Individual dense layer there and now we can find the patterns learned by a particular layer


Using the get weights method. So let's try that out. So


Get the patterns of a layer


In our network


So we're going to set this up as weights and then biases we'll have a look at that in a moment model 14


layers


one dot get weights and you can just search this method up if you want


But I prefer to figure out things ourselves and then we go shapes


So we'll view weights and then we'll also view weights dot shape


Okay, so this is what the internal patterns of a single layer or


Specifically the first hidden layer of our neural network look like so to you


They might just look like a whole bunch of random numbers


And I mean to me that's what they look like but to our neural network it considers these as


The patterns which contribute to the decisions that it makes so we're working with


fashion MNIST data so the shape of


this weights matrix


corresponds to


The shape of our input data. So remember our input data was a 28 by 28 image


That's where this number comes from. So if we go 28 by 28 and


Then we remind ourselves of what our model looks like what our model input shapes look like


So if we see here


this is a 784 the flattened layer to begin with and then this 4


Comes from the number of hidden units in our first dense layer


So that means that for each


Data point in our input tensor


our weights matrix has


4 because see here 4 has 4 numbers that it starts to learn and adjust


to find patterns in these 784 numbers


So just looking at this off face value


Can be very confusing and don't worry if you're not sure of it to begin with


This is the first time we've ever cracked open one of our neural networks and this is kind of where the term deep learning gets


the idea of black box from as in


When you crack open a deep learning model you get all these random numbers if you or I would try to interpret them


We can't really interpret what's going on here, but somehow they correlate to our model finding patterns in the input data


now


What each one of these values does so each value in the weights matrix it?


Corresponds to how a particular value in the data set


should


influence the network's decisions


Now you might be wondering how does our neural network even create such values?


How does it learn these values well and let's go back to the keynote to see a high-level overview of how this might happen?


So we're working with the input data here, which is grayscale images of the fashion m nest data set


So we have coat ankle boot shirt sneaker


Wonderful, and then we might encode them to well we should have actually normalized this data


Shouldn't we we might transform them into a tensor?


To pass it to our neural network, and then what's going to happen in our neural network?


We've seen this kind of overall schematic before is that it's going to learn the representation


Another word is patterns features weights these can all mean similar things when you're hearing our neural network talk


For now we're referring to them as weights, but I have been referring to them as patterns


And so what's going to happen is at the beginning our neural network is going to initialize itself


Automatically with random weights at the very start so if we come back


So all of these numbers the weights the internal weights of one of our neural network layers are going to start off as random


Numbers just completely random and it does this we can look at if we go to the tensor flow dense layer


It does this


using


Which parameter?


There we go


kernel initialization


Now a little bit of extension on this video is that you can read into what this is glow root uniform


But it might actually tell us here


Kernel initializer initializer for the kernel weights matrix so glow root uniform is a form of randomness


So just take that from here. We're not going to dive into what glow root is but just understand that


It's a form of randomness now if we come back


So it's going to initialize itself with random weights in each particular layer, and then what's it going to do?


Well, we're going to show it different examples of the data


We'd like it to learn so we might show it images of coats images of ankle boots


shirts and sneakers and it's going to keep looking at these and


Slowly update its representation outputs or weights and biases


Into a different kind of tensor which has come back to here


If we come back to our notebook, this is what it's going to learn over time these representations as we


continually


Repeat with more examples. Does that make sense our neural network?


We feed it input data. It starts off with random numbers random weights random patterns


However, you want to refer to it and then as it looks at more and more examples gonna go


Hey, I'm going to try these random numbers and see if they correlate to any of the patterns in the data


And if they don't well


Just get the neural networks going to try and correct itself


Thanks to our optimizer Adam and then we're going to repeat it with more examples as we have more data


Then it's going to slowly adjust these patterns to better suit the data as best it can


to eventually hopefully it starts to output all correct predictions. That's the ideal case


Let's come back


We've only talked about the weights matrix so far, but alongside a weights matrix is also a bias vector


So let's come down here. Let's have a look


Now let's check out the bias vector


Let's go bias and


Biases shapes. We'll go biases and


Biases dot shapes


So again, this is from a single layer


More specifically the first hidden layer in our current neural network. So we come down


What does this correlate to?


Number four, so this is a bias vector. There's only four here


So it's this four here


So that means for every hidden unit in our neural network in the first layer


It has one bias vector whereas for a weights matrix


This is the difference the key difference between a bias vector and a weights matrix a weights matrix has one value per data point


Whereas a bias vector only has one value per hidden unit


And so what the bias vector does is run this every


Neuron has a bias vector


each of these is


paired with a weights matrix


and so the bias vector also


Gets initialized but this time let's have a look up in the dense layer. What is our bias?


initializer


Zeros so you can kind of intuitively guess what zeros mean


If we come down, what does it say? How does it get initialized initializer for the bias vector?


gets initialized as


zeros


at least in the case of a


TensorFlow


Dense layer now where it kind of gets tricky, right?


Is that sometimes depending on what layer you're using in deep learning? So within the tensorflow carers layers module


Your weights matrix or the kernel initializer and the bias may be initialized differently


However, as you can see we never actually set these variables. They got initialized by themselves


So this is what I'm saying a lot of what tensorflow does for you


The majority of calculations are done behind the scenes


So of course you can dive deep into this as much as you want and I actually am big advocate for that


but to begin with I just want you to focus on writing as many neural networks as possible and


Just getting them working and then once you want to know more start diving into the nuts and bolts of what's going on behind the scenes


so


We've said what a bias vector is now, what does it do? So the bias vector


dictates how much


the patterns within


the corresponding


weights matrix


should influence


the next layer


Okay


So for every hidden unit, there's a weights matrix and a bias vector


So if we change this to 10


How many weights matrices would there be and the same thing for this one?


How many bias vectors we change both of these to 10? How many weights matrices would we have and how many bias vectors we have?


Just have a think about that for a second. So this is a key point here as well


How much the patterns within the corresponding weights matrix should influence the next layer?


Another big thing about deep learning. Let's read familiarize ourselves with our model architecture. So


model 14


summary


what it looks like


Okay, so we've built a few of these we've built a few deep learning models


So now's right about the time to point out the whole concept of inputs and outputs


Not only relates to the input layer of a deep learning model and the output layer


It relates to every single layer within a model


So let's go to the keynote you go to the next slide. So this is inputs and outputs layer by layer


So if we imagine this is our model here very similar to what we've just built model 14


And if we we can create this in a second


But if we imagine we have this input layer that takes in our input data in our case, which is images 28 by 28


Into a tensor we should have normalized this if we were really preparing our image data correctly. So this is going to take the inputs


And they output it to the flatten layer


The flatten layer outputs to the dense layer or the first dense layer


The first dense layer outputs to the second dense layer


And the second dense layer outputs to the output layer


So for each layer in a deep learning model the previous layer is its inputs


So this is the crux of deep learning. This is what we call the deep learning model


This is the crux of deep learning. This is what makes deep learning deep is that


You have multiple layers and of course as you added more layers the deeper the neural network will go but each subsequent layer


Does its part to find patterns in the original data and then feeds it on to the next layer?


So as you keep going through the patterns get more and more refined towards the ideal or hopefully they get close to the ideal


Outputs that you're after so let's have a look at how to replicate this


And I believe that will probably be more than enough to wrap up this video


so


Let's check out


another way of


viewing our


deep learning models


So we can do this with from tensorflow.keras


Utils import plot model


And see the inputs and outputs of each layer


Plot model


Model 14 and show shapes equals true


Beautiful so this is what we have


As I said, this is inputs and outputs layer by layer


So it starts off the input the none here is for batch size. We'll tackle that in a future video


Students can look that up if they want. Let's just say we put in 28 by 28 images. So that's the input then this layer outputs


The inputs to the next layer and then this layer outputs the inputs to this following layer


And then so on and so on and so on until we get our idealized output


Which is this 10 shape here because that's how many classes we have


Now we've covered all of this


Very quickly of what's actually going on. I just wanted you to get familiar with cracking open one of our models


Now if you want to dig deeper on what's going on behind the scenes here and how these calculations actually happen layer to layer


I've put some resources in the extension section


But otherwise, I think that just about wraps up


everything in the introduction to neural network classification with tensorflow


so


Go back through


Pat yourself on the back because we've covered an incredible amount take a break to let things sink in


And when you're ready, make sure you check out the exercises. So i'll just put this here next check out


exercises and extra


curriculum


Check out the exercise and extra curriculum to practice and cement what you've learned


I've put in some stuff there to not only practice everything that we've gone through that's in the exercises


But as I said before the extra curriculum will help you really dive in to what's going on behind the scenes here


So with that being said congratulations on finishing section two neural network classification with tensorflow


I will see you in the next module


Holy bejeebus


If you've made it all the way to this endpoint and you followed along you've gone through over 14 hours of video on youtube


And I trust you've learned a fair bit about tensorflow a fair bit about how to write neural network code


to solve regression problems in other words predicting a number and


Writing neural network code to solve classification problems both binary classification and multi-class classification


Which is some of the most common problems in the field of machine learning


so this is a


Me signing out to you and you're probably wondering has he worn the same shirt through all of all these little clips?


And yes, I have it's the magic of cinematography


But this is a big props to you. Thank you so much for watching through this


If you've loved it, please let me know in the comments if you have any questions


Remember the github discussions page is your friend


Otherwise if you'd like to sign up to the full version of this course and learn a whole bunch more about tensorflow


And deep learning please do so


I'll be a link below sign up to the zero to mastery.io academy and


If you've seen the other videos, you'll know what code to use when you sign up


That's it from me. Congratulations. Happy machine learning. Happy deep learning and all the best learning tensorflow in the future


Data Engineering on Google Cloud Platform Specialization Progress | Learning Intelligence 47

We're off to our first lab of the Google Cloud Data Engineering certification, otherwise


known as Serverless Data Analysis with BigQuery.


Now, you're down there, I've been standing up.


Let's come down to the same level.


Sounds like the standing desk needs a bit of oiling.


Now, first of all, I want to give a big shout out to Coursera for sponsoring this video.


I love Coursera because they partner with people like Google.


Just have a look at this.


Got that little Coursera.


You see that Coursera for Google symbol there?


I'm going through the Serverless Data Analysis with BigQuery.


I even said it before, I can't remember it here.


What exactly is BigQuery?


Well, according to the formal definition, it's a no ops data warehousing and analytics.


Now, I just read that because I purely copied it down.


To me, I understand it as it's a place where you can, if you have a big amount of data,


you can send it up to the cloud.


If you want to store it and leave it there in a safe manner, you can.


It gets cheaper over time if you store it there for a longer period of time because


Google do something smart with their storage.


If you're not accessing all the time, you don't have to use that electricity and all


that sort of stuff.


Storage gets cheaper over time, but it means that if you want to do analysis on that, you


can run some queries on it, which is literally why it's called BigQuery.


You can do it fast and you can do it effectively with SQL, SQL-like queries.


Why is that so handy?


Well, the fact is that it's serverless.


To me, I don't know anything other than data warehousing that's serverless, which means


living in the cloud, you don't have to necessarily boot up a computer to run stuff on it.


Whereas in the past, you'd have to have your own-


Sorry, Daniel.


There's nothing to repeat.


Siri is just going crazy.


Anyway, in the past, my phone should be in the drawer.


That's a note to self, right?


Keep your phone in the drawer.


In the past, you'd have to have this big computer on site and run all this data stuff.


You'd have three different people to do a job that you and I can do right now in the


bedroom, I guess.


But we're going to jump into the first lab.


Now with this course, the reason why I love it is because it's structured labs.


It actually shows you what to do if you actually want to start running it on your own data.


I'm planning on getting three labs done.


There's six labs in total, three labs, and then break, and then we'll do another three


labs.


Oh, yeah.


I think I'm going to try that.


We just did our first- It's really dark in here.


I think there's a storm brewing.


We'll get to that in a second.


Actually, there's enough light for this.


We just finished our first quiz on the Google certification data engineering.


We've been through three labs, so we've done the goal.


We've done the goal for now.


It's break time, I think.


Lab one, you can see, was building- I think that should be and?


Building a BigQuery?


Lab was loading and exporting data.


How do you get data from your computer to upload it to BigQuery?


You can do it- The two methods we went through was the command line interface, so terminal,


and then otherwise just that regular upload button.


Then it was a lab on advanced capabilities, so advanced SQL queries.


I'm finding SQL seems a bit like Python and Pandas, right?


It's more to me seems- I don't want to say simple.


Simple is not the right word for it, but it's like you're literally going select this from


here, where this, and group it by that.


I mean, it's similar.


There was a saying when I went to Bali once.


When you go over there and you buy a t-shirt, but it says it's a Supreme t-shirt, but it's


not Supreme.


They would say it's same, same, but different.


That's what I feel SQL is.


I think we've got a storm coming.


Check it out.


That sky is looking gray.


It's looking dark.


The rains are coming, or actually the rains are kind of already here.


Seven, you got a new toy there?


You got a toy there too, Belle.


We were going to come play in the backyard, but I guess since it's raining, let's do a


workout inside.


Yeah, that's a good idea.


Now I went for a bike ride earlier this morning in the sun.


That was amazing.


How good is the sun?


Check this out.


We just finished some riding, writing, not riding, we're riding now.


Now we're up to, we've got BJJ training and the Google Cloud certification on Coursera.


It's a big day.


So I've done some physical exercise today.


Oh yeah, I had BJJ training.


I got a tournament in like 10 days, which I'm really excited for.


But I like to break up study with movement.


So I've been sitting down for a bit now.


It's time to move the body.


The puppies love moving too.


We're going to do something nice and simple.


We'll start off five minutes of stretching and then we're going to do a hundred pushups


and a hundred squats.


And then we'll get back into the other three labs that we have to do.


Let's go.


We'll start off with one of my favorite ones for the hip.


You know, sitting down for so long, studying all day, sitting on a computer screen, your


hips can get real tight and that's, that's not a good time.


So what we want to do is get one leg back like this, that leg up forward, hands, shoulder


width apart.


And then you're just going to straighten your back leg, thrust your hips towards the ground.


All right.


And if you really want to step it up, you can just hold it here.


You can go down like this, right?


Keeping that same posture.


Or what I like to do is get up a back stretch as well.


I'm going to do this for five, five each side.


Should feel it all through the hip, all through the leg.


Really make an effort to keep that back leg straight.


For this one, we're doing a posing movement to what you usually do when you sit down,


right?


You usually sitting at a computer desk, hunched over like this.


Right?


So the spine is curved and the shoulders are rounded forward.


So this one is to counteract that.


You're going to come in just like this.


You can start on your knees if you want, or I prefer start flat on your stomach.


And we're just going to pretend like we're on a surfboard.


Keep your hips on the ground, push up like this.


So now we're doing the reverse of that curve that happens when you're sitting at a computer


desk for a long period of time.


I'm just going to hold this in.


We're going to make the arms nice and straight, spread the fingers and hold it there for about


20 or 30 seconds or so.


Just to feel that stretch, should feel it right through the spine, right through the


hips.


All right.


Push ups and squats.


Let's do it.


We call this one the lounge room workout because you can do it anywhere.


So to do these 100 squats and 100 push ups, we're going to do 20 push ups straight into


20 squats.


We're going to have a minute rest.


Do that five times.


I like breaking up things into intervals.


Study intervals, workout intervals.


Let's go.


Set one.


For the record, I'm done trying to make y'all comfortable.


For the record, you ain't trying to grow then it's done for you.


For the record, lab on me going all the way.


For the record, ain't trying to link no time to waste.


100 push ups, 100 squats, five minutes of stretching.


You like that Bell?


Yeah, you had fun, didn't you?


Feeling good now.


A little bit out of breath, but that's good.


It's definitely raining outside now.


Check this out.


Seven, you made an absolute mess there, didn't you?


All right.


We're going to get some water, wipe this sweat off, and get back in the last three


labs.


We'll finish this certification tonight.


Oh yeah.


My gosh, it's dark in this room.


Alrighty then.


You get it?


That was like Jim Carrey?


That's not that good.


Don't say that again.


I thought it was pretty good.


Yeah, all right.


Okay.


Let's get back into some Google Cloud Platform dot engineering, lads.


I wonder what we... I can't exactly remember what was next.


I think it's map reduce, which I'm gonna have to look up the definition to as well,


because there's a lot of terms in here.


A lot of this stuff is new to me, right?


And I think it's important to recognize that even though it's all new and you're going through,


I'm going through these labs, or you might be going through these labs or whatever course,


you're not gonna get it the first time around.


Like, I find that a lot in jiu-jitsu, right?


So I go in, I do a move, and I'll stuff it up, and then the next time I'll go in,


it'll be the exact same move, and I make the same mistake.


And my coach is always like, he's a black belt, so he's wise.


Awesome dude.


And he's like, you know, make the same mistake 100 times, 500 times,


but eventually you'll realize, hey, I should probably stop putting my arm there


so I don't get choked out next time.


Anyway, let's get back into it.


We've got three labs left, Dataflow, Data Pipeline,


oh, there we go, Lab, Data Pipeline, there was MapReduce,


that's what I was talking about before, you're gonna have to look up that.


I think it's reducing the dimensionality of your data,


or if you know what MapReduce is, leave a comment below and explain it as best you can,


and so you can help me and others understand.


And then we have another lab, MapReduce in Dataflow, so we've got Dataflow Pipeline,


MapReduce in Dataflow, and then a final lab, which is Lab, no, Lab Inputs, Side Inputs.


And then we finish up with the last quiz.


Easy money, it's 6.42, we're gonna finish this in time for dinner.


Woohoo!


All right, catch you on the flip.


Check it out, we have the last lab.


What have we got? Side Inputs.


Boom!


So Side Inputs are like different sources of data into your main pipeline.


So imagine you've got the stream of data here, Side Inputs,


another one coming in over here.


That's how I'm thinking of them anyway.


All right, let's get into it.


Finish this lab, and then we'll go get some food, and then wrap it up.


All right, I'm on to my last attempt on this quiz for the last eight hours,


or for the next eight hours if I don't get this quiz right.


I always, I wrote an article the other day on how I always stuffed up in exams.


Exams get me, man, they get me.


All right, so which of the following statements are true?


I think it's this one.


Dataflow executes Apache Beam pipelines,


and Dataflow transforms support both batch and streaming pipelines.


And this one I know is BCA.


If not, we're completing this tomorrow morning.


Jump in.


Yes!


Oh boy, oh boy, oh boy.


Some resources here.


You can check those out if you want.


Actually, there's a great Medium article which is here.


I'll link that below so you can check it out.


It's one of the sample projects, right?


It's like Google put all this GitHub data on BigQuery so you can test it out.


So it's two billion files.


And then we did one before which are like which Java programs need help?


So the Java open source libraries which have to-dos or fix-mes in them.


And there was two million one hundred and ninety six thousand two hundred forty seven files.


Something you don't really want to process on your little baby laptop, all right?


You want to use BigQuery for that.


So we've gone through a lot of stuff today.


Let's see if we've completed it all.


From Dataprep, well Dataprep is a new one actually that I just learned about.


BigQuery, Dataflow, MapReduce.


Have we completed the required stuff for today?


Oh no. What is this?


We haven't passed one of these things.


I'm going to have to go back.


It was that dang side inputs lab.


The last lab, best lab.


All right, well, we're going to have to go back.


I'll see you in a sec.


All right, we're back.


So I don't know what's going on.


There must be an error. I need to reach out to Coursera.


Because refresh this page saying I haven't completed this lab.


Then we go into here, the exact lab, lab completed, elapsed time seven minutes.


It's seven minutes because I've done it four times now and I'm really quick at it.


You don't need the certificate right now.


You think so?


Yeah, yeah, you can fix the bug tomorrow or whatever, but what's most important is you've got the skills.


Yeah, you're right.


You've got an incredible platform like Coursera.


It helps out 35 million people worldwide, partners with incredible companies, incredible universities.


What you want to use it for is to get the skills that will take you to where you want to go.


The thing about studying is you can spend the whole weekend in this room.


Hours in that chair there.


Then you go to work on Monday or you see your friends.


No one knows about the hours you put in here, but that's okay because you're on the journey.


You're on the journey to where I'm from, 2048.


I want to thank you.


I want to thank you for taking care of your health and taking care of your knowledge.


Step by step, I mean, you may not notice it now, but it all adds up over time.


Trust me.


Keep learning, keep moving.


Too true, Future Dan.


Yeah, I got to keep that in mind.


You heard the man.


Keep learning, team.


Special shout out to Coursera for sponsoring this video and special shout out to my boy Lack.


Lack's back, baby.


Or Lock or Lack.


I still get his name wrong.


It's Lack.


It's definitely Lack.


Lack, you're the man, man.


See you, team.


I got to get some dinner.


Keep learning.


Oh, yeah.


If you want to try the course, there's a link below as well as a link to that Medium article,


which was a tutorial on BigQuery.


But I'm enjoying it except for this little one part that's not letting me pass, but we'll get through it.


Have fun.


I mean, you can do it tomorrow, but the certificate's not the be-all and end-all.


For the record, I'm done trying to make y'all comfortable.


Udacity Artificial Intelligence Nanodegree Project 3 Progress | Learning Intelligence 8

What's happening y'all? Welcome to Learning Intelligence 8.


Today's been one of those days, you know when you get up in the morning


and all these things happen that you weren't really expecting to happen


and the sort of running in the background of your brain and you're like


I need to get these things done before I can do anything else.


So that was where I spent most of my morning.


But no complaints, got into some studying.


It's fairly late here, well not late but late for me to be studying.


It's about 8 p.m. at the moment and I've got about a week or so


until project three of the Udacity.


AIND or Artificial Intelligence Nanodegree is due.


I know I'm trying to reduce that massive saying


but I think it's I got to get more used to it.


So I spent the last hour or so planning.


I also spent an hour or so reading this puppy,


slowly working through each of the chapters.


I'm finding it more and more fascinating as I go through.


A lot of dense material but I want to show you something cool.


I got this playlist here that I call Conquer


and it's like my get shit done playlist.


GSD, might be a little bit burnt actually, I just realized that.


I went out and did a workout in the sun today.


It's been raining where I live for like the last week.


I put on the playlist, put on the headphones


and punched out some planning here.


So now I've got a whole document here dedicated to planning project three.


So as you see there and this is a project three on GitHub.


So essentially I've got to plan out an airport cargo system


and then analyze it.


I'll link the Udacity GitHub in the description


so you can check it out if you want to.


I think this will be the most complex project I've tackled yet.


I haven't dived fully into it.


I've only got a plan of attack.


I'd like to spend at least a few hours planning


so I know I can tackle it step by step rather than just diving in.


And something I'm also trying to work on is improving my ability,


improving my Python code.


I think that's one of my weaknesses at the moment.


It's turning the sudo code in the textbook into functions.


So I'm going to be dedicating at least I think 15 to 30 minutes per day


just specifically on taking a function from the book


and trying to replicate it in Python code.


I think that's a good goal to have.


I got a tip from the mentor actually, from my Udacity mentor.


She said if you're having problems with that,


try practicing it a little bit each day.


And it's simple advice, but great advice.


I just went through a review of the notes I took last week


and the classes that I did last week on search and planning


in preparation for project 3, which is my next major milestone.


So I've put things like DataCamp and other coding stuff I was doing on hold


to really focus down on getting everything ready to go


for project 3 in the AI Nanodegree.


But I have three cool things I want to show you.


Number one, explain Lycam 5.


This is a Reddit thread where you can go and find advanced topics


such as, or in my opinion, advanced topics, the A-star search algorithm.


And you'll have the best, smartest people in the world commenting on Reddit.


You got to verify to make sure their answers are legit


in a way that's easy to understand.


The A-star algorithm is quite complex if you've never heard of it before,


but essentially it can be broken down and considered


to be the best estimated total path cost first.


If you have a route finding algorithm you want to get from point A to point B,


A-star search will find you the best estimated lowest cost route.


That's what I gathered so far from the Eli5 thread on Reddit


and I'll link that in the description by the way.


So any topic you're having trouble understanding,


search Eli5 and then whatever you're looking for


and hopefully that someone's made it on Reddit.


If not, if you decide to, if there's no thread on it,


maybe you can create your own and help out the rest of the community.


Number two, so you know how I've been doing all these search algorithms


or if you've been watching along the vlogs,


I've been learning about different kinds of search and planning algorithms


in artificial intelligence.


And so I found an awesome, this is from the Eli5 thread by the way,


an awesome visualizer to demonstrate different search algorithms


such as breadth-first search and depth-first search and A-star search.


I'll let the visualization do the talking.


Check this out, you start at the green dot,


you draw these walls by just dragging the mouse


and then it wants to get to the red dot


and you can choose your search algorithm here


and then adjust the heuristics.


So if we go, let's start, this is the A-star search,


this is how it works, a visualization.


So I'll link this in the description


and then it finds the parts of the gold and let's check it out.


Want to do it in time, 10 milliseconds, or sorry, 10 seconds.


I thought that was really cool and failure of the day,


I'm going to put right here.


I applied for an AI role in a startup in Silicon Valley for Superhuman


who are building a revolutionary email client


mainly because I'm in love with communication and I'm in love with AI


and I didn't get the role, so that's all right,


I get to go back to what I'm doing


and I think I'm going to try to do that at least once per week,


have some sort of failure, at least something that I've tried and failed at.


Might not be a job application, but we'll see what happens.


Have I told you guys before how much I love walks?


I just went for a 20-minute walk to clear my head,


I've been working on my project all day.


Let me show you what I've done actually.


I love walking, it clears my head.


I was in a bit of a rut with this project I'm working on


and I decided it's time to get moving,


get the subconscious working on the problems


and now I've come back, just been going down to the water,


amazing area down there.


My head is so much clearer, ready to take on the next stage.


This is where I'm at.


I've been going through the GitHub page


and just writing out all the things I need to do,


even the next function I need to implement.


I've started it, I've left it here to come back to,


to implement with handwriting before I implement it with code.


So I'm gonna do about another hour and a half of study


before I call it a day.


My goal is to have a rough outline of the next functions


I need to implement in part one of the project


and then after that, I'm gonna do some reading


because I need to do a report for part three of the project


on the history of AI in planning, in planning.


So the project is on, I've got to develop


a planning domain language or some planning functions


to plan transportation to and from certain airports


with planes and cargo.


But I'll fill you in more when I've learned more about it


and made more progress.


Side note, the next time you're stuck on something,


try going for a walk for 20 minutes or so.


You know, Harvard published a study recently that I read


that said walking for about 20 minutes or so


promotes creativity.


I guess that's why Steve Jobs had all these meetings


on walks.


Hacking into the database.


We can do this.


He's on the run.


We got it.


Tracking down.


Where is it?


Did we do it?


Fail.


Errors.


Plenty of errors.


I always feel like I'm an elite level hacker,


like one of those ones from the movies


when I'm working in the terminal.


I just spent the past hour and a half or two hours or so


implementing a few functions into my project.


Three for the AI nanodegree.


And I had my noise cancelling headphones on


and I was in deep coder mode.


And I think my mom came into my room


and said something to me.


And I must have replied like a zombie.


I don't know about you, but when I'm in coding mode,


like that's all I'm doing.


I still got an error coming.


Got a failure here.


That's all right.


I'm going to call it a day for the coding portion of things.


I got to still...


I'm going to take these off.


I'm still going to be reading the history of AI and planning


and the artificial intelligence textbook that I have.


But otherwise, really cool news I'm excited for later this year


and actually in a few weeks time.


I got offered a role at Code Camp,


which is an equivalent of like a school holidays kind of boot camp


coding session for kids where I can go along


and teach kids how to code.


And I'm really, really keen for that.


So I'll see if I can film there.


Probably not because there's kids involved or whatnot.


I'll bring the stories from there to the V-log.


That won't be for a few weeks.


I'm thinking I think it's about December or so just before Christmas


and after Christmas on the school holidays.


I'm really excited for that.


Like if you can't tell, I know, I got a smile on my face.


I read the email and they they offered me a role and I'm like,


yes, I could teach these kids how to code.


So it's a few days a week essentially starting from the ground up


and getting kids upskilled with the most valuable skill in 2017


in my opinion.


I'm biased of course because I'm learning it.


But I think anyone everyone can gain value out of coding


and it's not just coding.


It's the underlying skill that comes along with programming


and that's problem-solving.


So I think that's going to be it for episode 8.


Thank you so much for watching.


Next week's video is going to be more of project 3.


That's due in a few days.


I'm going to crunch it out and we'll get it submitted


and then I'll show you when it's done.


In-depth deeplearning.ai Specialization Review

So you're interested in learning deep learning and you've checked out the deeplearning.ai


specialisation but you want to know a little bit more, well you've come to the right video.


What you see here is I want to learn deep learning. So that was me and it could be you as well but


what I think you should start with and this is what this video is going to go through. It's going


to go through the why, what, when, how and where of the deeplearning.ai specialisation. But what


we're going to start with is why. So I want to learn deep learning because