And then once I'd finished a module, say for example,


How I passed the TensorFlow Developer Certification exam (and how you can too)

I finished the computer vision module


of the TensorFlow and practice specialization,


I'd wrap it all up by watching the MIT deep learning lecture


related to that module.


So this kind of tri-grid approach,


I find I learn best from multiple different sources.


You're probably similar.


So if you have someone,


you might have some teacher explain something


and you don't really understand it,


but then you read it, you read it in a different way,


and you understand it better,


and then kind of you create your own knowledge


through practice.


So that's how I approach things.


In terms of going through all of this,


how long should it take?


Well, as I said, it took me about three weeks,


but that being said,


I have experience with using TensorFlow.


So whatever your timeline is,


you wanna adjust it for your own level of knowledge.


And of course, not everyone has the luxury


of being able to study four to five hours a day.


So again, fit it into where you can.


And remember, learning anything worthwhile takes time.


All right, you've done all your study.


What happens with the actual exam?


Well, the actual exam costs 100 US dollars to take.


And if you fail, well, you have to pay again,


but you also have to wait two weeks before you can do that.


If you fail again, you have to wait a little longer.


And the timeline you have allocated for the exam


is five hours.


So when you press start,


or by the way, tidbit, when you sign up,


when you register for the exam,


it takes about an hour, well, it did in my case,


for your, because you have to formally register,


you have to scan your documentation.


It takes about an hour for things to go through.


So if you're expecting to take it as soon as you sign up,


you might have to wait a little while.


So that's, I was like, yep, I'm gonna take it today.


Here we go, what do we have to do?


Oh, your ID has not been verified.


I probably should have done this first, shouldn't I?


Your ID must complete verification


before you may redeem or attempt the exam.


Great, so I had to wait an hour or so


before I could start the actual exam.


But once you do start the exam,


once you hit start, officially hit start,


a timer gets set up in PyCharm,


and you've got five hours to complete the exam.


Now, as I said, I made sure TensorFlow worked in PyCharm


before I started, but I spent about the first hour,


you could have guessed it.


Well, goodness gracious me, we got it fixed.


Ran into the absolute epitome


of TensorFlow errors just then.


Turns out I had to get into the Python source code


and change something in Python 3.7.3,


which I have no idea why I would have to do that,


since yesterday it was all working.


Here we go, this is the thread.


I'll put the link somewhere.


Now the exam stresses that you need Python 3.7,


but for some odd reason, when I have Python 3.7,


for some odd reason, I found myself


for the first hour of the exam


deep in a GitHub issues thread of TensorFlow


trying to figure out what was going on.


An error kept coming up every time I ran TensorFlow.


It turns out I had to change a line,


or specifically line 48 in line cache.py


to fix the error in the Python source code, that is.


And now I'm not sure, I'm probably putting that down


to some sort of user error.


Maybe I've done something wrong,


but once I implemented the fix I found on GitHub,


everything worked fine.


So if I didn't have that sort of roadblock


at the start of the exam of trying to fix everything,


I think it would have taken me probably under three hours.


So the five hour time limit is so that you can train


deep learning models on your computer


because the way you get marked


is by submitting a trained model file


which gets automatically validated.


So again, another tidbit here is that


if your computer isn't fast enough


to train a deep learning model,


these are not large models, they're quite small,


but if it's still not fast enough to train it


in that five hour timeframe,


you could train it on Google Colab, for example,


download it, and then store that saved model file


in the PyCharm directory


that the TensorFlow developer certification creates for you


once you start the exam.


If all of this doesn't really make much sense,


it'll be clear when you sign up for the exam


and read the instructions of how to complete it.


Now, do I have any feedback for the exam?


Like, how was it?


Like, did I feel like it was testing my abilities?


So probably my main piece of feedback would be,


it would be great, and I know this is,


it's only the first iteration,


so it's no doubtedly going to be changed over time


as these things go,


because you get the certification in the last three years,


so going forward, if you want to recertify,


if you want to resharpen your skills,


you're gonna have to take it again.


But I would like to have seen it a little bit different


to the Coursera TensorFlow in practice specialization.


So if you're a keen student, you'll probably notice


that the TensorFlow developer certification handbook


is basically, or the criteria for it,


is basically the outline of the TensorFlow Coursera,


TensorFlow in practice Coursera specialization.


That is a mouthful.


So yeah, that is why I recommend that


as being the number one resource that you should go through


if you're looking to take the certification.


As I said, the certification only focuses


on the modeling part of a machine learning pipeline.


Maybe in the future, there'll be others


on pre-processing, modeling,


and then serving a model, but we'll see.


So you've finished the exam, what do you do next?


Well, the first thing's first is that


once you've submitted it,


and you'll have a pretty clear indication


as you go through as to how your progress is going,


because for each model you submit,


it gets marked fairly quickly,


and you get a score out of,


I'm not gonna ruin anything,


but you just get a score for that,


and you'll know pretty quickly whether or not


you're on track to pass the exam.


So once you go through, you get an email at the end.


Once you click submit, you get an email which will say,


congratulations, you passed, if you did.


If not, that's okay.


There's always, you can always take the second attempt,


but you won't get much feedback.


It'll just say, hey, you passed, or hey, you didn't pass.


But within that email, there will be a form


that you can go fill out that you can receive.


You're a fincil.


Official TensorFlow developer certification.


This bad boy here.


I actually just got mine this morning,


so it takes about a week.


And you can register to the Google certified


developers network.


So what that means is that if someone is looking for


a certified TensorFlow developer online somewhere


in their region, they can search up your name


and find you and go, hey, well,


this person's got TensorFlow skills.


They've taken their certification exam.


They've proven that they can build things with TensorFlow.


So I think that is pretty cool.


And then once you get your certification, of course,


and your badge and whatnot, you can share that


on your professional network, you can share that,


put it next to the projects that you've made


on your portfolio, and that way, again,


just as I said at the start, it's nice to have,


it's a nice thing to be able to stack next to


other things that you've worked on to say,


hey, look, I've got these skills, here's the proof.


Finally, and most importantly, I don't know if you can see,


but if you can't, there's a big check mark here


that says today, get TensorFlow certified.


We're gonna cross that right off.


Boom, we're done.


And as I said, it is not about the digital certificate


or the badge, it's about acquiring the skills,


the skills to use TensorFlow to build


machine learning powered applications


that enable the change you wanna see in the world to happen.


If you have any questions about any of the resources,


how I study, how I created my curriculum,


or the actual exam itself,


I can't share too much about that,


leave a comment below and I'll answer it,


but don't forget to check out all of the resources,


including the curriculum that I created in Notion,


try creating out one of your own


by clicking the duplicate button.


They'll be in the description below,


as well as a full blog post detailing


all of the information we've talked about.


As always, keep learning, keep creating,


and I'll see you next time.


Why Multimodal Machine Learning models do not work. Part 2/2 – The CAUSES

Disclaimer. This video highlights the not so favorable aspects of integrating vision and


language. For information on risk and side effects, please read your favorite publication


and ask your machine learning researcher or engineer.


Hey there! Did you watch Ms. Coffee Bean's previous video? If not, check it out right


now and then come back to this video. Last time, she told us about the symptoms that are a clear


hint that something fishy is going on when integrating vision and language with deep


learning methods. It seems that one modality is usually almost forgotten, even though the tasks


are defined to require the understanding of both modalities to deliver the right answer.


In this video, Ms. Coffee Bean is going to tell us what she found further on her journey through


the dark paper forest. Ms. Coffee Bean, what are the reasons for problems in vision and language


fusion? On my journey, I encountered two sources of all evil. Some say the source of problems when


integrating vision and language is in the data, while others say it is in the neural network


based models. Let's start with the accusations against the data. The VQA dataset was released


in 2015 and a lot of work was invested by researchers to get on top of the challenge


leaderboard. Then, 2017, a version 2 of the VQA dataset had to be released, since obvious biases


in the data were detected. These biases were making the undesired happen. The model could


focus only on the text ignoring the image. It's in the title making the V in VQA matter. For example,


far too numerous how many questions have the answer 2. If the question is about sport,


then usually the correct answer is tennis. Also questions like is there a special something in


the image bear the answer yes, since humans contributing for the dataset collection


usually do not ask if there is a horse in the image if there is none to be seen. How would they


even get the idea of a horse in the first place? So, these obvious biases were removed, but there


are suspicions that neural networks can also rely on not so obvious biases and many models still


seem to ignore the image far too much. What holds for VQA usually holds for visual dialogue too.


As visual dialogue is simply put VQA with history. This paper shows a very simple statistical method,


not even neural networks, without accessing the image or the sequence of the dialogue.


And this method performs on par on some metrics with extremely complex neural models.


Of course, authors of the original visual dialogue papers did not take the critique well. If you want


to read how researchers fight, check the links to these papers and the response in the description


below. But whatever side you are on, I think that one must take this alarm seriously. It is always


tricky to produce clean training data, because the fingerprints of the data collectors are everywhere


on it and neural networks are extremely good at exploiting every hint and bias to get to the right


answer, even if for the wrong reasons. But apart from neural networks exploiting bias in the data,


it seems to be something fundamentally wrong with neural networks trying to fuse modalities.


So it is time for us to discuss the problems that models might inherently have. We think this


question is still very, very poorly investigated, but we have one research product to show you.


This paper conducts a very simple yet fundamental experiment. Given an image that can be described


by only one word, a neural network has to construct a word that is describing the image.


Not as a classification task, where the network has to choose the right index of the right class


to which we humans have assigned the labels, it is rather a regression task, where the model has to


come up with the right word embedding that best fits to the picture. Simply put, they try to


translate images to single words rather than a whole sentence like in image captioning.


What do we expect? It should be an easy task, right? Image captioning is a much harder task,


and neural networks can do it, so why should it not be possible to translate an image to a single


word? And you are right, it is possible. The neural network performs with an impressive accuracy,


only that the paper looks a little bit closer. It also measures how far the neural network can


transform the input space to make it look like the output space. And here the results are surprising.


It turns out that the neural network is really bad at that. It stays much closer to the input


modality, which is the image. Even though the network is trained to fully transform the image


space to the textual, word embedding space, it preserves the neighborhoods of the input space


rather than construct the output space. Why? Because neural nets are guaranteeing continuity


and preserve topology. We think that there is still very much to investigate along this line,


especially the case when both visual and textual modalities are mapped by a neural network into a


common space, because this was only a special case where the image only was transformed.


But even this special case analysis is a strong alarm signal when talking about neural networks


in multimodal research. As they are universal approximators given infinite training time and


data, we expect them to be the solution to all our multimodal problems. But perhaps we expect


too much from them in our real setting with not so much training data or training time.


Like in the example seen before, the neural network was able to transform single data points


from the image to the textual modality, but was not able to sensibly transform the space in between.


With infinitely many training samples, this would not have been a problem, since the network would


have had a training sample for every point in the space. But even if we would have these infinite


resources, we have seen how problematic data and biases can be when integrating vision and language,


and that we can insert multimodal fusion difficulties when choosing a model or architecture.


We could end our video here, since this is what Ms. Coffee Bean found on her search for hints and


reasons on why current vision and language models fail where they fail. But we want to spend a


minute on multimodal models that are coming out right now, beating the previous state of the art,


but have not been yet more thoroughly investigated for problems. Yes, we are talking about the


multimodal transformers, which are all variations of the same theme of processing images and text


and combining them with transformer modules and cross-modal attention. For more details on how


this architecture works, check out our previous video on Wilbert linked below. Multimodal


transformers show through their better accuracy on multimodal tasks and attention visualization,


that they are better at combining the visual and textual modality of the model.


But still, a recent publication suggests that for multimodal transformers, it seems again that,


quote, the textual modality plays a more important role than image in making final decisions.


So here we are again, at the same problem that existed even before multimodal transformers,


the inequality of modalities and the curse of biased datasets towards a textual modality.


So what do we do now? More research! The relevant papers for this video are in the description


below, if you are interested to investigate the root of the problem yourself. This video is of


course not an exhaustive literature enumeration of all problems in multimodal integration.


It's rather just a hint of where to look when searching for trouble, I mean in an academic


sense. Anyway, what do you think? Let us know in the comments what your thoughts and observations


Setup Apple Silicon Mac for Machine Learning in 13 minutes (TensorFlow edition)

Hello there and welcome to the M1 machine learning cooking show. I mean recently you might have picked up one of these new bad boys


This is a 14 inch MacBook Pro. You can tell with the magsafe. It's one of the new ones with the M1 Pro chip


I'm not gonna go over what it is. It's a beautiful machine. It's brand new. I can show you that it's an M1


Pro here. There's M1 Max. The code in this video will work for the M1, M1 Pro and M1 Max


At least it should from my tests


We're gonna go ahead and set up an environment or get our machines ready for machine learning


specifically using some data science the common data science packages like


pandas numpy matplotlib scikit-learn and Jupiter and


TensorFlow and even more importantly running TensorFlow on the GPU that's inside the M1, M1 Pro and M1 Max


So without any further ado, let's get into it


I'm gonna jump in if you want any of the links, by the way, they'll be in the description below


This is on my github. This is just some code that I use to set up an environment to do


Speed test of all these new machines


So I'm going to open up terminal


I'm gonna zoom in here


Let's go over there


So this is going to be a relatively fast-paced tutorial if you'd like an explanation of anything that I'm doing


Please refer to the links below. There'll be a longer text based explanation and you can see the stuff there


So let's jump in. So the first thing that we're going to do is get homebrew from brew.sh


Now there's a beautiful one-liner of a code here homebrew is a package manager


That should really come installed on Macs and Apple laptops, but it doesn't it helps you install other software. So when you


Copy this code into here. You're gonna have to enter your password. I've just entered mine


It doesn't show up because it hides it automatically. We'll press enter. It's going to install a whole bunch of different stuff here


I'll press enter again. It might take a few minutes depending on your internet connection


And so I'm going to speed this up until it gets to the next step and then we'll go over that together


Beautiful and that took a couple of minutes on my machine your experience may vary now homebrew


Tells us a whole bunch of things that it did you can go back and read those if you like


As you can see here installation successful


But now it says to run these two commands in your terminal to add homebrew to your path


This will just allow you to use the brew command such as like here now


I will put a little caveat here. That is there are ways to do this without installing homebrew, but


Just take it from me that you should install homebrew because you're going to eventually end up using it


It has a bunch of useful stuff for you. Now. Let's go back to


the instructions


We're going to now download mini forge 3 which is a conda installer if you'd like to learn more on that there's links in the resources


But this link is going to directly download an sh file to our machine


We see there mini forge 3 Mac OS X arm 64


Basically homebrew and mini forge are package managers and package managers help us install other software to our machine


So once we've downloaded that I've got it saved to


Downloads there you can also see it down here


We're going to run these three commands here


This is step three install mini forge 3 into home directory ch mod changes the permissions on the file to make it executable


That's what the X is sh is just basically saying hey run this shell program


And then source mini forge 3 bin activate is going to activate the environment if it gets installed successfully


I'm going to run all three of these press enter follow the prompts


You have to read this you have to go right down to the bottom and once you get down to the bottom you can press


Yes


So let's just get there. This is basically saying you agree to all of the license terms


Where do you want to install it? I'm just going to install it in my home directory under Daniel press enter


It's going to install a whole bunch of stuff. It's going to enable us to use conda. That's the most important thing


We get python 3.9.7


Do we wish to initialize conda yes


Beautiful thank you for installing mini forge 3 now


I'm going to clear this again and to make sure the changes take effect


We're going to restart terminal you might also notice that there's now base here in front of our command prompt


That's just to say that the conda prompt is available now


But sometimes it messes up, so I'm going to restart terminal just to quit it and


Open it up


Terminal there we go. Just zoom in so you can see


Take that to the side now. I'm going to create a directory to test a tensorflow environment


I'm going to call it tensorflow test using the mkdir function and


Then I'm going to change into that you can call this directory anything so if you're working on a project say


TensorFlow computer vision you might make a directory called tensorflow computer vision you can really call this whatever you want


So make the tensorflow test change in that directory


Beautiful now you see we've got tensorflow test here ls. There's nothing in there. We're still in the base environment


Beautiful now. It's time to make a condor environment an environment. I think of it as like a room in your house


so you can imagine in the kitchen you've got all the tools to cook stuff and


It'd be weird to have an oven in say your bedroom. You might have enough in your bedroom


I don't know, but it's the same on your machine you create an environment in this case


We're using conda other other tools to do it a virtual lamb and whatnot


But we're using conda create an environment so that we can work on


different projects


Within that environment and have access to the tools that we need so they're not getting mixed up in different rooms on our


Computer or different files on our computer?


It's going to copy this conda create prefix environment the dot here means that it's going to create an environment in the


file path


everything before


Slash-m so it means create an environment that lives in everything before em and then we're going to activate that em


I'll show you what that looks like


Beautiful


Environment location so see what I mean because I put the dot there


It meant that I didn't have to type all of this so now my environment


And I called it em is going to live within the tensorflow test folder, so I'm going to say proceed


Thank you, and now to activate the environment


You can type in the long version of conda activate the file path uses Daniel tensorflow test em


Or in my case or in our case. We just ran conda activate dot slash em


Which means hey the current directory?


Slash em now if we go LS. We've got the M folder there. I'm going to clear all this so we've got some room now


We can get into the fun part. We've got an environment ready to go


We've got a room ready on our computer that we can install different software


Let's now install the em folder


Apple's tensorflow dependencies, so apple have made it really accessible to start using


tensorflow on new M1 machines


So they've created a conda channel called apple


This is what this command stands for conda install from channel apple tensorflow dependencies


Intensive flow is quite a large piece of software


Which is why it comes with a lot of dependencies and it's a lot of work to do


So let's run that it's going to take a few minutes depending on your internet connection and


You'll see a whole bunch of things come up here in a second that is just going to install a lot of different software


Within our current environment. We're going to say yes, please install all of that


the OS X arm 64 stands for arm 64 is the


Software arc or the hardware architecture that the M1 chip is built on


And this is going through nice and quick. I've got some good internet where I am at the moment


So we see carers pre-processing there's a package being installed there a few more other things there's tensorboard


Wonderful so I'm going to clear that again make the screen a little bit more clear now we can install base tensorflow


So that's with this command here. I'm going to run that and talk about it


So python M pip install tensorflow Mac OS


So I'm going to run that again


And we're going to see that the M1 chip is built on a new M1 machine


And that's why it's called arm 64 because it's a lot of work to do


So that's with this command here. I'm going to run that and talk about it so python M pip install tensorflow Mac OS


Tensorflow Mac OS is Apple's fork of base tensorflow, so it's the exact same as what you'd find on tensorflow.org


Tensorflow.org wrong website


Tensorflow.org except it's been optimized for Apple hardware. Oh


Beautiful that installed nice and click quick. Let's go back to the github and


Then we can now install tensorflow metal now. This is one of the most important points. It's going to install quite quick


Beautiful look how quick that was tensorflow metal is going to enable GPU


Acceleration on M1 M1 Pro and M1 max why is it called tensorflow metal well if we look up Apple metal?


Metal is Apple's


GPU framework yeah accelerating graphics and much more so if you're using a Mac which


If you're watching this video you probably are it uses metal to accelerate graphics, and if we go Apple tensorflow


metal


Apple have developed a tensorflow metal plug-in which is a pluggable device in tensorflow to enable there we go


Accelerate training with metal on Mac GPUs long story short if you only install tensorflow Mac OS


Which is what we did here. You can still use tensorflow. It just won't leverage the GPU


That's on your new M1 device, so that's why if you want to use GPU computing we install tensorflow metal


So now let's get rid of that


Finally if you want to install tensorflow datasets and run the benchmarks that are included in the repo


I'm going to have another video that's going over the M1 machine learning test you can install tensorflow datasets


I might just do that because it's nice and quick that'll give you access to a whole bunch of different datasets


Packaged within tensorflow, so we'll clear that and then if you want to install some


Standard common data science packages you can run conda install this is going to install Jupiter for Jupiter notebooks


pandas numpy matplotlib and scikit-learn


I use these almost every day


I'm gonna go yes, please install all of that


Fair few packages here, so I'm gonna speed this up as well


Beautiful and that's all done. I'm gonna clear the screen make it more visible and then to test it


We can start out a Jupiter notebook


That's going to open up a Jupiter notebook in my browser


Wonderful if everything worked you should get something like this


I'm going to start a new notebook and I'm going to copy this command here


Bring that into the notebook. Just going to maximize that zoom in so you can see and


So this is going to import numpy pandas scikit-learn


TensorFlow STF and matplotlib and it's also going to check whether


TensorFlow has GPU access or not and the current version of tensorflow


So let's run that it may take a little while if you're importing things for the first time


So once again, I'm going to speed this up and I'll come back when it's got the output


Beautiful look at that now that took about a minute maybe two minutes on my machine


So it says here that tensorflow now has access to the physical device


It has access to the CPU and it also has access to the GPU which if we have a look at here system information


We have on the m1 Pro


We have a new m1 Pro


And we have a new m1 Pro


We have on the m1 Pro we have a 14 core GPU wonderful


So that just means that oh, there's metal metal GPU family Apple


So that just means if we run tensorflow code now, not only will it be able to leave it as a CPU


It will also be accelerated on the GPU. So with all that being said happy machine learning


Oh typo, and if you have any questions leave them in comments below and if you have any further resource needs


Two things I'm doing wrong (running an online business)

Testing baby, one two three. Monday, you know what that means? Push-ups.


We're gonna bust 100 out before we get started. Five sets of 20, not in one go.


Push-ups done. Every morning, physical activity. Alright, now the rest of the day is easy.


Have a bloody go at this. Mobile workstation. Real mobile.


Oh, look at that. Fits under one arm. Pad, book, keyboard, iPad, book, pad. Two pads.


So unorganised, I need two pads to write everything here. Well, we're gonna get a


little dopamine hit because we can tick off a few things here. What have we got? We done reading,


we done most of writing, I'm gonna do some more beta. I cleaned my room as well. Tick.


Okay, some of the comments you guys have been leaving on previous videos have been absolutely


amazing. So thank you, thank you very much. I thought I'd address maybe one or two really


quickly. Someone asked, how do I avoid distractions? And so the real answer is I don't have a really


in-depth one, it's quite simple. And I don't say simple as in you should do that, but this is just


kind of what I do. I have a list of things that I write down and the brightness is too much. There


we go. I have a list of things that I start off the day with. Same with the whiteboard,


you just saw that one. That's over there. And I really don't do anything else but what's on that


list during the day. So don't get fooled. Hold on, I need it. Don't be, don't misinterpret my videos


when I put like all the fancy edits and fancy music. They are pretty fancy, eh? Don't misinterpret


them because really most of the work is just me sitting there doing what needs to be done.


Like if I put that clip in, have a look at this. Right, that's no music, no edits. That's just what


it is and that's for hours upon hours during the day. I think that's the most valuable skill I've


kind of developed over time is to just be able to sit there and get something done. So that's my main


tip when it comes to dealing with distraction is just if you want to, I get lost in thoughts of


what I have to do. So if you want to avoid distraction, write down a list of things you're


going to say yes to for the day. It should be a relatively short list depending on what you've


got going on and say no to everything else. The other one was lifetime value, the metric I discussed


in the last video, being a hard metric for personalised services which is 100% correct,


which is kind of what I'm doing with my client work at the moment. I thought I'd address maybe


one or two really quickly, really quickly, quickly. Could be using that effort to something


scalable like wildfire, which is why I have a bias towards code and media. So that's the ideal


scenario where I'm heading towards is developing a product, building media around it. The product


could be code or something like that rather than me inputting hours upon hours in tailoring a


personalised service. Okay, that's enough rambling for today. I'm just thinking out loud, right?


Thinking out loud. If you have any advice or feedback or questions that you'd like me to


interact with, leave a comment below. Otherwise, I'm very excited for later this afternoon. Today


is the 15th of July and that's a very special day, but I'm going to do some work first and then


we'll show you why I'm excited later. Oh, it's cold today. I need my bigger hoodie.


Okay. I got a call in two hours or so. So that should be a pretty good session. Let's set a


challenge. See what we can get done before then. Alexa, two hour timer. Two hours starting now.


Hoodie on, adds 10% productivity. Door closed, adds another 10%. We're working on bonuses today.


Alexa, stop the timer. That was a good session. Phone call time. Hello. Yeah. How are you?


Bye. Bye-bye-bye. I always get confused where to look at when you're talking to someone,


because if you look in the webcam, you're making eye contact, but you want to make eye contact


looking at the screen. That's a technical challenge someone can probably solve by manipulating eyes to


look like you're looking at theirs. Anyway, long story short, eye contact is hard in video calls.


Anyway, a tip that I've done for myself is kind of anticipating the questions someone's going to have


and then answering them before getting on the call so the call is not just like long and dragged out.


I always try to have a few dot points written down about how I can answer questions or


what questions I have and whatnot, but I'm going good. Tracking well. All we've got to do is do the


work. It's food time. Oh yeah, I'm excited. Training done. Arms a bit tender. Thanks, Dave-o.


Gonna have a shower and then it's time. It's time. All right, it's ready. It's ready.


Ready for this. So excited. 10 days in the making. Family, what's cracking? Slim Shady, how you doing?


Ready for it? There they are. The spicy dogs. So 10 days ago, I made my own pickles and now


they're ready. I got dill, mustard seeds, garlic, jalapenos, chilies. Might have to bring out my own


line of pickles. Help if I can open. Moment of truth. Oh baby, that is spicy dog. Pickles. Cooking


show coming soon. You want a pickle slip? What do you rate it? Is it spicy dog? Approved.


Yesterday was push-ups. Today's gonna be yoga. I like watching YouTube for yoga. Yoga with Adrian


is an amazing channel and my partner is the star of this fight time. Morning girls.


I feel amazing. Time to take on the day.


I don't know what that little clap was but yeah, we'll roll with it. Parfee's making a mess. Me too.


Okay, today is gonna be the same as yesterday. I'll take care of Borty. Well, you can't really


see that because it's so bright in here. Thanks for the feedback on the last video. And then


ride to the cafe. Read. Write. Client work. That's a good day. All right, it's my turn to cook dinner


tonight. So that'll be in a few hours. So I'm gonna, we had a good session going before then.


Alexa, two and a half hour timer. Two hours and 30 minutes. Starting now.


Alexa, stop the timer. What a day on tour. Nice and simple. That's how most of these days are.


You just need to get in, get the work done. I'm gonna go cook dinner before training. See you tomorrow.


Morning. Straight into it today. No cafe, no riding because I'm helping my friend move house this


afternoon. And if you've ever moved house before, you know it's a big deal. So I'm gonna get what


I need to get done done this morning and then go help my friend. Yeah. Email sent off. Projects


updated. And by the way, I had some questions in recent videos of people asking what exactly am I


doing? Well, the main two clients or main two projects that I'm working on for cashflow is


creating educational content for a cloud guru and doing the same thing for data camp. So the way it


kind of works is I work on something, it goes to an editor and they check my work and give me


feedback and then say, hey, you could make this a little bit better. Can you change this to make it


better, more suited for the platform? So that's how I'm getting cashflow for the time being.


But I've kind of figured out what I've kind of been thinking about it for a while actually,


what I'm doing wrong. If we come over here, go to Bawdy. So what I'm doing wrong, signaling. Now,


what's that? Well, signaling is someone who talks a lot about doing something, but doesn't actually


do it. So you need to monitor someone's actions. That's the way I believe is a much better way of


judging someone or doing something. And so now what am I doing wrong? What am I saying, but not


doing? Now I say I want to get into product development, I want to start building things,


but I'm kind of dancing around the point, right? And now that's where we go back up to here,


go straight to the target. The quickest way to get to any goal is to go straight to it. And now


I'm guilty of this as well, right? Like I'll think about doing different things, but then you find


you find for some reason you stop yourself from doing it when you're so close to getting it,


when you know exactly what you have to do, you hold yourself back. And now I'm guilty of this,


but I've through self analysis and through writing and through watching myself and through


making videos and whatnot, I'm kind of working out where I'm going wrong. So what I think, what


what I know I need to do, I already know it, but I'm not heading towards that. So that's what I


need to fix. And that's going to be the goal basically for the rest of the year, because


where I want to go is come back to the whiteboard is creating something like this, Nutrify. What if


you could see how a food was going to affect you? And now why, why do I want to create that?


And this is fun because we're going back and forth with the whiteboard, but I want to,


I did a nutrition degree, but I still don't know how to eat. I said that in the last video,


I'm just a whole bunch of repeating myself here. Essentially everything about nutrition fascinates


me and I want to work out what is my own personalized diet. And so I'm going to do a


bunch of experiments and try and do that, but can I use technology to do that? Now I'd like to build


some kind of product around that. However, I think that will take a lot more time and money than what


I have available. So what I have to do in the shorter term is create a way for cash flow to


start coming in more stably, right? Then sort of just doing back and forth projects here and there


so that I can spend a large chunk of my time building Nutrify. Now, how am I going to do that?


Well, I haven't quite figured it out yet, but I've got a few things playing out in my head.


And so what I figured I'll probably need, my cost of living has been, I've greatly reduced that.


And so it's like basically about 1500 Australian dollars per month, which is relatively cheap. So


what I'm thinking, I'm going to need about two years of cash stockpiled so that I can just do


nothing except work on my company. So that's what I need to do within the next six months is create


that cash flow stock somehow and then use that to fund my longer term goal. And that was a bit of an


extended rant. I'm going to end this video here and then we'll make another one. I'm going to try


two this week. So let me know. This video might not turn out to be too great, but it's just me


rambling. Anyway, long story short, I've got to set up like a war chest of cash to work on my longer


term goal. That's what I need to do. And the thing I'm doing wrong, again, singling, saying I want to


do something, but not actually doing it. So the way to sort of judge someone is by their actions.


And that goes for everyone, including me. You have to be your own, your own biggest fan and your own


biggest critique at the same time. That's what I'm trying to do. I'm going to go help my friend move.


I will see you tomorrow. As always, keep learning, keep creating. And remember,


A New Physics-Inspired Theory of Deep Learning | Optimal initialization of Neural Nets

Hello and welcome to this special video! We want to share with you some exciting recent


developments in mathematical deep learning. For this, Ms. Coffee Bean and I are here today


with guests from Aleph Alpha, the sponsor of this video. Today we're gonna talk about


a recently published theory that uses physics to gain a deeper understanding of deep neural


networks and to improve them. We'll introduce the basic principles that underpin the theory


and show you what makes this so exciting. In particular, we'll see how it can be applied


to optimize the initialization of neural networks, which we'll verify experimentally. The whole


video is made with the help of Kuhn Ostermayer and we will do a second video like this one,


so if you are interested in mathematical deep learning, subscribe to stay tuned! So, we


were saying that some exciting things are happening in deep learning. This has culminated


in a theoretical framework that allows us to better understand deep neural networks


and derive practically useful insights. The theory was introduced in a 470-page book by


Dan Robert, Shaw Iaida and Boris Hanin, which is the result of several years of research


from the authors. But of course, they could not have done it without earlier works by


people such as Neil, Schönholz, Lee, Matthews and Jacob, among others. In our opinion, the


biggest contribution of the book is that it provides a framework that unifies this previous


research and extends it. The theory and the book are quite mathematically involved, so


we'll do our best to make the explanations as intuitive and visual as possible. What


do you need as prerequisites for this video? We hope you're familiar with the basics of


statistics and multi-layer perceptrons or short MLPs, because then this video should


be understandable for you. It also draws heavy inspiration from physics and makes some really


interesting connections and will go into that direction of physics too. Hopefully you are


as excited by this as we are! Deep neural networks are notoriously hard


to capture mathematically. They're made up of millions if not billions of parameters,


data are pushed through many nonlinearities and hidden layers that mix them up until everything


becomes completely entangled. This is why neural networks are often referred to as black


boxes, in the sense that the only effective way to see how they work is to just simply


feed them data and see what comes out. The current situation we're in is similar to


that of the history of the steam engine. Back when the steam engine was invented, it was


also regarded as a black box. You would put in coal and get out work. The desire to better


understand how steam engines worked to increase their efficiency led to the development of


thermodynamics which resulted in a number of fundamental laws that could describe the


workings of the steam engine in terms of simple, measurable quantities such as pressure and


temperature. When physicists discovered the laws of thermodynamics, largely experimentally,


they didn't know about atoms yet. Later, Boltzmann, Maxwell, Gibbs and others found


first principal explanations and it became clear that pressure and temperature are emergent


phenomena resulting from the statistical behavior of many tiny molecules. In other words, they


realized that when looking at a large scale, many of the irrelevant details of the individual


behavior of the microscopic components get washed out. If you want to know how hot the


steam is, you don't need to know the movement of every single particle. All you need to


know is their collective behavior described simply by a single number, the temperature.


Now to get back to neural networks, in analogy, the microscopic statistical constituents,


such as water molecules, correspond to the numerous parameters of a neural network. They


behave statistically, since they are drawn from a probability distribution. However,


for neural networks, we already know their microscopic behavior very well. We are the


gods, the programmers, that define them in our code, bringing them into being when we


run our networks. Therefore, to better understand neural networks, we need to follow this story


in the other direction. What is the macroscopic behavior of the neural network given its microscopic


details? Finding the neural network equivalent of the fundamental laws of thermodynamics


could help the development of intelligence in silico in the same way understanding thermodynamics


unleashed the full power of the steam engine during the industrial revolution. Of course,


this might probably just be wishful thinking, and there are many differences between neural


networks and physics, but the theory that we're going to discuss in this video is a


step in that direction. Because we want to illustrate the ideas from the theory in the


simplest way possible, we'll only use basic MLPs with Gaussian weights and biases. But


in principle, there is no reason why it can't be extended to more modern architectures,


such as CNNs and transformers.


Okay, so we want to understand what happens when neural networks become macroscopic. What


exactly do we mean by this? A network becomes large as the width of the network, so the


number of neurons in each hidden layer, increases. With an increasing number of neurons, the


network becomes more and more independent of the details and can be described by a small


set of variables. However, if we go all the way to infinite width, too many details would


be averaged out. The model you'll end up with has mathematically pleasant behavior


and nice closed form solutions, but on the other hand, it has lost a lot of what makes


neural networks interesting and effective in the first place. We're essentially approximating


cows with perfect spheres. For some of the calculations, this infinite width approximation


works fine, but for others, our cow will need her legs back, meaning we don't go all the


way to infinity and rather look at what happens at large but not infinite width.


As we'll explain in more details in a bit, we can describe this mathematically as a series


expansion in powers of one over the width. In this expansion, the zeroth order term is


a Gaussian or more precisely a Gaussian process and is very easy to work with. The other terms


in the series scale with powers of one over the width, where the higher order the term,


the more complicated its structure is. So the correction to the Gaussian model form


a hierarchical structure of increasing complexity. This is great news, because this means that


for sufficiently large N, we can keep only the simplest correction term and neglect all


higher order terms as long as we are fine with living with a small error. By doing this,


we strike a nice balance between the theory remaining mathematically tractable but still


allowing for non-trivial finite width effects. We will show you how this can be used to derive


optimal initialization of neural networks by tuning the variances of the weights and


biases and understand how the activation functions influence this. However, the theory is much


broader and also covers training itself and how to improve that. That is actually the


topic of the next video in this series, so stay tuned!


First, let's understand how a Gaussian distribution arises in infinitely wide neural networks and


how finite ones deviate from this. To get a feeling for this, we'll play around with


some neural networks and see what we get. To start with, we'll take one of the simplest


neural networks there is, one with just two hidden layers and only one neuron per layer.


You probably already know this, but just to shortly recap how forward propagation works,


we start with input data, this is the vector x, in our case three-dimensional. Then it


is multiplied by a weight matrix visualized as the connections in the first layer to get


the pre-activations of the first hidden layer. Now we'll not include any bias terms to keep


it simple. We'll call this Z1, where the 1 refers to the fact that this is the first


hidden layer. Next, an activation function sigma is applied to each of these neurons.


This gives us the so-called post-activations. In our case, we choose a ton-h activation


function. This is repeated for the other hidden layers to finally produce the output y. We


want to investigate what kind of statistical distributions the hidden states of this neural


network are, so let's start recording its values into histograms. Then we reinitialize


the network and see what values we get this time and record them. As this process is repeated,


distributions emerge. Note that the input data is kept the same, so what we're seeing


is the result of randomness in the weights, not in the data. In other words, the weights


induce a distribution whose parameters are a function of the input data. So just keep


in mind that we treat the dataset as fixed and right now we're looking at a single sample


from this dataset. So what did we get? For the first distribution, we see that it's


exactly Gaussian. To illustrate this, we can draw a Gaussian with the same variance and


mean as the empirical distribution and see that the lines match up pretty much exactly.


This makes sense, since the neurons in the first layer are equal to the sum of the inputs


multiplied by Gaussian weights, and the weighted sum of Gaussian variables is also a Gaussian


variable. After the activation function is applied, it's pretty obvious that the distribution


isn't Gaussian anymore. For one, it gets squeezed between minus one and one, the minimum


and maximum values of the ton-h function. Moving on to the next pre-activation, is this


Gaussian? It might have become a little more Gaussian, but it's a bit too pointy. So even


though this layer is also a weighted sum of one element of Gaussian variables, the weight


is multiplied by the stochastic variable sigma of Z1 rather than a fixed input data vector.


And when a Gaussian variable is multiplied with another stochastic variable, the resulting


distribution in general is not Gaussian anymore. Therefore, this layer is non-Gaussian as opposed


to the first one. But this is again expected. After all, we promised it would only become


Gaussian for infinite n, and n equals one is still a bit off from infinity. So let's


move one step closer to infinity and set n to 2. So we add an extra neuron to the hidden


layers and repeat the process. Just like before, the first layer is Gaussian, whereas the second


is not, but this time the second one is slightly wider. To see better how it has changed, let's


draw back in the distribution from the previous network and the Gaussian. Indeed, the difference


to the Gaussian distribution, the purple line, seems smaller compared to the n equals one


distribution. And this difference shrinks even further as n increases. So for sufficiently


large networks, all the pre-activations of the hidden layers are Gaussian distributions


with a variance that is a function of the input data and deviations from this scale


inversely with n. The reason why this is happening might already be familiar to you. What's


at play here is essentially the central limit theorem, which says that when you take a properly


scaled average of a number of random samples drawn from a probability distribution, this


average will tend towards a normal distribution. This is actually the backbone of a lot of


applied statistics, where we don't know the underlying distribution, but knowing that


it becomes Gaussian is enough. In neural networks, the value of a neuron is also a sum of random


samples, namely the post-activations multiplied by their respective weights. The scaling of


the variance makes sure this sum is properly scaled. We can split off the effects that


come from the width and write the distribution as a series expansion in 1 over n. At the


zero's order, we find our Gaussian, which is indeed the only term that is left when


n goes to infinity, and all the other terms, scale with increasing powers of 1 over n and


reshape the distribution.


So why do we need to add correction terms to the Gaussian? Obviously, it gives a better


approximation, but there are also a number of qualitative phenomena that a distribution


with non-Gaussian corrections can handle that the purely Gaussian one can't. To see where


these differences come from, let's start with the expression for the Gaussian probability


distribution, describing a single, pre-activation neuron in the infinite width limit, like in


our previous experiments, and build things up from there. Here the neuron value is called


Z, it's centered around the mean nu and has a variance of k. Normally, there would


be a constant in front to normalize the distribution, but we'll leave that out for now. The first


thing to note is that the mean of the neurons is zero. The network is initialized using


Gaussian weights that also have mean zero, so we don't have to worry about this.


Like we saw before, the variance is a function of the input data. For each data point, the


distribution will have a different variance. To account for this, the variance has to become


a covariance matrix, which tells you what the variances corresponding to the different


data points are on its diagonal, but also covariances between data points in the off-diagonal


elements. We'll label the data using Greek letters. To keep track of this, we'll write


the matrix multiplication as an explicit sum. The covariance matrix can also be viewed as


a kind of kernel or similarity metric. You can take two data points and the kernel will


tell you how similar their resulting neuron distributions will be. For MLPs, the covariance


matrix of a layer is a function of the inner product between the post-activation values


of the previous layer. To get a distribution for all the neurons in the layer, we generalize


this expression further and add neuron labels with a lowercase roman letter.


Okay, so our distribution P of z now describes all the neurons for all the different data


points in a single layer and we have a z for each neuron and each data point. Note that


the covariance matrix doesn't depend on the neuron index. This implies that the distributions


of all the neurons are the same. It has to be like this. That's because if we were to


interchange two neurons, nothing would change, so they're all the same. This also means


that the neurons are statistically independent. Recall that two variables are independent


if their joint distribution is equal to the product of their individual distributions.


Because we have a sum in the exponent where each of the terms depends on a different neuron,


we can easily rewrite the distribution as the product of the distributions of the individual


neurons. You might think that the fact that we ignore correlations between neurons is


overly simplistic, but this is a feature, not a bug. The simplest level of approximation


should be simple. As a consequence, a lot of the formulas can be expressed in terms


of easy one-dimensional Gaussian integrals. However, there are indeed two limitations


that an infinite-width network has and which can be fixed by adding finite-width corrections.


These are the inability to wire and to learn representations.


Wiring is a concept that originates from Hebbian learning, a neurological theory about how


neurons in the brain adapt as they learn. It is often summarized by the phrase, neurons


that fire together, wire together. In the context of artificial neural networks, this


refers to two neurons that are correlated such that if the first neuron has an atypical


value, so it is firing, the other neuron will be more likely to have an atypical value as


well. Neurons that deviate similarly like this will be pulled together during the training


process, strengthening their correlation. This is what the wiring refers to.


The other thing that the infinite-width network cannot do is representation learning. When


a finite-width neural network is trained, the distribution of neurons in the hidden


layers will evolve, meaning they learn non-random internal features. But in the infinite-width


limit this process gets washed out, meaning the distributions before and after a step


of gradient descent are equal. In such models, the second to last layer outputs a set of


random features and the output layer is basically a linear model that is fitted on these. It


therefore doesn't matter how many layers it has, which completely defeats the whole


point of deep learning. We also wouldn't have had to bother with SGD since training


the network has been reduced to a simple linear regression problem for which there is a closed-form


solution. This is why infinite-width neural networks are said to be linear.


The root cause of these issues is the uncorrelatedness of the neurons. Wiring requires correlations


between neurons and representation learning requires correlation between layers. As promised,


finite-width networks alleviate this by virtue of the fact that they have higher order terms,


the first of which is V. If we include only this one, the expression for the distribution


looks like this. To avoid getting confused by all the indices and sums, let's restrict


to only two neurons and a single data point, so we don't need to write the data indices


and can label the neurons with 1 and 2 respectively. The previous argument that we used to show


that the neurons are statistically independent, namely that we can factorize their distribution,


doesn't hold anymore. The Z1 squared times Z2 squared terms ruins this by mixing up the


neurons. We don't have the time to go fully in-depth into how exactly this causes wiring


and representation learning, but to give a rough explanation. The new V term acts like


an excess kurtosis term, that makes the distribution more sensitive to outliers, causing a wiring


effect with a strength proportional to V. And the V term modifies the effective covariances,


allowing for nonlinear effects.


Now that we understand the structure of the probability distribution of neural networks,


there is still the open question. How do we even go about computing it? So far, our analysis


has been quite abstract and you might expect that computing distributions for all the layers


of a deep neural network gets very complicated, but luckily that's not the case. We've


seen that a network is completely specified by its covariance matrix, in terms such as


V that are functions of the input data. So if we can compute those, we know everything


we need.


There are two insights here that we can use to calculate these. First, we already know


exactly what the distribution of the first layer is, even at finite width, namely perfectly


Gaussian as we saw in our experiments. And secondly, the layers are related to each other


via the layer equation, which tells you the values of a layer as a function of those of


the previous layer. If we can somehow bootstrap from the first layer to the second layer,


then from there to the third layer, etc., we can work our way through the whole network.


Indeed, that's what we will do. For all of the parameters that define a neural network,


we can derive the so-called recursion relations that will allow us to compute their value


as a function of the values in the previous layer. Then we simply apply this repeatedly,


starting from the known values of the first layer and find all of them.


Let's illustrate this by deriving the recursion relation for the variance at infinite width,


resulting from a single data point. Don't worry, this will only take a few steps to


derive, but we'll show you the basic idea.


Okay, so how do we turn the layer equation into an equation of variance? Well, we can


simply compute the variance of both sides. Recall that B is the bias term, W i are the


weights and sigma is the activation function. On the left-hand side, we have by definition


the variance of layer L plus 1. And on the right-hand side, both the bias and the weights


have predefined variances, C B and C W divided by N respectively. All the neurons have the


same distribution and are independent, so we can ignore the index i. And that's it.


The left-hand side is the variance of layer L plus 1 and the right-hand side is a function


of the variance of layer L, because the expectation value is computed using a Gaussian distribution


with that variance. Let's also appreciate how simple this expression is. We've condensed


a complicated high-dimensional neural network into a simple one-dimensional integral formula.


In the finite width limit, the above calculation would still hold, but we would have an error


term of order 1 over N. And when we apply this recursion relation multiple times, these


error terms accumulate linearly and will therefore scale like L over N at layer L.


Before we go on, we should probably check empirically how well the theory actually works.


So far, this has all hinged on the assumption that the terms in the 1 over N expansion shrink


fast enough that you get good approximations for realistic neural network sizes. In addition,


we also saw that error is accumulated as the network gets deeper. We take a network with


ton-h activation functions at different widths. On the x-axis we plot the layer number and


on the y-axis the variance. We first work out what we would expect theoretically, starting


from an initial variance of 1. This is what the dots indicate. Then we run some simulations


at different widths. The variances can easily be found by recording the pre-activation values


squared. As the width increases, the empirical results lie closer to the theoretical one,


which is to be expected. Already for a width of 50, which is well within the reach of what's


practically possible, the errors have already become pretty small. Furthermore, it seems


that for larger values, the variance converges to a fixed value, so at some point the error


levels out. This of course depends on the variances of the weights and biases and the


activation function, so this is by no means a general result. But it shows that it is


possible to get a good correspondence between theory and practice even for relatively narrow


neural networks. Before we go on, let's give some interpretation


to our new perspective on neural networks. We saw that we can understand neural networks


as sequential processes that change a representation of the data from layer to layer. These representations


are expressed as covariance matrices and other parameter tensors like V and together they


define the neuron distributions of each layer, or in other words the internal features. Our


formalism tracks the evolution of features throughout the network. When the network is


trained, we start from a representation, which contains all the information and step by step


this gets distilled into a more abstract, higher level representation. This is very


reminiscent of a heuristic picture of how neural networks work internally called the


hierarchy of feature maps. You've probably heard this idea before. In a CNN that can


detect cows, this would look something like this. The features of the lower layers are


responsible for detecting small, low-level features such as edges, lines and textures,


and in the middle layers these are turned into mid-level features like shapes and basic


patterns, which are then further combined into high-level eyes, ears, others and other


body parts and finally the most abstract representation, namely the label, in particular whether there's


a cow or not. Another idea that is similar to this one is


one of the most widely used techniques in modern physics, called renormalization, which


shows up everywhere from phase transitions, superfluidity and magnetization to particle


physics and certain formulations of quantum gravity. This is a process of removing unwanted


degrees of freedom. Sadly, we don't have time to go deeper into this, but the techniques


used in renormalization are the basis for our analysis in the next part.


We can imagine the parameters of our distribution, such as k, starting at a point in parameter


or theory space and watch how they run as we repeatedly apply the recursion formula


with the layer number acting as a kind of time index. In this diagram we see the values


of the covariance matrix of a neural network with Jell-Liu activation functions for different


start positions. The x-axis shows the variance of the neurons induced by a data point labeled


alpha, and on the y-axis there is covariance between data points alpha and beta. The counter


on the right indicates which layer we are at. This is known as a representation flow.


One thing we immediately notice is that the values that start in the bottom half all converge


to a single point, whereas the top ones drift off to infinity. This is called a fixed point,


which is defined as a point in parameter space that when you apply renormalization you end


up at the same point again. We treat infinity also as a point, so we can say that all parameters


in this graph converge to a fixed point, which is about 1 for the parameters starting in


the bottom half and infinity for the ones starting in the top half. We also saw this


behavior in the experimental verification in the previous part, where the quantities


converge to a specific value. We can compute a value of the fixed point by taking the recursion


equation and solving it for kl equals kl plus 1, which we'll call k-star. Like we said,


at this value by definition the input value will be the same as the output value. There


are essentially two different kinds of fixed points. Trivial fixed points, where the value


of the parameter diverges exponentially and shoots off to infinity or dies down to zero,


these destroy the structure of the data at an exponential rate. This is what you want


to avoid when training a neural network. Non-trivial fixed points are those where the value of


the parameter converges to a finite point, preserving some of the structure of the data.


Flowing to a non-trivial fixed point seems to be a necessary condition for a deep learning


network to perform well. This can include the fixed point value zero as long as the


convergence isn't exponential. This technically also destroys the data, but at a manageable


rate, and in practice we see that this also works well.


We can further characterize fixed points by looking at their immediate neighborhood. Imagine


you are drifting close to a fixed point. Again, there are several things that can happen.


It could be that the flow will suck you into the fixed point, or that you drift away from


it and if you are exactly on the cusp, you remain at the same distance. To quantify which


direction the current will take you, we introduce a parameter chi, called the susceptibility.


For chi ≤ 1, you will be pulled in at an exponential rate towards the fixed point,


while for chi ≥ 1, you are pushed away exponentially, and for chi ≥ 1, you stay at the same distance.


For neural networks, since we are working with higher dimensional vectors, the deviation


from the fixed point can be both in the direction of the data or in parallel and perpendicular


to the data. This means we have two types of susceptibility rather than one. The first


quantity is called the parallel susceptibility, obviously, and the second one is the perpendicular


susceptibility, also obviously. To get a bit of intuition, the first corresponds to the


difference in magnitude between two nearby inputs as they pass through the network, and


the latter corresponds to their relative distance but also controls vanishing and exploding


gradients. In practical terms, a susceptibility of 1 is beneficial for neural networks, because


this means that points that are apart will not collapse into each other. What we find


in practice, though, is that having a susceptibility close to 1 is enough to get well-behaving


flows. We also see that the perpendicular susceptibility is much more important than


the parallel one. This has probably to do with the fact that it controls the size of


gradients and lastly exploding perturbations such as chi-perpendicular greater than 1 are


more harmful than shrinking ones, chi-perpendicular smaller than 1. The equations for these susceptibilities


aren't too hard to derive, they're basically taking derivatives of the fixed point equation,


but we've done enough maths for today. In the end, we have three equations that determine


what values CB and CW have to be a network that is theoretically initialized optimally.


Now, to the part which really shows the power of the theory. Strictly speaking, all of what


we have just said only holds for neural networks at initialization and there is no theoretical


guarantee that any of this generalizes to the rest of training. But what we find out


in our experiments is that actually, the initialization has a strong effect on how well a network


trains and is largely determined by the parallel susceptibility. Having said that, we should


keep in mind that these are vanilla MLPs, so modern tricks that make sure networks train


better such as skip connections and batch or layer normalization are not considered.


For the experiments, we ran some sweeps over ranges of the variance of the weights and


biases CB and CW respectively for a 50-layer MLP with a hidden size 500. Each network is


trained for 10 epochs on MNIST and at the end of each training run we recorded their


accuracy on a separate evaluation set. Solving the previous equations, we find that there


is a stable fixed point at CB equals 0 and CW equals 2. This is also known as sky-ming


initialization, which is just a special case of our methods. ReLU has another special property.


When CW equals 2 and CB is not equal to 0, the variance grows linearly in the layer number.


Since this is not an exponential growth, this is technically also a non-trivial fixed point.


The line represents the values of same perpendicular susceptibility and the background color is


the accuracy obtained on the test set. So basically, the brighter the better performance


the network. The solid line is the theoretical optimum, which lies indeed in the region with


the best accuracies, but values near the line also perform well. Performance drops off quicker


for chi-perpendicular greater 1 to the right of the solid line than for chi-perpendicular


smaller than 1 on the left. This agrees with the idea that vanishing gradients are less


bad than exploding ones. The yellow high accuracy is well enclosed by the dotted lines representing


the range chi-perpendicular in between 0.8 and 1.1. Overall, this shows a very strong


agreement between theory and practice. Repeating the same analysis for tanh activation functions


gives the following plot. We see again that the high accuracy region is nicely enclosed


by the dotted lines and that higher than 1 susceptibility is worse than lower than 1.


It is kind of hard to see, but as we follow the lines upwards, the accuracy slightly drops.


This is due to the parallel susceptibility, which decreases for larger values of Cp.


For Jellu, we see again that the accuracy is strongly correlated with the perpendicular


susceptibility. On the left side, the lines coincide with the yellow region. On the right


side, however, the lines move into the low accuracies. This is an area of quickly changing


susceptibility, causing this to be an unstable region. Rellu 6 is the same as Rellu, but


values larger than 6 get squashed down to 6. This causes the lines to tilt to the right.


This is also reflected in the accuracies that we find from the experiments, which again


match up with the lines.


So I hope you are convinced by theory and experiment. And wow, what a video this was!


I hope we have a better understanding now of what weights and biases and activation


functions do in a neural network and that we have a pretty good idea how to initialize


the network optimally. Thanks to Carla for visiting us today and thanks to Kun that took


care of the script of the video and the cool visualizations in Mannim. This video would


not have happened without him. And thanks to Pip for the amazing drawings. And of course,


thanks to LfAlpha for making this video possible. LfAlpha from Heidelberg, Germany is one of


the Europe's leading AI research and development companies. Loaded with top international AI


talent, LfAlpha researches, develops and operationalizes large generalizable AI models. Just very recently,


they released two large language models from their Luminus series. The Luminus series offers


users for the first time a made-in-Europe alternative to large AI language models from


the USA or China. Their multilingual multimodal AI models can be accessed via an API as well


as through an interactive user interface. Additionally, they just released a multimodal


method called Magma, which enables pre-trained transformers to use images as input format.


I mean, it even recognizes Ms. Coffee Bean, look! LfAlpha is committed to spark constructive


discussions with research and open source communities to enhance the development of


transformative AI applications. That's why they recently open sourced the codebase for


Magma. Find the link to the paper in the video description. From research to development


to implementation, LfAlpha strives to align modern generalizable AI research sustainably


with ethical values. To achieve that, they continuously search for the best talent and


partners. For more information, visit their playground under app.lf-alpha.com and don't


forget to subscribe to their Twitter account via the handle lf__alpha.


It's going to be a big (and fast) Summer of learning | Machine Learning Monthly August 2020

You know what time it is. We're a few weeks late this month but as you might be able to tell if


you've seen a previous video, I'm in a new space. I kind of moved place so I needed to reorganize


my studio. I'm still figuring out a few little kinks in terms of like there's a road you can't


see it but there's a road like just there. So I'm kind of figuring out whether how the best way is


to record audio. So if you have any feedback on hey Daniel this video sounded like crap leave a


comment below but otherwise what we're going to do is every month I write an article called Machine


Learning Monthly. Let's have a look at that. Let's get that on the screen. Boom there it is on the


Zero to Mastery website. There's the handsome chap that wrote that, George Trulli and it just collects


all of the or about 10 or so things of the coolest stuff that I found in machine learning over the


last month and because it is September, it is September 19 today and we're going to go through


last month's. We're going to make a video version of this and we're just going to have a little bit


of a chat and so I can catch you up on some of the latest and greatest in the machine learning field


and it's not always about the latest and greatest. There's also stuff in here that could be a little


bit old now but it's it stood the test of time and it's amazing and it's worth checking out if you


want to of course. But that being said let's get into it. So this is the Machine Learning Monthly


August 2020. I'm not actually sure what this one will be titled yet but usually we title it based


on something in here. If you want to get this article as a newsletter in your email inbox once


a month and go to this website. It'll be the top link in the description. You can sign up and then


you'll get this every month. Otherwise you can just watch these videos on the channel and just


hear me talk about it. Though the article is probably a quicker read because I tend to be a


bit verbose. Alright so what you missed in August as a machine learning engineer. So this is what I


did. I wrote an article called How I'd Start Machine Learning Three Years In. So three years ago


there's my old room. Brings back a tear. If you want to know more about the the new place leave


a comment below and I'll might do a video or something like that about it. I really like it.


Anyway I started learning machine learning in this room and three years on it's about actually


three years to the day today. I think it was September 17th. Anyway that doesn't matter.


I know a lot more than when I started that's for sure but let's be real. I'm still figuring this


out too. So anything I say on all my videos always say this take it with a grain of salt.


Trust your own intuition and so in this article I put together a few points. Things mostly pieces


of advice that I tell my past self. So things like the curse of the the engineer the technology nerd


is use less tools. If you want to build things that's the number one thing that I would change.


I won't dive too deep into this because you can read the article. I would spend more time


building things. So getting projects into the hands of others. That's that's my sort of nature


has been more practical and so that's the major thing that I would change. There's also a video


version to go along with it. There's my old haircut. Thankfully I've grown some more hair


since then. Anyway that's not what you're here for. You're here for the best from the internet.


Now is my face getting in the way there? Nope. I'm not going to read straight from the article. I'm


just kind of going to click on the resource and then talk about it. So there is nothing more


that I love. Oh this is going to play out loud. I'll mute that. Check out this. Look at that.


I need to figure out how to do animations like that. If you know a resource where I can learn


to do these things I might actually reach out. So this is an amazing series on a visual introduction


to neural networks. So if you're like me I like to learn things with multiple senses. So with my


hands I'll write it out. With my eyes, with my ears. I like to listen to things especially if


I'm learning something for the first time. I like to hear someone else talk about it for


however long and then try to reinterpret it in my own words. But this is a little series by the


YouTube channel V Cubing X. Definitely worth checking out. Subscribe to them. On neural


networks just look at that. The inside of neural networks. So I mean it's one thing to write neural


network code. It's another thing to know just what's happening inside. Look at this. See I'm


making things with Keynote and these guys are making it with full-blown animations. So note


to self step up your education game Daniel. Really big shout out to this series. So go check out V


Cubing X. I'll leave a link to all of these things in the description by the way or you can just jump


back to this article and they'll all be there. Anyway that was number one. Number two is using


machine learning to detect deficient coverage in colonoscopy screenings. Now I thought this was


pretty damn cool to see as my face I might move my head at there. There we go. I thought this was


I thought this was pretty damn cool to see Google or how machine learning is being used in healthcare


to aid like physicians. So I don't think it'll ever fully replace like this is what a lot of people


get worried about is will machine learning or AI replace my job. I would say if it's a continually


repetitive task chances are that AI will eventually be able to do it better than you.


But in terms of being able to provide care for a person I think AI is a very long way off there. So


what my take is in the in the healthcare space of machine learning and healthcare is let computers


do the computer stuff like the calculations the finding patterns in data etc etc. Let the people


do the people stuff like just simply being there for someone who who is unwell and just needs the


the the space or that the energy of someone else around them. That's what people are good at. I


think we're a fairly long way off for that of computers. But essentially what Google did was


they use a machine learning based algorithm. Let's get some photos. So what happens with


a colonoscopy is you get usually get a camera shoved up you know where and what it's going


through close your eyes if you're not willing to look at some some graphic stuff. It's going to go


through and take photos of your intestines. So this is what we all look like inside.


And then what it's looking for what the camera is looking for what a doctor usually looks for


is something called a polyp. Now I don't know if there's an maybe yeah see a motorbike just went


past and that's the kind of thing I need to figure out in this new studio is that there is a road


right there. So this is the kind of thing that it's looking for. But the problem is is that when


the camera goes through. So I'm not sure if we go here. Yeah. So it can detect where the coverage


is deficient. So you imagine right. You're getting a camera going through your intestine.


Your intestines fairly large. There's a chance that the doctor might miss some portion. Let's


just say I can't I don't know which is missing which is not but let's say it missed the red


portion here. And then in that red portion was actually a polyp. And so that polyp turned out


to be dangerous. So even or cancerous. So even though you got this scan right there was still


missing potentially cancerous tumors inside you. And so what the they called it C2D2.


C2D2. What this does is it uses an unsupervised learning method and it kind of figures out. Hey


we're going through your intestine or like the camera. We've looked at all the camera footage.


This is the algorithm right looks at all the camera footage and goes you know what


we've reconstructed this this person's intestine and we think all of these red parts you've


actually missed. So it's probably worth going back through and checking out these red parts.


So we can find something like this. Now I think the metric here that was worth paying attention to


is this is performance on synthetic videos. So of course take that with a grain of salt


because it's synthetic. The real metric is real videos that you want is that mean absolute error


which indicates the how much the algorithm's prediction differs on average from the ground


truth. We find the mean absolute error is below 0.1. That's amazing. So meaning the prediction of


C2D2 is within 7.5 percent of the ground truth on synthetic videos. But on contrast a group of


physicians so a group of doctors given the same task achieved MAE 0.177. So almost over double


2.4 times. So the algorithm on synthetic videos is 2.4 times better than physicians. So that's not


to me that doesn't replace physician that just gives them another tool to use in their arsenal


to provide better care. And then we got here performance on real real videos. I think I


remember they said it was difficult. Yeah and difficult to assign a score directly.


The task of verifying so what the test here was the physicians verified the algorithms predictions


and so C2D2's score matched up with physicians 93 percent of the time. Now that's that's pretty darn


good. And so here's some just demonstrations of what it looked like. So here's a poor score. You


can see that they're pretty poor images. Here's a fairly OK score and here's a really good score.


Right so they're pretty clear. I know it's like a full blown tunnel through your intestine. That's


crazy. So check that out if you're interested in machine learning and health care. This is


this is cream of the crop stuff for you of how machine learning can be used in the health care


space. Let's go back. Of course and why that matters is because colorectal cancer results


in nine million million deaths per year in the US alone. So that doesn't even include my home


country of Australia. That doesn't include if you're not from the US. I mean colorectal cancer


could be pretty prevalent in your country too. Anyway on to number three. Oh I'm very very


excited about this one. Now this is almost a month ago so it seems old but it's as I said


these things are these things are evergreen. Fastai courses are some of the best online. Well


I would actually if I was to say if you were to do one course on AI it would be fastai. Now I


haven't personally gone through this yet but I just know how quality it is. I do actually have.


So this is this is a multi part one. So if you want to learn deep learning check out fastai2.


There's actually four big releases and I've got one of them. So not only do we have fastai2 which


is a library we have three other mini libraries and we have another course and we have this book


Deep Learning for Coders with Fastai and Pytorch. This just arrived the other day.


I'm so excited to go through it. I think this is going to be my summer study.


So Australia our summer is December to January so I think this is what I'm going to go through


fastai version 2 and read the book over the summer. So I'm really excited about that. So


check it out there's a new library brand new. I remember my first real interaction with fastai was


when I was working at a Brisbane tech company as a machine learning engineer and we were stuck on


this natural language processing problem and I watched one fastai lecture and then applied the


technique from that fastai lecture to the problem that we had in natural language processing. It


was text classification and the technique that I applied from fastai wiped the floor seriously with


everything that we'd just been spending a week working on and that ended up being the model that


we deployed in practice the fastai model which is built on top of Pytorch. So incredible resource.


Jeremy is my favorite teacher out there. Go check him out. Go check out fastai's new stuff all of


their new stuff and of course read their blog posts. They're actually real good quality blog


posts and examples of how you can share your own work and if you don't have your blog to share your


own work use fast... is it fast pages? Yeah this is a bonus fast pages. Boom here's how you can


make your blog fast pages. All right we're getting distracted and there's a couple of bonuses in


there like the podcast that Jeremy did with Lucas Biewald who's the CEO and founder of weights and


biases another one of my favorite machine learning tools and there's also the Zero to Hero with fastai


series by Zachary Mueller. Oh it doesn't work. Anyway so that might be I would just google that


google Zero to Hero series with fastai zero to hero with fastai.


Boom. Oh.


There we go. So yeah google if you want that that's a great series of blog posts by and who


was it about? Zachary Mueller. I just said his name. So big shout out to Zachary Mueller. That's


some amazing work of deconstructing say a new tool or a new way of doing things into his own words.


So not only did he take the time to sort of learn himself he took the time to put it in a way that


you can understand as well and that I really find like that was one of the points in my last video


was if you want to learn something teach it so give that a go. Anyway number four is Min GPT by


Andre Kapathy. So if you've been browsing the internet of machine learning you've probably found


that GPT-3 is phenomenal. Like the results it's getting it's just it's just hard to even comprehend.


I mean in a test where GPT-3 was asked to produce a fake article or just to write an article of


about 500 words in length the full version of GPT-3 so I believe it's 175 billion parameters


was producing articles that were almost or basically indistinguishable from a human written


article at 500 words of length. I think it was a team of humans were were asked to decipher was this


GPT-3 or was this a human article and they got about 52% accuracy which is basically a coin toss.


Anyway Min GPT is a pure pie torch implementation of a mini version of GPT and this basically


explains it. So this is GPT-3 big warship and this is Min GPT but if you want to see how a giant


language model is built well then this is the repo for you because I think it's about 300 lines of


code or something like that. If we go in here model there we go 10 look at that 200 lines of code.


I mean there might be some some extra things here like this is a trainer.py I think this


is just boilerplate.py to watch the end there we go that's about 100 lines you know what a good


little project would be to just basically if you want to understand GPT because GPT is the


architecture that sort of builds GPT-2 GPT-3 would to be just to go into this repo and rewrite the


entire thing or copy just copy it line for line and so you can understand it yourself. That's what


I find really helps me if I'm trying to understand something I just literally look at someone else's


code base and retype it myself. So let's go into here and what do we have number five less supervised


computer vision. So that's what I'm noticing a trend lately is that I think that was the last


machine learning monthly the trend was or the video was called the future will not be supervised


and it makes sense right. So if you think about supervised data how much effort does that take to


label it all and just collect it and whatnot. So now with GPT-3 that's an example that's a completely


unsupervised model yet it's getting results that are better or if not equal to supervised learning


methods and so I've got a few links here on on different weekly supervised methods in computer


vision learning with limited labels tutorial from NVIDIA at ECCV I think that's the European


conference for computer vision 2020 and then deep learning with limited labeled data data seminar by


Colin Raffel and if we go here so there's the tutorial on weekly supervised computer vision.


So this is what I'm interested in I think over the next I would say year or so maybe is two


things is uncertainty in machine learning model predictions I really like that and models which


use less data but still perform really well. So check out that if you're interested in computer


vision learning with less labels and same with this similar topic there's a lot of stuff here


on videos look at that so limit labels in a labeless world self-supervised learning so that's


really really really I think this is this is the future of deep learning machine learning is


self-supervised and unsupervised and then here's a great little seminar by Colin Raffel on a whole


bunch of different here we go the content here on basically all of the recent advancements in weekly


supervised learning semi-supervised sequence learning big self-supervised models are semi


are strong semi-supervised learners I think maybe it goes through like the most recent papers yeah


there we go so this is a reading list but if you check out the github repo I believe there is


all of the materials for the for the course as well so big shout out to Colin Raffel there


and now number six is ml ops so this is a tutorial series let's check it out on it made with ml by


the way that's I'm real big fan of made with ml so make sure you check them out too what is ml ops


so if you imagine so machine learning is not just building a model in a jupyter notebook


it's what happens after what so say you did build a model what happens after that how do you get


that into the hands of people how do you how do you build an application around that model so that


someone could use it say on their cell phone so if you wanted to take photos of dogs and identify


what breed it is and you trained a model how would you get that onto a mobile phone so this is where


ml ops comes in it brings all the practices from dev ops which is a big thing in software engineering


into machine learning so if you come here we got here number one it's a great youtube tutorial series


beautiful by the people at dvc dvc.org beautiful I believe dvc is also data version control so


that's a great little tool but check out this if you want to learn more about ml ops I definitely


go through these videos here it also utilizes github actions which is a great tool I'm not


going to dive too deep into that but definitely something worth checking out if you're looking


into getting into full stack ml which I definitely am so maybe I'm gonna I'm I need to you know what


I need to do I need to create myself a summer of learning curriculum and I think this might be part


of it alongside the full stack deep learning course of course and the fastai course we gotta


it's gonna be a big summer anyway that was number six number seven is how to deploy and host a


machine learning model with fast api and heroku speaking of deploying machine learning models


this is something that you might want to check out on testdriven.io blog so basically fastai


api I'll just I'll just let this read it for me fast api is a modern high performance batteries


included python web framework that's perfect for building restful apis it can handle both synchronous


and asynchronous requests and has built-in support for data validation json serialization authentication


and authorization and open api my goodness I'm not even quite sure exactly what fastai api is


but just that sentence alone makes me want to use it but if you read this tutorial the goals are


develop a restful api with python and fast api build a basic machine learning model to predict


stock prices deploy a fast api app to heroku now heroku is a cloud provider and what that means


is that it's some some computers in a big warehouse all around the world and if you deploy your


application to there what it means is if you've got a script say a python script that's a machine


learning model that predicts stock prices if you put that onto heroku if you do it in the right way


it means that say if I did it I could then share a link with you to go and check out the code that


I've I've written in a website so really cool tutorial there if you want to learn how to deploy


a machine learning model looks like it's all in pure python so you look at that how cool is that


an extension to this project would be figure out how you could do it with


take the machine learning model and build a streamlet application around it


but anyway that was number seven number eight I actually really liked this one so this is a great


another example of how machine learning can be used for me it's in a healthcare setting or just


in a in a product based manner and speaking of products this is on device supermarket product


recognition so you imagine I think it's not only just so where's the demo


there we go that's a really cool demo by the way this is using augmented reality computer vision


and a bunch of just normal software engineering processes so what it does is it uses it looks at


what product it is and then brings up information about that product so what they said at the top


google is one of the greatest challenges faced by users who are visually impaired so of course this


this app would definitely be helpful for someone who's visually impaired but I think it's not even


not even for those who are visually impaired it could be for anyone who just wants to learn more


about the the food that they're eating so I think this I've spoken about this a lot before but I


think this I want to work on some sort of project like this in the future maybe that'll be like


my next end-to-end series like the airbnb series because at the end of the day


like food labels or just food in general could be quite confusing I mean you've been told all these


different things of what to eat what not to eat so this is where I see a use case for this has


been really helpful in not only helping people who are visually impaired but just helping people


who just want to know more about what they're eating in general so check out this blog post


I think it's all built with media pipe as well which is an open source framework yeah look at


that media pipe that's really cool for building I think it's yeah an open source computer vision


framework for putting together multiple computer vision models in in succession so if you imagine


I believe the article says it goes to accomplish this look out so that's what the app is called


includes a supermodel product detection and recognition model and an on-device with


an on-device product index along with media pipe object tracking and an optical character


recognition model so yeah media pipe is creating a pipeline of different computer vision models


character recognition models that the product index is on device so that makes sense


but really cool and so the resulting architecture is efficient enough to run in real time entirely


on device so important because if you're in a supermarket you might not have that great an


internet connection and you want it to be quick you don't want to be sitting there waiting for


information from from your phone so check that out if you're interested in uh building products


like that and finally number nine I'm really excited by this one actually so my last video


or one of my last videos you saw that I built a deep learning pc using a navidia titan rtx well


it turns out that a couple of days later I think it was the first of September navidia just


announced their rtx 30 series which are just phenomenal gpus like they're where's the graph


we've got benchmarks here so and not only are they phenomenal they're also at a an incredible price


point compared to the previous models um compare specs okay I think it was maybe it was in the


keynote anyway I'll let you check that up because um unless it's somewhere here


anyway if you want to learn more about these I do have a great blog post and Tim there we go


there we go which gpu if you just google I think it will just come up which gpu for deep learning


this is this is what I'd recommend if you're looking to build your own deep learning pc


you do not need one to get started use colab until you do you'll know when you do um


yeah this post by Tim Detmers is probably the best online for it yeah look at this look how much


this is so in depth if we go here the most important gpu specs for tensor processing tensor


cores of course but there was somewhere where's the graphs here we go so boom if you want to build


it with one gpu so the performance per dollar the rtx 3080 so 10 gigabytes is I think by this graph


it's the best this is the new gpu here and where is it so that's this one here rtx 38 so 699


us so what's that in australian dollars it's about a thousand australian dollars


so check those out some amazing new gpus what this just means is that


building state-of-the-art like hardware is getting more and more accessible I mean that's the trend


that it's always going to take right but at the end of the day it's not the tools that you have