also apply it on continuous features by doing some discretization just before also tabular data can have special special and temporal information in them so you may group by different filters with different granularities and they also derive your own the regularities from - - to make your blue pipe working so we in the two sigma competition within that end up in the second place


just by using those those kind of features so we have some technical indicators and some fundamental indicators and we we did this on some features and our our global our while Peter was the day so for each day we just compute the mean and look at the deviation of each sample of each observation to the mean a touch even day and we end up with different models that we went together and that

Gold is easy: Kaggle tips and tricks | by Chahhou Mohamed | Kaggle Days Dubai | Kaggle


was enough to win the competition just with those kind of the turtle third parts for improvement is about to be on fire so here I'm going to talk about our pliers in the targets that you are trying to predict so our pliers can be due to different reasons so for example the home pricing you may contain some typos maybe you have some under priced sales because this happened between


family members or maybe there was some fraud that happens so you know something that you can not predict because it's under price or maybe you have something over priced some cells that's big houses on some big island but you have very few samples of them so you can not find some interesting patterns not enough samples to learn anything interesting from London so how to deal with this kind of


our piles so during the trading pains you can start by keeping the outliers and have some kind of deadlines for then you can apply some transformation of the target and see if there is any improvements this alone you can do what we call insulation so winds organization and we've talked about it later this is the second thing that you should try and the third thing that you should


try is removing the outliers so winterization is just you need to cut the target so if you see some values that are higher than a given threshold you just put any higher value to that threshold okay or you can also cut the error so joining for addiction during the learning phase you have to write your own loss and what you do is you compute the error and for big errors you


need to cap them so that your model will not will not be biased towards those predators those outliers especially if you are using some metrics like the mean squared error which encases the importance of the outliers so after trying the winterization will have a new spot that you can compare with your baseline and the third approach is the removal of the outliers so how can you


do the removal you just build a model and predict on the train dataset and you have to use a cross validation approach it's mandatory otherwise it's not going to work and you will predict and using those prediction you will get an error and based on this error you will find an optimal threshold and you will remove from your training data all the samples that have an error higher than the


threshold that you have fixed during the testing you should not apply the winterization or the test but regarding the removal of the outliers in general you can keep the outliers in your testing data set unless you are sure 100% that the outliers can make a hug in no case they can they can be predicted so you can remove them from from your from your test also when you see


improvements when you are doing feature engineering and you see some improvement with a new feature how do you know that this feature is important because if you keep the outliers in your testing data maybe with some luck you are predicting a good out player that will improve your score but the feature by itself is meaningless it's just some luck that happened on some of my own so you should


always check on test data with and without our players to see if the improvement is from some kind of general pattern that your model has have discovered or it's just luck some outliers okay some additional tips that I can give you here always try blending your models so what I mean here is doing when working on tabular data you will do a lot of feature engineering


you will sometimes end up with hundreds of feet off so instead of using all those features in just one model you can create make different models with different subsets of features and try to blend together all those models and among with with the the feature engineer that we did in the two sigma competition we tried five different models I mean five same models but with different


subsets of features and did how about a lot because what we have no ties we are we were trying to predict in the future so that was the time series time series problem and we were trying to predict in the future on a large time sample and we noticed that sometimes adding one feature in our model did improve our prediction for some modes but decreased our score for other mounts so each


feature was not working on the whole time or the whole unseen future so they were working on some periods I'm not working on some periods and so what we did is we clustered somehow those features in two different spots though the features that were working on some periods we put them together and those that were working quite well or some other periods we put them on another


model and by blending all those model what else we end up with something that was quite effective on the whole period also another advice that you should always write here is to try to when using begin so you have the same model and you are bugging with different seats try to let your models over fits a little bit by religion them over fits a little bit sometimes the packing will be will


increase your score more than trying to find the best number of epochs for example if you are using neural network when you are using a lot of features try a very small feature fraction so this is one of the parameters of the gradient boosting decision tree this is the number of filters that will be used for each decision tree so most of the time we are using something like 80% or 90%


but if you are using a lot of features you should use something or pill or rather try something like 10% or even 5% of the filters so your trees will be really very low correlated they will be almost independent from each other this will increase significantly your finance ecology some some tips about the neural networks neural networks are doing feature engineering by 5 in 10 so


most of the time and I will fill neural networks I don't do any feature engineering I just try to find to come up with a good architecture platform so when you are doing this and in order to increase the feature engineering parts of the neural network try to apply the impedance also on discrete numerical Peter so most of the time we know we only do impedance on categorically


categorical features this help helps in fitter engineering for the knowledge report but to doing that on the street needle computers you will embed those features and you will probably encode more information in those impedance than the initial value such that you are provided with so I have noticed that those impedance of this place inverter filters always increase by final spot in


this in order always to another network even if you are working on tabular days they work great in ensembles and they have they are almost as good as great important decision so when you are doing feature engineering use it for example I GPM and you have lots of teachers that may be used to do feature engineering it's very difficult to tribe or combinations with all teachers you may


end up with thousands and thousands of pizza before finding boss that may help you so one thing that's I usually don't I find it quite useful is that I make a neural network and the gradient boosting decision tree I have two scores to baseline scores I remove the same feature from both the models and I check the difference in the scores if I see that the spot of the


neural network have decreased much more than the graduate within decision tree then I know for sure that the neural network while doing some kind of fitter engineering using the Twitter so that helped me decide which feature should I focus on when doing little engineering in writing poster decisions another trick does may use may be used to improve the generalization of your


models especially when working with neural network histor and only replaced some of the filters with newer values okay during the training phase you have some some generator and but for each epoch you replace some values randomly with the newer values most of the times doesn't work but sometimes is may be helpful for the generalization also when working with neural networks and this is


this always help when you are working with neural networks when you have a filter a numerical filter with null values always create an additional feature which is just a boolean filter indicating if the corresponding value doesn't know or not because you are replacing the null values into the original filter with some kind of mean or median or something like that and you


need to help your neural network by this additional information so back to the outliers so when you are doing the removal of the outliers from your training set you know that later on you will do some feature engineering and you can you can ask yourself well if I remove those outliers now maybe filter engineering may help me later to predict those outliers color correctly so how


can I decide if the removal of the outlier should be done before or alternate feature engineering so first approach just keep the outliers until you are done with the feat of engineering so you are sure that nothing can help you in predicting the vampires or you can do something that I usually do and that is that is what I did in the second stage of the zero competition is


that I made neural network because neural networks are doing feature engineering implicitly okay so if I see that I'm not able to predict with my neural network my based knowledge network I'm not able to predict suppliers and knowing that neural networks are orange is already doing the feature engineering part that I have some insights that probably feature engineering with rotten garage


in person decision tree will probably not be useful to predict guitar players so this helped me to decide if I should remove to our players soon or Lenten okay so I will show you here some case studies some example of competitions and you will see how it's maybe sometimes easy to get a gold medal or even a top three position so we start with the mercury competition that was a capital


carrier competition so you have limited hardware just one hour to run the training and the inference phase of your model so we we were provided with those eyes with those features the item description that was the most important feature the name which was just this short description of the the item category condition shipping and brand name so when we started this competition


I was in the team and we were doing the team and we focused on Elijah be a model because they they were so good we tried some neural networks some rich models and the light GBM was the best model that we can that we could make and even though it was our best model we could not make it in the top ten of the leaderboard so we tried some pencils with neural network they didn't work we


even tried to add some impedance coming from neural networks as new features in Nigeria and it didn't work and the running time for alleged GBM was so high that we couldn't add any importance other models because we are we were limited to one hour of training so if something doesn't work change your approach we were so sad to throw dancing that water that took us so


much time to eat to make but we have no choice but dropping that model and come up continue so we tried different models that were pretty fast to train and we used to neural networks and one rich model so one model one neural networks used embeddings of the the words as as input and another model just were used to the one hot encoding as input so we have a sparse


input and some kind of dense in H which work or in vengeance so if you look at the architectures of our robots one model is just some dense layers it straightforward feed-forward neural network with a sparse input which is one half hour one hot in fighting so the count vectorizer here one of the parameters that I forget to put here was binary you can equal as through so the


world is seen with the item description Swann otherwise it's you know so it's somehow some kind of one not encoder the second model was using the embedding and the importance see that had to be done is you know is to use a shared embedding because the name the item description were using the same vocabulary the words have the same meaning so you have to use the same embedding photons details and


then we did something that that you can that have been used in the past X which is just try to capture the meaning of the sentence using an average of different different impedance of your sentence of your item description and as you can see there is no competition here just every agent of embeddings and putting forward in these two dense layer was enough to have a


good spot that was our second model and the third model was a simpler each model taken as he put the description in the name filters so in quran 2.1 and we made a cat sound and grin with n equal to 2 and here instead of using the tradition to classical vibrant with a sliding window we use all possible combinations of by grams for the name because the name was so short that we were we could


authorize or sell to use those kind of eyebrows and using classical background and his kind of migrants make the difference in the final concern so why we went for this kind of approach when we notice that the score of each model by itself is not important so even though our light JPM will have the best score well it was not helpful in any ensemble and we note iced by that by


Insane plane or light GBM with a rich model the score of the rich model by itself has no importance so we could let out reach over feet or not over feet the score of the rich model give give us the same final performance so we decided that instead of looking for the performance of each model we should optimize for the performance of the ensemble by Infirmary and we tried to to


art and models that were improving the final in fact another thing that have versus let's let in the models of repeat a bit but have have that helped us in the finally movements we also because we have only one hour to run the car locks so instead of decreasing the learning rate after each epoch to epoch we were increasing the batch size there is a nice paper about


skis that you can find on the internet and also removing the mean from the target before the training was quite help second competition this was a solo competition I did just for fun because I was waiting for the zero results so here we have just one Peter which is you have a question and you should predict if the question sensor or instance so this is an NLP problem I started training from


scratch I have a baseline score [Music] sorry little when I and you cannot question so training from scotch give your baseline it's obvious that it's not gonna work because the data set was too small so next I used some embeddings with fine tuning it was helpful but not enough to to get an eye sore again so didn't work what did work is to use the ambient the impedance and don't


fine-tune your model so the layers were not trainable for the impedance and this did work so I list the model train for some epochs and fine tuning just the last book was I was helpful in improving the score and then by began back in two different models with different different seats the despoiling moved much more so I think that almost everyone on capital used this approach


so what makes the difference in getting that gold medal so here are the few things that I made and were quite different from other controls so because we are we are not changing our embeddings we are the layers are not trainable so you have to find the maximum impedance of every world you cannot learn impedance for words for new words for rare words so you have to find


those embedded so what I did is an iterative pre-processing step if I don't find the word in an embedding I do some pre-processing and check again if if a building is available and I did this so for example tried lower case and upper case in remove the accents etc and check again if the embedding is available and the last pre-processing step this Damon this is the last thing that I do


and if I don't find the the stem of the world Indian bed and then I considered the world as a rare world the nice and this was the most important tips that I have used in the pre-processing and it improved significantly the performance of the model the next thing that I did is because the impedance of the rail words are not available and I'm not gonna train I'm not gonna


learn them because my layers are not trainable so I cannot use them then when I will do some global enumeration or global max so I always multiply those entities by zero otherwise I will use some random values that will not change during the trader this part this was quite helpful in to improving the scroll and later I didn't use the global average chain because I have a lot of


zeros embedded in my in my my inputs because of the padding and because of the rare words that I multiply by zero so I have a lot of zeros and averages would not work so what I need is instead of mid narration I just sum up the values of the intervals then I apply some budget augmentation and here is by our total that's the full network and the spread is straightforward to


understand so I have an input which is which is the question by itself and another input which is just a binary indicator telling me if the end if the the world is in the middle or if the world is a rare world which is not in the ended then I use the the globe in bed in here so I don't initialize my impedance with some randomness that I use a transfer learning to initialize my


impedance and then I multiplied this is the tricky part here I'm not supplied by embeddings with the mask so rail words will be multiplied by zero impedance with the equal to zero otherwise I will use the transfer learning then there are two parts here this is the first X approach you just try to capture some kind of meaning of the sentence so usually what we do is average all the images but the


Brazil was not working because I mostly have zeros my sentences so what I did is just some the impedance apply much normalization with my second part of the network is just use an STM or on my input and applying a global maximum and again I cannot use the global average plane I just used this found concatenate everything and straight forward turn the neural network as you can see quite


simple but those three little tricks make the difference and I end up just in a few days of work in the 14 place I was ragged 14 and I got about middle which is which something as simple as that the last competition I will talk about is a recommendation system to or to music so the problem is the following we have a user who is listening to some song but sometimes type t1 and we want


to predict if the same user went to read listen to the same song at some time same cheap g2 and t2 minus t1 is less than one mouth so we are trying to predict if there is any releasing of the Sun for each user we have some statistic here but the important thing to note here is that the contract problem was was extreme here we have almost 26 percent of new songs in


the testing data so we have to deal with this false start problem those are the features that were provided and the main the most important feature where the user ID the song ID the suicide and the artist day so does work those are the features I will focus on so what we did so we used here the light GBM water so we computed different statistical features based on the user the song and


the artist and we noticed that using statistic using the target was not working and you will see next okay so the data set didn't include any time stamp or session information but we noticed by doing some analysis that it was coronary ordered so what we need is we approximate sessions of the users by using some little trick just look at the indexes shifted by one try to check the


difference if the values are small this means that those samples are in the same session if the value is high this means that this happening in some other sessions this is the first trick that we use second one we notice that using the user and the stock ID did helped in our cross validation but the model was really overfitting orders and as I said before when you have features and your


model is overfitting on it either you remove them or you apply some transformation because they will happen we apply some transformation and we come up with different SVD embeddings for those filters we also noticed this is the main target was evolving and as you can see there with there is a nice trend here and we tried to capture that trend so we made two linear regression


models we trained them on the the sessions and we extract those features from the linear models and we added those as filters in our light area also in order this is my last slide in order to to somehow overcome the cold side problem of the songs what we did is we created a judge Jason's in matrix of the last songs that each user has at recent two and so we took the last 30 songs of


each user and we put a one in our matrix with those with two suns are in those last songs otherwise mr. zero and then we applied the nest video dot and use the evasion it did help in to overcome the called such problem because you cancel some new sound that have never been seen before it will be related to other songs that have been different by the users and the


invasion will capture this kind of information and we end up again like we did in the two sigma approach by using five models by flight GPI models and we just use different subsets of features for each model and then we play together all the moments thank you for your attention any questions I don't know if sorting so okay oh yeah I can Early's do you recommend any good like books or


learning materials what like learning materials or books do you recommend for anymore I'm asked to give some recommendation about good materials I think that's Tagalog clear knocks on table discussion are the best materials when I started doing data science I learned everything from the cargo platform I didn't started by reading papers or reading books I start directly doing


competitions and reading discussion and Carols later on to travel they don't understand then I start reading papers scientific papers to to get more insights about why things are working or not you have a question can you talk a little bit more about what is the use of fitting base models for example linear or logistic regression and using them as in input to


more sophisticated model so the question is in the last exam that I will show you why did I use the linear regression models and why I used them in my life GB model so because we had that trend of the meaning of the targets that was clearly decreasing in time I should capture that trend and decision trees based models cannot capture this kind of trends so the ideal is to use a linear


have you ever thought about joining a cago competition but don't really know where to start well you're in the right place I'm title data scientist to Rachel's hatman and today I'm gonna show you how to enter a competition alright to start off with we're gonna pick a competition and I already know I want to enter the housing prices advanced regression techniques so before I join I'm probably

How to Enter a Kaggle Competition (using Kernels) | Kaggle


gonna want to read through the rules but I actually read through them earlier and you can do that on your own so I've accepted the rules for this competition and I've joined the competition let's find a kernel to work off of so let's see kernels that people have recently run so looking through here oh look here's an example submission rachel is that you yeah


that's me I did I made this earlier so I'm going to click on this example submission here and this is gonna walk me through all of the steps I need in order to enter a cankle competition so rather than just reading through it here I'm actually gonna copy and edit this kernel so I have my own copy that I can work in it's gonna take just a second to spit up when you enter a competition


there's a couple of basic steps you need to do first you need to train your model I'm doing all of this in kygo kernels which means that our code is going to be run on Kaggle rather than on my local computer so this competition this kernel already has a lot of the information we need in it so we are reading in some helpful libraries this is using R you can also write a kernel in Python if you


prefer loading in my data and I'm setting a scene for reproducibility from here I want to actually train my model so it looks like this person it's me has already started training a very simple model we're doing a test train split for cross-validation so I can check that my models not overfitting and then there's a little bit of pre-processing we're taking the training data we're removing


the column that we are trying to predict to sale price and then he are we are converting strings to factors and then we are doing some label encoding so this is just taking categorical factors like I don't know zoning and turning that in to a number so that we can create a matrix and from there we can use our matrix as input to an XG boost model so let's train our


model really quick and if we look at the mean average error which is what this competition is going to be graded on I can see that it looks like it's actually still going down as we continue to do lots of boosting rounds so this is an X G boost model and the number of rounds you train it for will affect how much complexity in the data you can model and let's make sure that we're not


overfitting by running the same model on some of our held-out data so doing some cross validation and it looks like the test error is actually smaller than the Train error so what I'm gonna do is just increase the number of training rounds and see if that's still the case all right yeah and it looks like it is so I think just by increasing the number of training rounds I should get a better


performance than this kernel originally had so once we have trained our model we can actually make our predictions so our predictions here are going to be based on this test CSV which if we look at our data set we can see that the Train CSV has 81 columns and the test CSV has 80 columns that's because the test data set has all of the features that you would need but it does not have the feature


that you're trying to predict so I don't know what the answers are over here that's what I'm trying to guess I will take these this test data that doesn't have the target column I'm gonna do the exact same pre-processing as I did before and then I'm gonna make some predictions using the model that I've trained up here to several more rounds than original submission and then I am going


to make predictions using this test data matrix and I'm going to save it as a CSV so what I'm saying here is I want a data frame with two columns one column is called ID and this has the ID information from the test data and one column is called sale price and this has my predictions from the submission prediction and I know that those are the columns that I need because if I look at


the sample submission here I have the columns ID and sale price and if I don't have the right column names here when I go to submit my file I will get an error all right once I have made my predictions I'm going to write those predictions to a CSV file so I'm saving a file out that has all of my submission information in it from here I need to commit my notebook


so committing runs all of your code top to bottom and it creates a stable version that you can refer back to later so if I'm making multiple submissions I can go back and I can look at the committed version I used for that specific submission and it looks like it's complete so let's open the version and here if I scroll down you can see at the bottom we have output files so I


have one output file submission CSV that has the predictions that I saved out in my kernel and I'm going to submit this file to the competition ah and it's submitted and the submission is complete and it looks like my score is 0.16 and if I jump to my position on the leaderboard I might have to wait a second for that to be that's to be done but you can see that I have my score I've made my


submission and from here all I need to do is keep trying new things and improving my model and then making new submissions as I think the things that I did will improve my overall model so that's it that's all you need to do to enter a kegel competition obviously I wrote some of the code off-screen we're all so of course welcome to use the kernels that people have made public for a


okay it's absolutely great to be here I am the official Kaggle representative kaggle comfy representative for this presentation or for this event we have a major Kaggle team meeting going on next week so very few people were able to attend and so I'm I'm happy to be here the event organizers actually contacted the cago leadership and they asked they asked could we get the the smartest

Keeping Your Skills Fresh When Everything is Changing | by Walter Reade | Kaggle Days Dubai | Kaggle


Tagil employee to come give this presentation but unfortunately that person said no so so then the leadership said if they organizers ask the leadership well maybe we can get the wisest Kaggle employee to come speak and unfortunately that person said no also so the organizers they got a little bit nervous and they thought oh maybe we can get the best-looking kaggle employee to


come speak at this and at that point I was very uncomfortable turning down the organizers three times in a row so here I am actually the truth is none of that is true I work with an amazing team one of the things that kaggle values as a company is low ego and I think that is really easy to do in this field because no matter how good we are there's always these super grandmasters who are so


smart and they're so clever and they do so well and it's very easy for us to you know keep our egos in check so I'm going to talk about a topic that I think many of us probably struggle with and that is how do you keep up in a field that is constantly changing all the time right we know we need to keep our skills fresh but how do we do it and before I start I


don't believe there's a right answer to this so I'm gonna share what I believe is right and the kind of the strategy and methodology that I use and hopefully you'll be able to find a few things that are right for you so during the recent Kaggle de San Francisco event we had Francois Holly so he is the creator of the Charis deep learning framework and he spoke and he


told an interesting story he joined Kaggle and entered his first competition and was competing and he wasn't doing very well so he thought well maybe kegels not for me he forgot about Kegel just didn't do anything for a few weeks and then all of a sudden he got an email from Kaggle saying congratulations you've won most of us do not win the first Kaggle competition


that we enter in fact most of us don't win any cattle competitions so it's very abnormal what are you shared most of us I think when we joined Kegel right some of us are very very skilled they come maybe with an advanced degree in data science or machine learning but a lot of us like myself come knowing almost nothing so so I'm gonna talk about some of the principles that will will help us


and again they're very applicable to me I really hope they transfer to many of you so I'm gonna start to talk about my education and my career and not because either one of those are particularly important but I really want you to understand where I was as an individual when I joined Kaggle in my first cattle competition so so that's me a long time ago I got my first computer in 1983 it


was a Timex Sinclair 1000 you've never heard of it because they went out of business in in like a year it was the company Timex that makes the watches and they wanted to get into computers it was an absolute disaster for them and that computer that I got into that by the way that's not a Timex Sinclair that's an Apple - that was my neighbor's that I used to go visit and and use their


computer but my first computer had 2 kilobytes of RAM 2 kilobytes of RAM just to put things in perspective and so I taught myself basic I really enjoyed it right I wasn't necessarily very good but I really enjoyed it my high school taught a class in Pascal programming probably maybe not super familiar to a lot of you I enjoyed the class so then I thought well what do I want to do for a


career and it turned out that everyone I knew was going into computers and I was worried by the time I graduated from college there'd be a flood of talent and all of the jobs would be taken right of course that was foolish because the right the industry took off faster than the demand pool and still to this day there are people in computer science and there's demand for the talent so I


thought well what should I study I chose Chemical Engineering and I chose it not because I knew anything about Chemical Engineering I chose it because it sounded kind of cool Chemical Engineering well at least it sounded cool in 1987 so please don't judge me for that so when I was an undergrad I took a class in Fortran 77 I'm sure most of us have heard a Fortran and again I really


liked it I went on to get a masters for part of my master's I used Fortran to write a program to model the combustion of a coal particle as it burns and I really enjoyed it so I went on to get a PhD in chemical engineering and this time a hundred percent of my work was computational so I did direct numerical simulations of turbulent three three dimensional turbulent flow and the


particle movement within that flow now interestingly enough I don't know if many of you know this but the Fortran originally you had to start in the seventh column right so space space space space space space and then you would start your statement and things like line numbers and whatnot could go in the first six but that was a throwback to the old punch card days and


with this was in the mid-90s they came out with Fortran 90 and Fortran 90 no longer needed you to have to worry about indentation which was really great right you could just write your program how you wanted and I thought it was fantastic how it unshackled me from having to worry about indentation now I also find it very ironic that in today's stage guido van rossum who invented


python has reshackle me to a programming language that cares about indentation so one other point about this again just to give you an idea of how fast things change so now we're a decade later we're in the mid 90s and I had an IBM workstation and I had received a $10,000 grant for equipment for my research and I spent that entire $10,000 to get a memory module for this workstation that


was 256 megabytes of RAM which was you know a very large Ram back then but $10,000 for that so things change so I got my first job out of school and so everything up to them was great loving life loving programming loving science loving Chemical Engineering and then I got a job in an industry that uses chemical engineering it was the paper industry and my job was to optimize air


and drying systems for drying cardboard right we used cardboard you know they make it wet it's got to be dried and it was I can't say I really enjoyed it you'd have to go and so you know but you can see this is a person you'd have to go and stand up here and you'd have this long temperature measurement that you'd stick in there and measure the temperature of these dryer cans then you


do the same thing with a humidity and you would do it on every single one of those on the back in the front it was hot it was sweaty it was not what I went to college for nine years to do so I decided to upgrade my career and then I went into consumer products making diapers are nappies as they call in some part of the world and slightly more glamorous I suppose but


still not necessarily giving me the career fulfillment that I had always dreamed of and by the way the technology that goes into making diapers is is actually pretty amazing it's just not glamorous work so now it's the early 2000s and I am absolutely bored out of my mind with my work it's just not exciting I I don't enjoy going to work and my company offered a distance


education catalog and I was looking through there and they had a course on computational intelligence and that was three things it was genetic algorithms that was considered computational intelligence back then fuzzy logic which was super hyped up at about that time that I don't know does anyone use fuzzy logic anymore probably not and neural networks so I took this


course now this was pre Coursera days they actually physically mailed you DVDs and you would watch the lectures you know on your television you would do your homework you would email the instructor two weeks later he'd email you back your grades so that was the distance education that was back then I really found it interesting this idea so they taught a very early version of


convolutional neural networks they didn't work well they were very narrow you know they're very shallow and we actually didn't program the so we're still on windows so the instructor would he emailed us a precompiled binary we would run it on our Windows machine which was probably a single-core CPU or back then but even if it had more than one cards we're only using a single core


but we would try to train this thing and and but the the problem was these little pop-can images I should have put a picture and I was blown away that a computer could recognize was it a diet coke was it a dr. pepper was it a sprite I was really fascinated by that the problem was is once I tried to apply this technology to anything in the real world like so in in in consumer product


manufacturing they're taking pictures of every that's being made to do some simple measurements and what and I thought well maybe we can use Ural networks for this could never get it to work so I said well neural networks interesting but not useful so so then fast forward another ten years I'm still bored out of my mind at work right but hey families got to eat and I stumble across this challenge


it was the Merck molecular activity challenge on a platform called kaggle it was fairly simple all you had to do is predict the molecular activity of these you know these molecules based on tens of thousands of molecular descriptors and I've got a PhD in chemical engineering I'm really good at data analysis top prize was $25,000 so I thought to myself this is gonna be the


easiest $25,000 I've ever made in my life right I was literally picking out the car I was gonna buy so the person who won that competition does anyone know who that is it's Jeff Finn so his research lab won the competition using deep neural networks and as Maria said I came in though there were two hundred and thirty six teams I came in actually 23rd but it was from the bottom I came a twenty


third from the bottom and it wasn't because I wasn't trying I was literally trying my hardest and it was it was actually a wait but I mean it was actually it was very demoralizing but it was also a wake-up call because I realized there was all of these things that I had no idea that I didn't know so it was just was like window opened into a whole new set of tools and


technologies that I could learn fortunately another thing happened in 2012 so you probably know Andrew Eng is speaking here in 2012 Andrew hang it was a co-founder of Coursera and I found his course on machine learning and took that course and again I was blown away that here we have a course that would literally have costed thousands of dollars for me to take it's online it's


for free so things were good for a while things were good because you could take that course there of course you could read the few papers that came out Jeff Hanna came out with a few papers but suddenly there is a problem this is just a screenshot of Coursera today and there are more data science and machine learning classes that any person here can take even if it's full-time if you


look at the number of machine learning archived papers that are published every year you can't keep up so on the one hand we live in an amazing time where you can learn the things you need to - like in my case change my career or become good at Kegel on the other hand it's it's too much and you can't keep up and it's information overload so my question is how do you know get given


them given the fact that you must everyone here must keep up their data science skills and the machine learning skills we have to keep up on the one hand on the other hand there's too much to learn so what do you do so the question is how do we keep up this is probably the most important point to understand is you don't you can't you can't keep up with everything new and


machine learning and data science there are people who are deep learning experts who can't even keep up in the deep learning field and once you recognize that it kind of then reframes your mind to say well if I'm not going to learn everything what can I learn and how do I learn it so I'm gonna give three guidelines okay on how you can continue learning - like if you're brand new how


you start your learning journey if you're experienced how you continue and the three guidelines are be sensible be smart and be systematic now I was thinking about branding these the three BS s of learning data science but I'm not sure if that's the best branding you can give me feedback if that's catchy enough so be sensible okay all right I crashed and burned on my first Kago competition


and like I said there was an enormous amount of things I had to learn number one my computer was terrible was four years old had eight gigabytes of RAM and I realized if I want to compete so I started upgrading my computer more Ram Moore's better CPU more disk space more even more RAM better seat a new motherboard right so I started that race that many of us do okay that got me kind


of far I was only using MATLAB in some proprietary software me neither of which you can actually use to win a kaggle competition and at least at the time MATLAB wasn't the best tool so I realized I've got to learn Python so that was another big major learning step the second competition I entered I actually did okay I came in 34th out of 950 34 from the top this time and I use


logistic regression not because it was the right tool it was literally the only algorithm I knew in machine learning so I used it right there may have been veterans uh let's see I know who won like there's some people here I know one that I can't remember I know there's some grandmasters here that actually won that competition so then I learned you know you start seeing XG boost XG boost


is winning everything so then you learn next G pose now uh deep learning I was sure that deep learning was a hype just like fuzzy logic and I avoided learning deep learning over time you know a year later I realized okay this is going to take over the world so I started learning it well that required hardware GPU it required you know I was using Windows so I better go to one to Linux


every time I made this shift I felt dumb again I felt stupid I felt like I don't know what I'm doing because it was a huge learning curve so when I say be sensible I want you to recognize that if you start feeling good about yourself and how good you are it's time to jump into something else and to kind of stretch yourself and feel a little bit intimidating again


and that's perfectly normal and that's gonna continue to happen III would say anyone who wants to be successful in this area has to get used to feeling insecure about what they know and what they don't know happens to me all the time not I'm I don't like when I say something and it's a mistake or I argue a position and it's wrong on the one hand on the other hand I'm actually glad


because right it improves me and I get better so be sensible set your expectations to something that's reasonable so the next thing is be smart and what this means is you have to build a learning plan for you if you go to github all these people say hey a step-by-step learning plan that might be okay it's probably not for you right and the reason is we're all in different


places if you go through and somebody's learning plan that is a waste of time to advance not advanced enough you will not do well so be smart so here's what I recommend when you set a learning plan excuse me so say a learning plan say you have five hours a week to learn you all right you say all right every week I'm gonna learn and spend five hours this is how I would recommend doing it


number one and I was talking to Pavel earlier this is he's very big on this strengthen your fundamentals so that could be the fundamentals of linear algebra the fundamentals of statistics those things don't change for me most of the time I spend in there is learning Python deeper right learning the language I my favorite book to read and reread is called learning Python it's


this thick it takes me forever to get through every time they get going new edition I start over again to refresh those fundamentals the other piece of that is I spend a lot of time learning and relearning pandas every time there's a new addition I learn what's new and I also scan the API for things that oh maybe I didn't realize there was something cool there so strengthen the


fundamentals of whatever you need to learn the second big chunk is your core work and your core tools so this is whatever you're doing for your project your problem you know the work you do at work for me is a kaggle data scientist a huge amount of that is domain knowledge so somebody comes to us and says we want to host the competition on earthquake signals I've got to spend time learning


about that so I can understand I can ask the right questions when Kaggle was acquired by Google all of a sudden we now and we using our local machines that now we're using GCP VMs right so learning kind of that system stuff I hate to bring this up but we constantly also try to learn the new ways that leakage can enter the data because there's so many thousands and thousands


of ways insidious ways for leakage to go and and trying to build our systems around how to how to minimize that that will be different from every one of you and then the last is spending time on the cutting edge now this is where most people make the biggest mistake when they try to keep up with the industry they spend way too much time because there's all this cool stuff there's a


capsule networks for neural networks or you know the ordinary ordinary differential equation neural network stuff that was from the nerves conference so some really cool things and I am NOT dissuading anyone from keeping up on those but if you spend the majority of your time learning those and and and trying to keep up the speed you will actually be sub optimizing what's


critical for you there are many tools that are going to come out that you'll be able to use right out of the box like Bert the NLP tool that Google pre train model that Google released right that's something that is cutting-edge and you should be able to pull it right back into your you know your tool set so that's approximate you know I would say you don't necessarily need to


balance that on a week-to-week basis but over the long term try to to balance that one other point when it comes to learning new things I strongly believe that if you're not writing code and trying it out you're not actually learning it so I fall into this trap I watch a lot of YouTube videos Leyland icon comes out I like to watch as many of the Python videos as possible and


unfortunately most of the time I'm not not actually implementing anything I watch it I go yeah that's a great oh that's a great a great idea and then I promptly forget it so if you see something that's important make it a project and get it into your your repertoire so the last vs is be systematic and this is the secret to sustainability this is how you can do


this in the long term and not two years from now 200 and I haven't kept up so what do I mean by be systematic I am a big fan of simple checklists and some of you may have read a many years ago I wrote a post mislay it was my most popular forum post I fangled or said here's my standard work that I do for every competition and it literally is I update my Khanda environment I you know


whatever so I do any environment updates I create a new repo I download the data I you know so it's just a very simple checklist of things I do so I use multiple machines so I make sure everything syncs out all those machines it wasn't anything that was complicated but the reason I use checklists is because it reduces cognitive friction when you're like oh especially with


Kegel we're like oh it's really late I really want to start this competition you don't want to have to think about it so having a checklist allows you to go boom boom boom boom boom and do it without having to think pilots use checklists pilots are some of the most skilled practitioners about what they do and the amount of training they have to go through they use checklists because


it reduces cognitive friction if they have a headache if they're tired and make sure they do the appropriate things so so how how do you use a checklist or a system when you're learning for me I have a list of things that I do every day to keep my learning up to date the first thing is super simple I have a bookmark bar for five different archives so archives that write the paper


repository archiving topics fluid dynamics data and statistics vision machine vision and machine learning and earth sciences I open those up and I every morning literally before I do anything before I check email I check what those are so that's one of the things I do it's a habit it's sustainable the other thing I do is write any time that any of the PI data


the PyCon videos come out I make a list of all the ones I want to watch and I don't get to more than ten percent of them but I make the list and then I worked through them systematically till the next ones come out all right so if you want so as I mentioned there are so many courses it's actually intimidating where do you start what's the right course I have become a big fan of CagA


learn not because it's put out by kaggle because I really actually agree with the philosophy what kaggle so if you think of a move if the time to make the presentation to record the presentation to transcribe the presentation to host the presentation it's a very very very very long time scale what learned has done is say we're gonna take some very key important topics and we are going to


let you do it in four hours and we're gonna focus on those key important things and get you pushing the buttons and learning these things what's nice about this there's no video actually one of the courses had videos but they're gonna get rid of them is they can freshen them isn't like as much as once a week somebody will ask you a question and they'll realize oh we can make this


better so they'll refresh the content so so for example if you if you look at this so it's kind of small but here's the current cago learn everything from python machine learning data visualization sequel micro challenges and the new one machine learning explained ability maybe you know those things but maybe you want a refresher it's really really fun and easy to go


through so I'd recommend checking out one so here's like this one on embeddings it's it's I went through it before I I had never got there I went Baba before you through it yesterday I thought oh yeah it makes a lot of sense so cago learned a great place if you ever feel lost go back to Cal go learn obviously there's tons of great stuff out there but if you're ever paralyzed


about where you should go go to CAG alert this is an example right so they walk through with words they give you some code you run it it kind of lets you know if you were wrong it gives you hints on very the common errors and it's fun it's interactive it's in kaggle kernels so you can access it anywhere and now that actually keeps track of your progress so you know if you stopped


one halfway through it'll tell you which modules you've done all right the other thing that this this kind of surprised me if you go to youtube.com forward slash Kaggle there is an amazing amount of content and it's added to weekly everything from coffee chats to interviews with for example practitioners at Google brain all sorts of pretzeled live coding gets recorded


there so this is another great example of a place you can start there's probably too much content there for you to keep up on and that's okay right again give yourself the permission to skip things and focus along those lines there's a point I wanted to make every cago competition that I joined my primary goal was to learn something new so embeddings for example I'm not an NLP


expert so if I join a cat go I did a playground who's spooky author so it was like three authors given a sentence predict which one so I did that with the sole goal to learn NLP you may under one to learn okay finally I'm gonna learn convolutional neural networks or ell STM's or some sort of you know whatever you need to go so if you do that no matter where you place


you will have one you will have had time well-spent so I want to close up a little bit with this quote it is by far my favorite quote it's a little bit intense so I want to explain it says the illiterate of the 21st century will not be those who cannot read and write but those who cannot learn unlearn and relearn a hundred years ago so many people didn't know how to read or write


and that prevented them from being in the workplace in today's world not just in data science and machine learning but almost any vocation you have to learn you have to be able to learn to unlearn and to relearn so that to me is the most important reasons for having yourself a smart plan that's sensible that's smart and that's sustainable so that you can learn


whatever skills and do that five 10 20 30 years from now to keep up and have whatever professional success that you want and with that thank you and I'll open it up to questions one second Ronnie Heflin let's see who has who has the mic area what I see a second period thank you oh that's a great question I I'm sure there are but I can't recommend any and the reason being is I just that's not


something I looked at does if anyone here know of good data science for kids I you know I would say I don't know for sure but Khan Academy kha in the anatomy is one that has a lot of great content but I don't notice they have data science No okay so sorry great question I'm gonna write that down and we'll follow up yes sir [Music] [Music] yeah it's a really question and and I


actually don't believe it's the type of data I believe it's the volume of data and what I mean by that is if your company wants to do a very large data project and you have data in multiple legacy databases that sit in multiple organizations that can be extremely complex and hard to get my recommendation is always to start with the smallest viable scope so try to get


a little bit of all the data if you can or just get the data that seems the most reasonable and start doing model building on that the biggest mistake companies make and I've seen this many times is they say well we're gonna have a two-year project to get all of the data collected and then they get all the data collected and they don't have the signal that they need or only five


percent of that data was important so I would say just like it was mentioned by grandmasters when your model building you go quick you go small you go fast and then go from there I'd say the same same with data so then then you may say there's there's some very expensive data here that it's worth the time cleaning up it's worth the expense but then you'll find a date over here that's not


worth the time okay the biggest mistake is companies say well let's clean up all of our data and that's just waste of time thank you question over here trick question take you back to your fluent that after navier-stokes yes and the reason is is because the navier-stokes is a you know a three dimensional coupled nonlinear equation what the neural networks theoretically can approximate any


function so neural networks can approximate the function so I just read a paper last week it was sent to me by a professor in Germany and so there's a there's a type of flow where you have a temperature differential and then you get these circulation patterns and if you look at if you ever seen a close-up of the surface of the Sun you see these like pockets that's what causes that and


what they did is they did very very high-resolution simulations of this and then they trained neural networks to identify so segments the the boundaries of this now they get the same results of these expensive simulations but it's orders of magnitude faster and there's a paper that came out a couple of years ago where I think Google was a co-author where like a smoke plume simulate that


but then use neural networks to approximate that so now you can model those dynamics so you you you will see more and more of that but the answer is absolutely yes it is exciting work I'll say another thing you're seeing more of this in chemical kinetics so chemical kinetics when you have multiple chemicals it's great it's it's stiff not nonlinear differential equations because


you have very fast reactions very slow and people are now using neural networks if you have the system of chemicals we model probably react will see more than yes I am NOT a deep learning expert but I crush because our neighbors are noisy and we do do-dah like 1 million picture is just fine but do you never you are dead picture so great great question so I'll answer that generally


first so in general you want to pick the tool that's appropriate there's no reason to do a very very fancy 3d instead if you have a small amount of data and you can do logistic regression then it works fine so I'm a big fan of doing the simplest tool now with neural networks we have the advantage of transfer learning where Google has spent a month training a bank of GPUs on these


large networks and so you can use those and actually I'm probably not a hundred but maybe a thousand images because you have all of this pre train models so you know something like that I always say try it right if you have a thousand images kind of fine-tune a pre training model on on 500 see how well it does on the other 500 it's going to be very context dependent I think you know


there's some image tasks that are extremely complicated that wouldn't work but there's some simple ones yeah that's a great question and you know to me to be very frank I am very bad at predicting what's next I've always seen what's happening and then act upon that you know there's a lot of people that talk about these things and you know I don't pay a lot of


attention to them to be fair i I just watch for tools that I can use in my work I will say this you know data science machine learning AI it's it's not going away in ten years it's not going to wait 20 years I think it's a fantastic field at the end so again is the people that are willing to keep up and whatever happens to be important and learn that I think are gonna do very


well for themselves but again I just so I would be out if I knew the answer to that I'd be on the main stage I think we have maybe time for one more question oh yes yes then little stars our afternoon break so we have like three after four minutes do you have any other questions [Music] yeah yeah that so if I understand your question is how do we help those who


maybe are discouraged along the path yeah that's that's a hard question of you know so I I came to cattle with a PhD in chemical engineering in a good career and I remember the sense of being very intimidated to ask anything on the forums right I felt I wasn't worthy I felt it was a stupid question that everyone knows it but me I think the more that we tell those stories the


people that that's natural that's okay everyone goes through that I think the better right that's why I'm happy to tell my story how poorly I did I tell my story how frightening it was to ask the question on untangle but we can also do a good job I think so tango we try to monitor the forums and I can't say that I do this all the time but if I see somebody that you know they're due and


they're trying to say we I'm new to this you know I much as possible try to point them to hate just keep at it don't get discouraged so you know every one of you here can do that you don't have to feel like you're a grandmaster to give people encouragement so I really appreciated that question and regardless of where your level is I think you have the opportunity to do that and again that's


what makes Cagle a wonderful community I love the cattle forums there's so much useful content people can enjoy each other a month ago is actually when I met my first Aguilar I had never met somebody from Kaggle but I feel like I know them just by the online interaction so it's a wonderful place and we just have to convince people that they can do that at home and themselves thank you


for them hey do you want to ask any more questions okay Pavel I can see your hand what is thank you but it will be the last question that's what your question my favorite game so probably he's maybe not as well known but a parent from Germany Mathias Mueller I liked him because I got ended up with him he was always very helpful he was always very friendly he explained


things that we easier though easy way for me to understand but but you know it to be fair I there's so many carers the grandmasters that I appreciate for all different reasons like like Abhishek I mean he's just he's just such an interesting character a funny guy as there has a lot to teach people yeah and I can go through the list but I'd say fair and fair and Mattias Mueller from


good morning everyone the first talk of today is by Jason Lee Corporation University of audacious of departing star customer meetings and over characterization and transition yeah so of course obviously deep learning super successful at blah blah blah it solves a lot of stuff it's successful in a lot of things some people I've even gone as far as claiming it's the new electricity of

On the Foundations of Deep Learning: SGD, Overparametrization, and Generalization


course when we have electricity we should look there's a field call electrical engineering the studies it I guess it's kind of a purpose of us being here this summer so deep learning is not quite a science yet it's fairly crazy that things regardless of whether these things where people will try crazy things like these cyclic learning rates learning rate to nice games are pretty


crazy they're fragile even things like changing seeds and it's sort of like we're trying to build a working system by trial and error instead of understanding something like civil engineering and then sort of you know been our deep learner has built this thing but you didn't take a structural engineering class so that's what happened so this talk is trying to about


unkind understand deep learning so motivate of course by this well-known paper I added an optimization at the end I think they forgot that they left out something pretty important there okay so understanding deep learning and we so there's obviously two aspects our goal is to minimize test error we want to do say multi-class classification as usual you can think of as two components an


optimization error and the generalization error something controlled by the algorithm and something simply statistical so there's of course this problem is challenging in both aspects the optimizations non-convex non-smooth the global landscape right nothing is very good about it lots of local minima and just in general it's difficult the reason about the only thing we can see are


really gradients at Hessians and so forth local information and you want to draw global conclusions on the statistical side similarly bad there's more parameters than samples that generalization error literally depends on everything your algorithm you're learning race schedule your architecture initialization scheme algorithm parameters such as momentum so forth okay so how to deal with


optimization luckily this is probably where it's most well understood is simply people have realized that you can always change the model that if one model is difficult to fit we don't have ever believed any models correct or anything like that you just fit a bigger model and bigger models are easier for SPD to fit which is somewhat of an empirical fact I know so you can have


this model with many bad local minima do something and then SGD works so over privatization okay here's an experiment showing this this experiment is essentially replicated from this paper of living Isha Mirror and she'll have shorts so on the left hand side the data comes from a teacher network that's fairly small compact as 50 neurons on the learner side the network has doubled


the number new ons there's a hundred neurons but still the data comes from a network with only 50 neurons so if you try to run SGD on the well-specified architecture which is very sensible assuming you saw through a statistics class if the model comes from the family you want to be well specified Emily's asymptotically efficient da da da whatever by the end of the day actually


fails so statistical efficiency doesn't matter right if you can't get training our low why do you care about statistical efficiency because you cannot find the mo e you run SGD five times it gets five different answers none of which are the global minima I could have done this five thousand times I don't think you would have found the global minimum okay now in stark


contrast simply the same data comes from the same source and I simply change slightly change the architecture the only thing is doubling the number of neurons and I run STD again five times and every single time it's lost value zero and this is also tester this is STD on fresh cause of data this is not training here so SGD is finding the global minima of the population loss


empirically so seemingly over privatization does in fact make optimization easier and in this case even made because it's tests are you could have even extracted some statistical statement here okay so recently in many a series of papers by myself yen's it also briefly talked about this yesterday Trenton has also worked on this I'm summarizing all of these results because they're all kind of


similar enough I don't want to spend time distinguishing they basically say if you have very wide networks and you initialize randomly with appropriate variances that are not crazy gradient descent converges to an epsilon global minimizer as long as the loss is convex your learning rate is small roughly speaking or works for any model so when you're sufficiently over privatized you


get an epsilon approximate global minimizer of the training loss let me sketch a proof of why this could be true of course when you're over parametrized looking at the parameters is not a great idea they're not identifiable there's permutations and all sorts of other invariances that come out that the parameters don't really respect it's a better idea to look at how the


prediction changes so if you compute the change in the loss you'll find that it's related to the outer product of a certain Jacobian matrix or though or the gram matrix as long as this gram matrix is strictly positive definite you get a contraction so then if I could ensure that this thing stayed positive definite then you convert to a global minimizer so why does this the first thing is to


establish that it started positive definite that's pretty clear random matrices are positive definite so this is some sort of just a concentration and perturbation analysis to show that initialization the gram matrix state is positive-definite and then simply when you're very over Prime Christ M is the width of the networks then it stays positive definite there's


always hiding a bunch of stuff it's whatever it's something and then you need am big enough to make this smaller than lambda zero okay so over parameterization essentially what it does is if it's forcing in this initialization scheme it's forcing the grab matrix to stay very stable close to this initialization and thus stay well conditioned and finally you get the


great from this you can extract statements of this type foursquare loss you can say exactly converges to global minima at a log 1 over epsilon rate if this will say logistic loss you could say that it converges an epsilon global minimizer where Absalon is determined by the width if you want in smaller you would need wider because logistic loss is not global we strongly come back to


so many locally well regardless for wide enough networks as long as your loss is any convex function your you can get a statement like this ok so after we so here's a simpler way to interpret these statements if you don't want to read 15 pages of calculating algebra one way to interpret these statements is a statement about the local geometry of the loss function you take your random


initialization draw a ball around it of some norms see for some appropriate seat that depends on the width you set every critical point in B is a global minimizer and there's at least one global minimizer here and Gd store stays in this set so it's a statement about local geometry is saying that locally things essentially look convex as long as you find a critical point in this


local set it will be a global minimizer and in fact you do stay in this set so roughly the global geometry of deep networks is it's probably pretty bad I think there's definitely it should have exponentially many bad local minima local minima with even high values of crane loss however locally there is a global minima locally there exists global minima and you can find a region


in which there are global minima but no local minima no local minima of higher loss yeah one way is you take in under parametrize network and start introducing neurons and then set them to be like to sort of cancel each other configure the ways you add them in a way that cannot help but it's still a local minima I can make at least I can make it a higher order saddle up certain but I'm


not saying you'll find them yeah I'm saying there are ones and certainly not near random initializing oh I think I can always do this I take your architecture I just make a tiny one if you believe there's one in a very small architecture I start augmenting neurons in some useless way so it will look like a standard architecture but maybe the parameters will be in a weird


configuration at least I can make a satellite high order satellite okay so global geometry probably not very good local geometry very good there is a global minima and gradient descent converges to it so we're kind of happy and then oh you think about it a little bit more and then some paper start coming out and then you realize of course locally it's very good locally


you can write down any function so f theta is my neural network function it's essentially equal to some F 0 it's not important think of f 0 as 0 or very close to 0 some order one quantity a gradient term and a second order term so of course since I'm local the second order term should be thought of as smaller than the delayed the linear term so we kind of just throw it away and


that's what these papers are doing and then so what's happening here is that if you're able to change the predictions f of X by an order one quantity while the Jacobian the gradient stay constant in a relative sense then the second order term is essentially vanishing the second order term is how fast this changes so if I can move f of X by order one which is all I need to move it because I need


to move it to match Y well this stays constant then I'm done I've found that I've constructed a global minimizer that is very close to my initialization close in the sense that there's no neg there's no potential negative curvature and that's essentially what that proof was saying it was saying this H of the H matrix the gram matrix is simply outer products of these gradients and it was


not moving a lot under these initialization schemes on widths and so forth ok so she's out and Bach came up with a nice sufficient condition of when you should expect this they call it this kernel regime to happen essentially the sufficient condition is that think of Y minus F 0 as order one we initialize in network so it outputs an order one quantity that's random so


this is like 1 because Y is order 1 and I basically pointed out that if you're Hessian divided by the square norm of the gradient is smaller than one then the second order term doesn't matter and your gradient dynamics track very closely for some constant amount of time the dynamics of a of gradient descent on a kernel machine this is very intuitive because it's essentially saying exactly


that how much the Hessian changes is very small to the gradient and the gradient when the gradient is big then you only need to move the parameters a little bit and you move your predictions by a lot so roughly in all of the ways we initialize the Hessian is something like going to zero with the width and the gradient is of one and so we have this sort of linear behavior in some


region around the initialisation okay so any questions here before I move on to you saying the final state is an interpretive approximation an initial and that this linear approximation is good is this sort of an idealized model motivated by a team that works and how you think about this analysis appropriate does this change so it was all like this so I would this result I


would think as if this sufficient condition holds and you use a very small learning rate then the gradient on F theta F theta sub peak and you can construct a new function that is linear call it f bar theta T these are very close for a constant amount of time so a linear that gradient dynamics track those of a gradient dynamics on a linear model for some amount of time right but


that's like a statement of the result yes the motivating slide you had a prompt was an appearance a set of empirical selfish genre that you know sort of empirical results instead in our models you know with these sort of things no this is my state oh yeah there was the ICL I think John was the little thing for two years ago I know that was that was on a state of the art right


this paper was about model this was about generalization I'm only talking about optimization right now this paper is about generalization this statement has nothing to do it these I mean you can make extract a statement about statistical error here but it's not a strong state the this is really an optimization thing I would say it's saying that I mean that's what I'm


asking well in your mind what does this say about the state of the art motivation like that this would you be able to give me you continue could ask whether the initial configuration the final configuration of pulse like this they're not close really not close like this so I mean that in your mind and with God is does this I think it tells you initial that you're you can get


training over small very quickly initially in the first few steps so it seems that one could come up with empirical protocols and somehow see how relevant this is to what's going on in practical also I've said a word that they're not quotes does that mean that we run experiments and you know this does not really capture what's going on in in standard models yes there's


there's some careful papers not buying me circle papers that say that this is not yes and is there any evidence that indicates otherwise or is it depends on like your parameter settings if you set the learning race small it agrees well if you set the learning rate big then no it disagrees if you set a small aircrews yeah but there are certain values that we usually


use in practice so and the ones like a good test error disagree Cilicia pretty simple question I hope but how should we think about Hessians for models that don't actually have Hessians you should think about it as okay one quantitative way to think about it is the change of activation pattern which is a second sigma prime is the activation pattern and sigma prime prime is how much those


change in some measure the Hessian is a border one over the has seen itself without the Hessian you don't care about the Hessians movement while you carry we are worried about is a gradient moving a lot because that means your feature scheme is moving a lot you want to ensure the feature scheme does not move so you need about the size of the movement of the feature


scheme which is the Hessian relative to the size of the gradient because the gradient multiplied by your movement is your change in F that's why it's a relative measure yeah so which Darby are measured that has some potential because still the size of the everything is just awesome Park is the only parameter moving is f so then the norm doesn't matter but the the size of the parameter


depends on that sorry you should be L to a spectrum but everything in that paper pretend so the only thing changing us out sure but I mean just even the site also yeah if you're careful you it's polynomial this paper doesn't talk about I'm not sure they talk okay so somewhat this at least tells us that at least locally we optimization is not a big deal so perhaps you can


believe that if you your learning rate initially is not very big you can get your optimization error kind of small if you try to use this kind of method to get an analysis of generalization error unsurprisingly since your feature scheme is not changing what you'll find is simply it has the same prediction exactly as that of a kernel method the kernel is called the neural tangent


kernel and your generalization bound will look exactly like rate regression will be something like like label Y transpose kernel inverse y / n whole thing under square root so that's some generalization bound you can extract from this style of analysis so I have no I I don't think it's very tight in fact I'll talk about later okay so let's look a little bit more at generalization


error what's happening a very standard found in generalization or some complexity over N for example in kernel methods it would be Y transpose K inverse Y that's sort of the RK just norm in Ridge regression it could be VC dimension VC dimensions roughly number of parameters times the depth in feed-forward networks the required list all kind of big they're bigger the sample size which is exactly


what the I Claire paper was pointing out it was pointing L essentially that the number of parameters is roughly 30 20 30 times the number of samples in many models and of course of course so this slide is slow and from naki or I think from telling you actually who stole it from naughty and I copied it from time so we've all seen this many times I'm sure by now so what they did in there my


clear paper is the number so you the black line is a training curve and if you took statistical learning you were to kind of expect the red line kind of goes well it doesn't it goes down so an over parameterize ation does not seem to hurt generalization in this case in fact in this one example it kind of helps a little you can improve your generalization even after interpolation and there's


even more evidence that number of parameters are hurting if you plot the imagenet top one error over time then you see that in fact for these very very big networks the top one error is quite small it means be creasing here I mean this is cherry pick from this paper but there's clearly networks with 600 million parameters and it has passed they are very low so throwing in


parameters in a non stupid way will not hurt your generalization I'm here okay so what's going on here let's turn to margin theory and see what it tells us so this margin theory is very classical it basically says that if you're very far from the decision boundary you should be very good why is that Pete Bartlett and Mendelssohn coalesced this almost 20 years ago now basically


your generalization cap is upper bounded by some complexity measure divided by the margin so if you can ensure some sort of Rademacher complexity that's sort of size free independent of the explicit independent of the width then you can get good generalization error so here's some papers that talk about how to opera BAM the numerator here I'm only going to talk about how to lower bound


the denominator okay so how do you get solutions with good market let's look at the simplest loss function with the simplest regularization scheme is you have logistic loss with a sum norm regularizer for example weight DK l2 norm you would kind of hope so this is the Globo max margin this is the best you can do if you search over all models in your parametric family and you thought


you kept the one with largest margin gamma star so you would hope you can get gamma star and in fact if you do a very good job of minimizing this regularized functional for small lambda then you do get gamma star so in other words if week l2 regularization leads to assuming optimization works week our two regularization will get you very high margin or the best possible March


okay this this proof is a simple unlike a lot of you into this proofs I think this is genuinely simple okay the proof is essentially you write down the logistic loss realize that logistic loss when this argument is very small can be approximated by Taylor's theorem and I could see an exponential exponential of a lot of terms added together you only need to keep the


smallest one because the rest are exponentially smaller okay so that this line is Taylor's theorem and then this one says if you add a bunch of like e to the minus ten plus e to the minus hundred you only need to care about either minus ten so only the smallest of these terms matter that's why there's a min here and then this from this you can kind of read off the results if you lambda very small


it's saying let me use as much norm as possible but among those I need a fit I need to make the worst case margin good so among solutions of the same norm we prefer the largest margin okay so how does so how does over parametrization improve the margin this is completely obvious if I have a network and this site if I have a sub network of this network the margin can only improve so


if you have a Rademacher bound that's a independent of the explicit number of parameters the margin is only improving this denominator can only get bigger okay so let's talk a little bit about how to optimize when you have a regularizer the first thing that realize is that none of these results using n key K and locally convex or whatever can ever handle this the regularizer induces


a sort of tie breaking among the global minimizer's and the global minimizer's are not equivalent like you can have two global minimizer's the unregular eyestrain loss and they will have drastically different regularization effects so the regularization induces a tiebreaking and now you really do need to find like a global minimizer of this regularize objective so how to do this


the answer is not fully satisfying but we can say something so take a very very very very wide network think of this as like wider than anything you've seen exponential in D or X Panetti whatever infinite okay then run gradient descent with a particular noise scheme then you convert your global minimizer in polynomial time polynomial in the dimension and in one over epsilon


something like Pauly D over epsilon four or something okay so gradient descent I'm very very very over parameterize networks converges to global minimizer's so over parameterization does help even when you have regularization the mechanism in which it helps is very different from this local convex intuition the intuition here is that when you're very very over primate


rising there's a descent direction in function space but if you have a finite a small width network you might miss the function space because you might miss the direction of this sign in the function space that's essentially what Frank wolf algorithm will do but computing as Frank will sup is a exponential time so but instead if you had a ton of neurons and you add noise


then there might be an exponentially small fraction of your neurons I see this Frank wolf direction but then great relu is homogeneous and then although it's exponentially small signal it goes up exponentially too and these things carefully balanced and at the end you do get a polynomial time result so because the number is very large are you what's relationship for the ndk regime


this no more I'm Kiki and you guys caught yeah you change anything NDK Scott it's very fragile and so I know I remember some of these results at that on laminate extremely small you set lambda and whatever way let's say you want gamma Magnus point one and you set lambda depending on that it's polynomial time to get within constant a polynomial iteration it's not pollen on with one


iteration is exponential time because you have exponentially many parameters some polynomial iterations or time in the pde sense and one other question so in the generation ban you're talking about a size free Rademacher bound right obtaining the margin with a very light regularization parameter president Ron a MicroBot might depend on say norms which is n have to be balanced


against this resulting mark yes you should think of whatever norms is this let's concretely take the goal of which one which is just a Frobenius norm of everything then that would be weight decay okay all right so so you might ask um okay so why add this regularizer I can already get the global min of the logistic loss without this regularization term what am I getting


here is my stat sample complexity much better and it turns out it is let's look at a very simple data set on the first two coordinates the data looks like this it's like two eggs or essentially then every other coordinate is just standard normal so there's two coordinates that have signal and that's the ones you want to pay attention to the rest of the coordinates are completely uncorrelated


with the label so they're noise coordinates are kind of for you and they exactly do full a kernel method a kernel method is unable to localize onto the first two dimensions so it has to look over all dimensions and it pays sample complexity at least d squared to get some air last if you want R less than some absolute constant you need to pay at least d square samples because a


cardinal method to solve this problem needs at least form pairwise degree to polynomials that has complexity d squared however for that sort of to XOR construction there's a neural net with four neurons and then so then the sample complexity of learning a neural net with four neurons is something like four D over N so D over and roughly so there's a clear sample complexity separation


with regularizer you can learn with D samples without regularizer uni at least d square samples so regularization explicit regularization or math you yes yes yes good you might put it it's not but we don't yeah it's not clear yeah yeah they might take exponential time we don't know no we know it takes exponential time how you scale things but I can scale this so it takes you


expert but then you can increase your learning rate anyway to analysis in the specific city yeah I guess I should be more precise to be more precise this one is better than the NDK one let's play it that way so then it covers the unregular eyes and the regular ice case okay so so do you need regularization it turns out you can do pretty well if you look at these numbers without regularization


so like the gain here is not from regularizing you gain 5% but if you just change the architecture you can gain around 10% so SGD without regularization does already have very good generalization perhaps it's not state-of-art but certainly the bulk is not from these regularization methods okay so let's turn to a simple example of logistic regression so even if you have


separable logistic regression the problems convex but not strongly or strictly convex so there's many many global minimizer's and you may be wondering which one does it converge to this is exactly what soldiery Hoffer and cerebro asked a year or two ago and they showed that this converges to the SVM solution in a very precise sense if you run gradient descent for a long long


long time then you normalize because the norm blows up you get exactly the auto SVM solution after doing some normalization this is quite amazing if you haven't seen it before there's all these directions why the heck should gradient descent get the 1 of minimum how to norm that maximizes the l2 margin right I mean this seems maybe I should depend a loading rate how


you initialize it could depend on anything and this result is quite stable any learning rate that's stable it converges to the whole to SVM solution ok so then so what is then you might be thinking what is special about gradient descent let me write down gradient descent in a very suggestive way I'm just write it down so it's trying to maximize the correlation you know yeah


because the logistic lost gradient is never zero so it's impossible to have a zero vanishing gradient even if you this is in contrast to least-squares I think which is where you're thinking yes okay what was I saying yes this norm okay the important thing is that you write gradient descent in this suggestive way so what it's saying is that gradient descent is trying to


maximally decrease the lost value of the function while paying infinitely small amount of l2 norm the key thing is is trying to use the least amount of Al to norm to achieve this goal so now you might ask why do I change a norm and this gives you a family of steepest descent algorithms and in fact you can prove that for this entire family of steepest descent algorithms you converge


the SVM given in that normal okay so for example that one norm this is exactly a form of boosting a to prove this a long long time ago in s PhD that in fact does maximize the l1 margin the same proof works for all norms okay so some examples coordinate descent which is steepest ascent with respect to l1 norm you're going to maximize l1 margin it's not quite adaboost it's adaboost with


some damping step size so add a boost with a dampened step size that maximizes our margin adaboost itself does not sign gradient method commonly used now the save on communication this gets you some L infinity bias and of course gradient descent is steepest descent with respect to l2 okay so that's kind of that's great we understand logistic regression very well that's sort of always the


starting point so now the question is what does it do on deep networks if you're if you're very very very optimistic you may hope that it solves max margin even if you do know regularization because then you will have some generalization bound that only depends on this up here and that seems good because now you can get the optimal margin of course notice that this is an


SVM problem but it's a non convex SVM problem now this SVM problem unlike the linear case has many many stationary points as first order stationary points so I can order station points on a global max so of course you cannot sort prove that it converges to the global Maximizer well you're able to prove is that it converges the first order optimal point on the following SVM this


is some nonlinear program it has first order KKT conditions and assuming the only assumption here is homogeneity then if you do gradient descent exponential loss logistic loss you get a first order optimal point the first order stationary point of this okay so you wanted to prove it gets max margin that's probably not possible because you're running a local search


algorithm but at least you can characterize that the whatever you converge to is very special in the sense that it's precisely a critical point of a fairly intuitive optimization problem okay so I've talked about I didn't really talk about the implicit regularization of n key K but if you think about what kernel methods is you can write down the sort of implicit bias of NPK or the


inductive bias of it as sort of you're trying to find the thing that best that maximizes the margin the worst-case margin but you stay infinitely close to your initialization so this top one theta hat K is exactly what would happen if you did n TK on logistic loss and terminated after some amount of time a very long time you had approximately get maximum margin maximum with respect to a


different sense in the sense that you have to stay infinitely close to this ball what the this previous result by us and Haiphong Liu and Jen we chose that actually you get a stationary point of this following program and of this program you're letting the prompter move infinitely far from its initialization so it's completely forgetting where it's initializing it's running forever and


ever which you'll always do when you have a logistic loss and when you run forever and ever you try to maximize the margin okay these are of course two endpoints of an extreme case on one of these you're trying to say infinitely close to your initialization the other you're moving infinitely far so they're very different with endpoints probably in my opinion


what's interesting is probably when this is not quite zero you're going slightly further than the linear regime but you're clearly not going to this super asymptotic regime you've deviated slightly from the N key K and what is happy there we don't really know but I think that's probably likely to correspond to closer to practice than the other end point because things like large learning rate


finite width will cause you to sort of stay infinitely close yeah okay so the final thing I want to talk about is how does architecture matter so I've told you kind of that gradient descent asymptotically the bias it gets you is an l2 regularization on all of the parameters but that's sort of uninteresting why should you ever care about the parameters again parameters


have no meaning in your own nets the only thing that matters at the end is your function f this is all you see a test time the parameter is just a way to encode F so what you should really ask is how does this bias translate over here so what is the bias on the prediction function and this is clearly where architecture matter so for example here 1 3 & 4 there was no regularization the


algorithm was just SGD the only thing that changes is the architecture yet you get a huge improvement in the test error luckily I don't know the answer for neural nets but in linear networks there's a very crisp answer you can take a linear function write it as a product of matrices then if you have outer regularization on w's which comes from gradient descent it translates a shot in


quasi norm on theta on the linear function okay so regardless of the depth and the width the prediction function class stays linear you know you can write a linear map a matrix as a product as one matrix as a product of five matrices is still matrix the function class number changes or simply varying the parameterization combined with algorithm changes the inductive bias


here and here you can characterize in a very precise sense but whatever gradient can descend converges to is a stationary point of some problem with this shot and quasi norm okay a similar phenomenon in convolutional networks same thing you can take a linear convolutional net and write it as of comp of predictors you'll also get a quasi norm except now it's sparsely instead of on the singular


values okay no it stops its learning like it's always changing zone on versatility the function is infinitely flat so in fact you can even grow the life some solar league rate of those convergence yeah you need sufficient design there's nothing okay so some random thoughts let me finish so of course we've seen now that by training Colonel training neural nets


now stay close to initialization using a kernel classes so how'd it go beyond kernels UN's I mentioned this yesterday and of course one thing missing here is how will distributional assumptions help most of these results have not used distributional assumptions in any meaningful way I would say I mean there's a lot of results about learning single relu single convolutional filters


they use Gaussian assumptions of course and those argue and even then you can see the Gaussian are helping you that much there were other ways to learn it just maybe not quite gradient descent so what are some reasonable distributional assumptions that can help us learn things via kernels don't really depend on the distribution their optimization at least so what happens right after the


ndk regime we figure out the end point in a sense we figured out what happens locally we've figured out what happens if you move infinitely far but the whole middle I don't know it's kind of interesting in architecture design an inductive bias so you know people come out with new architectures every week and how does this change the inductive bias of SGD if I do


SGD on ResNet versus amoeba net block then you know what is changing here let's say I just use STD on both but the functions I learn are different how are they different and why does amoeba net whatever generalize better [Applause] nothing GD is just easier to analyze most of the results hold for a skewed yeah that was for single oh no that's a matrix this one's a matrix this one's a