we're going to be pretty aggressive about trying to get the blood pressure down we talked about the cholesterol medicines these statins these are often very controversial medicines that patients have a lot of rumors that they've heard about that these are going to cause their muscles to dissolve or their liver to cause problems but these medicines seem to have benefits above and beyond just reducing the cholesterol they have what are called pliat trophic effects which means that they help preserve that endothelial cell layer

Stroke: The Basics


that smooth layer that's helping to protect that plaque from rupturing so that there's benefits from these agents in in many ways but we want to see the cholesterol down did a big study many years ago called the sparkle study where we took patients who'd had a stroke we put them on a high dose of this atorvastatin medicine lipitor which is a common statin 80 milligrams and many of the patients were nervous oh that's too high it's a big big dose and the other comparison was the placebo right so big dose of atorvastatin versus placebo and

Stroke: The Basics


the number who complained of muscle aches in the statin group was identical to the number complained of muscle aches in the placebo group so the conclusion is in our age everybody's got muscle aches right and we like to blame it on something so patients like to blame it on these statins but the stands really don't cause muscle aches a lot more than than not taking a statin there's some very small risk it's gonna irritate the liver so you need to check blood tests periodically to make sure that it's not doing that but most people tolerate

Stroke: The Basics


these medications very well so virtually everybody comes to the hospital with a TI a or an ischemic stroke is going to expect to go out on one of these statin agents unless their cholesterol is really low on its own and just like blood pressure what the studies have shown is that the lower the bad cholesterol the better so the LDL cholesterol we used to think get it down to 120 then 100 now we're targeting more like 70 or 80 for the LDL bad cholesterol again particularly in people who've got evidence of atherosclerosis

Stroke: The Basics


we'd had a hope that giving estrogen replacement to postmenopausal women who had stroke would help them have a lower rate of stroke and when we tested it we found it didn't work if anything it made things worse so that the estrogens can make the blood a little bit stickier so that's not something that we do for a postmenopausal woman who has a stroke and even somebody on birth control pills is that a little bit increased risk of forming clots and their legs are having a stroke because the estrogens can make the blood a little bit stickier so how

Stroke: The Basics


do we make the blood a little bit less sticky well here's a detailed picture schematic looking at blood clotting so blood clotting involves something called platelets which become activated when you have injury right to help you form a blood clot but they can also be activated by that cholesterol plaque the atherosclerosis and they can trap blood cells in this fibrin fibrous net to form a blood clot so taking a medicine that makes the platelets less sticky can reduce the chance that a blood clot is going to form on a sticky surface so the

Stroke: The Basics


most famous aspirin it's a good medicine to make the blood less sticky then we have other prescription medicines like plavix which is clopidogrel or aggregate ragnaroks which is a combination a diaper a tamal which is another prescription medicine and aspirin they can make the blood less sticky if you have a blood clot that's forming in the heart usually aspirin is not enough to prevent that so somebody with atrial fibrillation we're going to use a stronger type of blood thinning medicine that we call an anticoagulant the most

Stroke: The Basics


famous is coumadin now we have four new prescription medicines that have some advantages over coumadin as more potent blood thinners and if you have to give it intravenously then its medicines called heparin okay so how are we going to choose what we need to do is the patient comes in and we need to sort out why they had the stroke sometimes there's rare causes that we don't have time to talk about tonight but most of the time you're going to find out that the guilty party is atherosclerosis or it's a problem at the heart or as I said

Stroke: The Basics


about 30% of the time we don't figure it out if we don't figure it out we read it as if it's atherosclerosis and we assume that there may be some mild atherosclerosis that we couldn't find yeah that we didn't just didn't have sensitive enough tests to pick it up so these patients are typically going to go out on antiplatelet therapy so the least expensive is aspirin clopidogrel which is plavix is also a popular choice if it is a problem with a heart valve atrial fibrillation a heart attack forming a blood clot in the heart then typically

Stroke: The Basics


we're going to use one of these more potent anticoagulants and you can imagine those are a little more dangerous they have higher bleeding risk they need a little bit more monitoring in general so those are the typical choices here's some studies so we've been involved in several of these over the years saying that if you have one of these heart conditions like atrial fibrillation that aspirin works but the stronger blood thinners like coumadin which is also known as warfarin they work better okay the way that warfarin was discovered is

Stroke: The Basics


that this is a medicine that can cause rats to bleed it's rat poison big dose of warfarin and the rat dies and they bleed so some patients don't like this because they don't like us to prescribe rat poison and as I said now we have newer agents that don't have that checkered history that work even better than warfarin and are very safe options now the other thing you can do is try to pull the atherosclerosis out so there's the surgical option which is called carotid endarterectomy this is a picture of somebody's carotid

Stroke: The Basics


you know like in that video I showed you this you know is blocked up with one of those cholesterol plaques it can cut it open and carefully dissect it off the other approaches you can go up there with a balloon and you can squish it against the wall and put a little metal stent in there which is called carotid stenting so this would be approach if you had a minor stroke or you had a TI a and you find out that that kurata it's blocked up we're probably going to want to think about cleaning that out and here's a actual picture of what this

Stroke: The Basics


looks like so you know the years of haagen-dazs and peanut butter and potato chips you can see that the remnants here have wound up in the carotid artery and what's this red thing boycott yeah so that's the platelets that have formed this clot and it's really a combination of the cholesterol plaque and the clot that's causing the trouble so how to prevent a stroke we're going to control the treatable risk factors high blood pressure being the most important diabetes smoking and cholesterol exercise you're going to

Stroke: The Basics


take an antiplatelet agent or an anticoagulant to make the blood less sticky and if you've got a big buildup of plaque we may be able to get it out of there surgically so what we hope to prevent strokes but when they occur we want people to call 9-1-1 and get into the emergency room immediately because if you're in the midst of having a stroke which is a blood clot sitting here the other name is thrombus we need to treat it so how could you treat it if somebody had a stroke that happened 15 minutes ago and they rushed into the

Stroke: The Basics


emergency room TPA what's that but a window of like three hours to give TPA TPA is like Drano for the brain right it is a clot buster it dissolves blood clots so it would make a lot of sense that if a stroke is caused by a blood clot like 85% of them are that if you could dissolve that clot and make it disappear quickly that you could make the patient better so TPA is a clot dissolve it's made by Genentech just up the road and that is the mainstay of treatment that was the first treatment ever proven to be effective for stroke the amount of

Stroke: The Basics


injury that's going to happen from the stroke is going to depend how long the blood vessel the blood vessel is blocked by the clot and how good the collateral flow is right so if you have other blood vessels who are helping out then the stroke will progress much more slowly so again we hope that we have good collateral flow if we ever have a stroke we also hope that that clot can be dissolved quickly so TPA stands for tissue plasminogen activator and what that is is that in everybody's blood they have something called plasminogen

Stroke: The Basics


which is basically a key that can unlock a clot and if you activate that plasminogen it will start to tear these ropes apart that are the mesh work of a clod the ropes are called fibrin so when you activate the plasminogen to something called plasmon it starts to eat away at these little ropes and it can dissolve the clot so your own system is going to try to do it on the on their own but if it's a big clot probably won't be able to solve it on your own and you probably won't have a TI a you know need some help from more

Stroke: The Basics


TPA to dissolve that clot so the FDA just as you mentioned said we have three hours to give it so the approval of the FDA is that we can give it if somebody shows up and we can get it into them within three hours when they come to the ER we have to take a picture of the brain because you can imagine if you have one of those bleeding strokes the worst thing you could do would be to give TPA they're already bleeding in the brain you don't want to give a clot dissolving medicine so it means we got to get a picture of the brain we have to do some

Stroke: The Basics


blood tests we have to check what's going on even if things go very quickly in the ER that typically is going to take at least 45 minutes to do all those things you need to do so a three hour treatment window means the patient better have arrived within two hours of when the symptoms started or we can't give it right so if you woke up with the symptoms we don't know when it started right if nobody was there to call 9-1-1 we don't know when it was started so this is the biggest problem we have a very tight treatment window there have

Stroke: The Basics


been studies that show that TPA actually works out to about four and a half or five hours and in Europe the equivalent of the FDA has approved it to four and a half hours so it's Stanford and many other stroke centers even though our FDA says three hours we'll go out to about four and a half five hours with TPA but we're doing this against the advice of the FDA because they say give it very very early but that means that only about 5% of stroke patients are going to get it because they don't come in soon enough we've done a study here at

Stroke: The Basics


Stanford showing very nice benefit of TPA out to six hours if you can take a picture of the brain and show that the stroke has not gotten very big in that six hours so this is called penumbral imaging the penumbra means the part of the brain that is likely to die over the next few hours but is not yet dead so to be able to get to this we developed an MRI sequence called diffusion imaging it was one of the Stanford faculty members who discovered this MRI technique which allows you to see the stroke as it's occurring before

Stroke: The Basics


that we would just do CT scans and a CT scan doesn't show up the stroke usually 4 6 8 10 hours it's too late so we want to see the stroke as it's developing so with this technique you can see this pink area is the tissue that is irreversibly injured it's dead but the green area is the tissue that's likely to die over the next several hours so if a patient comes in like this one at six hours after symptom onset and they have a small amount of tissue that's are irreversibly injured but a large amount that is still salvageable

Stroke: The Basics


we call that a large penumbra that means that there's areas to save salvageable tissue and for these patients we're going to be very aggressive we're going to try to get that blood clot dissolved even though they're beyond the approved window for TPA so we can give TPA a little bit longer than its approved or we can physically go after the clot and the first mechanical device to be approved by the FDA to pull clots out of the brain is this mercy retriever this is from a company in Mountain View called concentric which has subsequently

Stroke: The Basics


been sold to Stryker so both TPA and this first approved mechanical device are local Bay Area products so the person who developed this was actually a French neuroradiologist and you can imagine was thinking maybe a little bit about wine and how you get the cork out of a wine bottle it's kind of the same idea of how you get a blood clot out of a blood vessel so this catheter is a tube that you're going to put all the way up into the brain and you're going to try to get this corkscrew like device into the clot so

Stroke: The Basics


here's a blood clot in the brain you bring this wire up by putting it through a blood vessel in the leg all the way up into the brain and it turns into a corkscrew you try to screw it into the clot get a hold of the clot and then pull it out and this was the first device we now have more sophisticated devices that don't look like cork screws but basically do the same thing they go up into the brain they capture the clot and they bring it out those devices have approvals out to eight hours so it gives us more time but still many patients

Stroke: The Basics


don't come into the emergency room until the next day right they think they slept on their arm wrong and then they wait till it doesn't get better the next day they call their primary care doctor and say hey my arm doesn't move they say go to the ER but it's too late okay we're gonna end and we're gonna end with this case study this is a patient I had a few years ago 45 year old man who was recovering from a surgical procedure at Stanford and he was there in his WA in the room with his wife when he had the abrupt onset of left-sided paralysis it

Stroke: The Basics


was confused so what do you think the problem might be right side of his brain yeah yeah he didn't recognize that he was having a stroke okay so it was again that area of the brain the parietal of where you don't recognize that your he knew something was going on and you'll see in the video that he was aware but his wife was on top of it and alerted the staff immediately that something was up with her husband now what do you think the treatment option might be ROK's up at the top yeah on the the right side you want to give somebody

Stroke: The Basics


like this TPA why not just had a surgical procedure right so TPA you have to think carefully before you give it because TPA can be a double-edged sword right it can cause bleeding as well as dissolve a blood clot so somebody who's either already got a big stroke or who has just had a surgical procedure TPA could be a problem so what other option would you have that wouldn't involve TPA yeah one of these clot removal devices and this was a patient who was in the hospital very shortly after that corkscrew-like

Stroke: The Basics


catheter was approved so the patient went straight to the cath lab they pulled it out and he got all better from the stroke so dramatic recovery and let me show you the video so you can look at the different perspective of the patient and his wife my speech was felt a little labored I felt tired what I felt the tactile sensation seemed to be okay every place on my bites and I could do that it was just that motor skills were a challenge it's not good that you have a stroke but we sure had a stroke of luck to be here so he was completely

Stroke: The Basics


paralyzed and did you hear his description a little problem with my hand a little motor problem so yeah it's the he's no never would have called night one one even though he's completely paralyzed on the left side so it is time for some questions or comments yeah go ahead yeah so it's a great great question the question is what's what's this three hour window business why are we being so restricted and we don't like being restricted the reason for the treat three hour window is because there were big studies done using the three hour

Stroke: The Basics


window back in 1995 showing benefit of patients treated up to three hours and subsequent studies that have looked at later time windows have not been as positive as I said there was a big European study that showed benefit to four and a half so we don't like to street stroke based on time because everybody's brain is differently it's different particularly the collaterals so we see patients like this one that I showed you here who at six hours had TPA and had this clot dissolved and had a fantastic recovery from the stroke because this

Stroke: The Basics


patient had good collaterals and only little damage was done by six hours we also see patients who had an hour and a half have massive damage so TPA will not help them they came in within the 3-hour window but the damage was already done so we need a different approach rather than TPA so the arbitrary time windows we don't like and what we've been doing is studies to try to show that it makes a lot more sense to look at the brain with sophisticated imaging and then try to figure out who has salvageable tissue

Stroke: The Basics


so if you wake up with a stroke and come to Stanford and we take a picture and it looks like this we're going to go after that blood clot if you show up two hours into the stroke and the tissue is already massively injured we're not going to remove the blood clot we're going to try to do something to reduce the swelling because a big stroke it's going to cause even more trouble from the swelling so it's a great question but that's the rule of the FDA yeah what difference genetic differences between people it would say some people

Stroke: The Basics


haven't need more cholesterol than other people no no no you know there's really not some people who need more it there was some concern that if you have very little cholesterol that your blood vessels may be a little bit more likely to cause a hemorrhage and there is some evidence that if you look at some Asian populations who tend to run very low cholesterol that they may have slightly higher brain hemorrhage rates but when you look at the studies that use the high dose of the statin versus a medium dose both were heart attack and for

Stroke: The Basics


stroke the lower the cholesterol goes the lower the stroke rate it's like the blood pressure the lower the better so there may be some point you know you can't go down to zero right it's like the blood pressure you can't go to low but in general what we found from the studies is lower cholesterol is lower heart attack and stroke rates the goal for somebody varies on what their situation is but if you've had a stroke from atherosclerosis that we want to get down below 80 on what's called the LDL the low-density right the LDL the bad

Stroke: The Basics


cholesterol by HDL you want to get it as high as possible yeah so the question is that passing out after eating so postprandial syncope that's not a stroke but it is a problem which can be difficult to treat and usually it's eating you know multiple small meals to try to deal with that you know it's difficult to comment about individual people's medical problems from from the podium but usually it doesn't get worse it's usually something that can be managed yeah okay up front yeah there was a team some years ago at University

Stroke: The Basics


of Iowa - look the ischemic optic neuropathy and found that it correlated actually with - too little blood pressure in the optic artery causing a loss of blood flow of the blood vessel itself collapsed and and they showed that the coupling of taking an antihypertensive when going to bed with with the natural diurnal Brides and follow the blood pressure that was the worst time to be taking a medication like that and it seems that when I mentioned this to people in the medical profession most people oh that's that that's new to

Stroke: The Basics


me I've never heard of that but I'm wondering whether anybody is looking at these 30% of cases that are idiopathic whether there's any chance that there was a blood vessel in the brain that actually collapsed because of because I heard a blood pressure drop and medication was a strong antihypertensive at the same time actually caused blood pressure blood pressure to drop in and blood flow to the brain to be kind of off to that or that part of the brain we kind of because there wasn't sufficient planning for this yeah so it's a

Stroke: The Basics


complicated question but what's being asked is that we talked about blood pressure being too high as a cause a stroke can blood pressure being too low because a stroke as well and the answer is yes that there certainly are situations particularly for patients who have you know major blockages of vessels where you know the plumbing is blocked up you can imagine if you don't have enough pressure you're going to run into trouble and typically the type of strokes we see from that are called watershed strokes where you have two

Stroke: The Basics


major vessels and in between them is going to be the area of low flow just like if you have two sprinklers right and you turn down the pressure you're going to get a dry area in between so there are issues and like most things there there's a you know an advantage and a disadvantage you do not want blood pressure to go too low under certain circumstances and the other part of the question was that you know the timing of blood pressure medicines which is very important what you can do is a 24 hour blood pressure monitor and sometimes

Stroke: The Basics


it'll be very surprising to see the over the course of the day things are fluctuating a lot and just taking a blood pressure in the office can be very misleading often times people's blood pressure is quite high when they go in because we're scary with our white coats on so it can be helpful and planning out how to give the blood pressure medicine certainly is an important issue in the back yeah it's a great question so question is had symptoms that may or may not have been at EIA the doctors didn't agree that

Stroke: The Basics


it's not unusual with a stroke it's pretty easy to say yes or no because the picture will say is there brain injury or not TI a no brain injury so there are lots of things that can mimic it the good news is that most of those treatments for at EIA are things that we should all do whether we've had a TI or not right blood pressure under control cholesterol under control taking an aspirin as a pretty benign approach so it is not unusual to have controversies even neurologists will disagree even stroke specialists who see a patient

Stroke: The Basics


they cannot agree on TI a if you image it at the time you can look at the blood flow and actually see a blood flow reduction but that doesn't happen very often that you're in and get the picture in the midst of the TI a or right afterwards where we may see a footprint of that blood pressure reduction so not unusual but the type of things that you're going to do after TI a are good things to do anyway yes great question does every er have TPI and you know many years ago the answer was certainly no that you know people

Stroke: The Basics


were slow to adopt TPA because it was a big change from what they were used to doing all this rushing around stroke patients there was no treatment they were like the last priority so in the Bay Area that most of the hospitals now are what are called primary stroke centers meaning that they have TPA and they have a plan and then some stroke centers have the comprehensive designation meaning they can go up with catheters and defense you're imaging so in our area we're pretty lucky of it if you call 9-1-1 at this point it's almost

Stroke: The Basics


certain that the hospital you're going to go to is TPA ready that wasn't the case a decade ago TPA is typically administered the question is how's TPA administered typically it's just given by vein intravenous infusion sometimes when we go up there with one of those catheters you try to pull out the clot and little pieces break loose and then we'll squirt some TPA to get the smaller cloth so it can be administered directly up into the clot in the brain but typically it's through the vein question in the back there yes your head

Stroke: The Basics


the question is for the statin agents does it matter if it's named brand or generic there doesn't seem to be a key difference there so I think the generic is fine far in the back yes it's inserted in the question is where as the catheter inserted its inserted in the femoral artery and then up into the brain yes so the femoral artery is it's right next to the groin area so it's the top of the leg where you you get that blood vessel yeah so question can you go elsewhere it doesn't really take any time to push it from the leg up to the

Stroke: The Basics


brain versus the arm and it's easier it's a bigger target it's a little safer to go into the leg than the arm but if the leg is all blocked up you can go into the arm years ago when they first started to do this they went through the neck that didn't go so well up right I'm interested in you mentioned there are those discrete situations that they would be used in or shouldn't many coumadin people be thinking about going on to those and secondly do the hospitals in our area generally yeah so the question about these new

Stroke: The Basics


anticoagulants and they're there for so some insurance was call cover one versus the other the first one on the market was called Pradaxa you've probably heard about that one most hospitals would you're gonna have access to that and many will have the other ones that have come out more recently so these are typically for patients with atrial fibrillation they have not been shown to be effective for people who have heart valve problems like a mechanical heart valve so it kind of depends on the reason that somebody's on coumadin if

Stroke: The Basics


they're on for atrial fibrillation and they're having some trouble with the coumadin there's a you know these offers some advantages for somebody who has a stroke from atrial fibrillation most doctors now would be talking about using one of these newer agents rather than going with coumadin because you don't have to do the frequent blood tests there's not so much in the way of food and drug interactions so they're much more convenient for the patient back there yeah you sorry I don't know anybody's name yeah so the question is can mini-strokes

Stroke: The Basics


lead to dementia so a TI a doesn't lead to dementia because as we said by definition a TI a doesn't damage the brain but little tiny strokes they can lead to dementia so dementia being a lack of cognitive ability so you can imagine if you're starting to knock off lots of little spots the connections between different areas of the brain are going to be affected those typically occur again in people with high blood pressure that the small little blood vessels deep in the brain get narrowed and you start to get these little tiny

Stroke: The Basics


strokes that people often call lacunar strokes and they can cause dementia so we don't want those to build up if there's if we can prevent a second row there go ahead yeah so question is about omega-3 and it's been a controversial area we find that the the statins have much more evidence to support them so we would recommend the statin agents the guidelines recommend the statins over the omega-3 question in the question over there have we expired our time I think one more question okay our curricular I have friends who

Stroke: The Basics


yeah so the final question is about these new anticoagulants and are they easier to regulate in the coumadin and they are definitely easier to regulate they were all tested all four of them in huge huge trials where one half took coumadin and had all the regulation issues and the other half took these new medicines with no regulation okay so you weren't upping and adjusting the dose like you have to do with coumadin and all four of these you know were either as good or better than coumadin in these big trials so they were as good or

Stroke: The Basics


better without all that regulation so somebody who's blood thinning is jumping all over the place on coumadin these are much smoother now the disadvantage is that we don't have great tests to monitor how much is in the blood with coumadin we can tell exactly how much is in there with it with the blood tests the newer agents that's a little bit trickier and then there's a little bit of an issue with reversal of these newer agents that coumadin is hard to reverse also but we have medicines with a lot more we have a lot more experience

Stroke: The Basics


hi and welcome to an illustrated guide to recurrent neural networks a Michael also known as learned vector I'm a machine learning engineering the natural language processing and voice assistance space if you're just getting started in machine learning and want to get some intuition behind recurrent neural networks those videos for you if you want to get into machine learning recurrent neural networks are a powerful technique that's important to understand if you use smart phones and frequently surf the internet odds are you use

Illustrated Guide to Recurrent Neural Networks: Understanding the Intuition


applications that leverages RN ends recurrent neural networks are using speech recognition language translation stock prediction it's even using image recognition to describe it content in pictures so I know there are many guys on recurrent neural networks but I want to share illustrations along with an explanation of how I came to understand it in this video I'm going to avoid all the math and focus on the intuition behind RN ends instead by the end of this video you should have a good understanding of RN ends and hopefully


have that light bulb moment so RN ends are neural networks that are good at modeling sequence data to understand what that means let's do a thought experiment say you take a still snapshot of a ball moving in time let's also say you want to predict a direction that the ball is moving so with only the information that you see on the screen how would you do this well you can go ahead and take a guess but any answer you come up with would be that a random guess without knowledge of where the ball has been you weren't having an


update of to predict where it's going if you record many snapshots of the balls position in succession you will have enough information to make a better prediction so this is a sequence a particular order in which one thing follows another with this information you can now see that the ball is moving to the right sequence data comes in many forms audio is the natural sequence you can chop up an audio spectrogram into chunks and feed that into RN ends text is another form of sequences you can break text up into sequence of characters or


sequence of words okay so our ends are good at processing sequence data for predictions but how well they do that by having a concept I like to call sequential memory to get a good intuition behind what sequential memory means I like to invite you to say the alphabet in your head go on give it a try that was pretty easy right if you were taught the specific sequence it should come easily to you now try saying the alphabet backward I bet that was much harder unless you practice the sequence before you'll likely have a


hard time here's a fun one start out the letter F at first just struggle with the first few letters but then after your brain picks up the pattern the rest will come naturally so there's a very logical reason why this can be difficult you learn the alphabet as a sequence sequential memory is a mechanism that makes it easier for your brain to recognize sequence patterns all right so are n ends have this abstract concept of sequential memory but how the heck does it replicate that concept well let's look at a traditional neural network


also known as a feed-forward neural network as an input layer hidden layer and output layer how do we get a feed-forward neural network to be able to use previous information to affect later ones what have we had a loop in a neural network that can pass previous information forward and that's essentially what a recurrent neural network does an RNN has a looping mechanism that acts as a highway to allow information to flow from one step to the next this information is the hidden state which is a representation


of previous inputs let's run through an art and use case to have a better understanding of how this works let's say we want to build a chatbot they're pretty popular nowadays let's say the chatbox can classify intentions from the user's inputted text to tackle this problem first we're going to encode the sequence of texts using an RNN then we're going to feed the RNA and output into a feed-forward neural network which will classify the intents okay so a user types in what time is it to start we break up the sentence into


individual words rnns work sequentially so we feed it one word at a time the first step is to feed what into the RNA the RNA encode what and produces an output for the next time we beat the work time in a hidden state from the previous step remember that the hidden state represent information from all previous steps the RNN now has information about the work what in time we repeat this process until the final step you can see about a final step the Arnon has encoded information from all the words in the previous steps since


the final output was created from the rest of the sequence we should be able to take the final output and pass it to the feed-forward layer to classify in intent for those of you who like looking at code here are some Python showcasing the control flow first you initialize your network layers in the initial hidden state the shape and dimensions of the hidden state will be dependent on the shape and dimension of your recurrent rail network then you loop through your inputs past a word and hence a into the artnet DRN and returns


the output at a modified hidden state this modified hidden state should now contain information from all your previous steps you continue to loop until you're out of words last you pass the output to the feed board layer and it returns a prediction and that's it the control flow of doing a forward pass of a recurrent neural network is a for loop okay now back to our visualization you may have noticed the odd distribution of colors in the hidden states this is to illustrate an issue with our n ends known as short-term


memory short-term memory is caused by the infamous vanishing gradient problem which is also prevalent in other neural network architectures so as yarn and processes more steps it has troubles retaining information from previous steps as you can see the information from the word what and time is almost non-existent at the final step short-term memory and vanishing gradient is due to the nature of back propagation algorithm used to Train and optimize neural networks to understand why this is let's take a look at the effects of


back propagation on a deep feet board neural network training and neural network has three major steps first it does a forward pass and makes a prediction second it compares the prediction to the ground truth using a loss function the loss function outputs an error value which is an estimate of how badly the network is performing last it uses the error value to do back propagation which calculates the gradients for each node in the network the gradient is a value used to adjust the network's internal weights allowing


the network to learn the bigger the gradient the bigger the adjustments and vice versa here's where the problem lies when doing back propagation each node in a layer calculates its gradient with respect to the effects of the gradients and the layer before so the adjustments in the layer before it is small then the adjustments in the current layer will be even smaller this cost gradients to exponentially shrink as it back propagates down the earlier layers failed to do any learning as the internal weights are barely being


adjusted due to extremely small gradient and that's the vanishing gradient problem let's see how this applies to recurrent neural networks you can think of each time step and over current no network as a layer to train a recurrent neural network use an application of backpropagation called back propagation through time the gradients value will exponentially shrink as it propagates for each time step again the gradient is used to make the adjustments in the neural networks weights thus allowing it to learn small gradients means small


adjustments this causes the early layers to not learn because of the vanish ingredients the RNN doesn't learn the long-range dependencies across time steps this means that there is a possibility that the word light and time are not considered when trying to predict a user's intention the network has to make its best guess with is it that's pretty ambiguous and would be difficult even for a human so not being able to learn on earlier time steps causes the network tap short-term memory okay so RNN suffer from short-term


memory so how do we combat that to mitigate short-term memory to specialized recurrent neural networks were created one called long short term memory or lsdm for sure the other is gated recurrent units or gr use LS TMS and gr use essentially fokin just like our meds but they're capable of learning long-term dependencies using mechanism called gates these gates are different tensor operations that can learn what information to add or remove to the hidden state because of this ability short-term memory is less of an


issue for them to sum this up RNs are good for processing sequence data for predictions but suffer from short-term memory the short-term memory issue for vanilla arm ends doesn't mean to skip them completely and you still more involved versions like LST MS or gr use RNs have the benefit of training faster and uses less computational resources that's because there are less tensor operations to compute you could use LST MS or gr use when you expect a model longer sequences with long-term dependencies if you're interested in


all right cool well thanks everybody um so I'm gonna give the second talk tonight which I'm not crazy about and and I don't want this pattern to to repeat but you know Andrew and I wanted to kick this series off and and felt like me talking twice or better than then not but we're gonna we're gonna get more diversity of folks if any of you want to give a talk yourselves you know somebody who you think might that'd be awesome but a topic that I feel is important for practitioners to understand is a real sea change in

LSTM is dead. Long Live Transformers!


natural language processing that's you know all of like 12 months old but is one these things I think is incredibly significant in in the field and that is the advance of the Transformers so the outline for this talk is to start out with some background on natural language processing and sequence modeling and then talk about the LS TM why it's awesome and amazing but still not good enough and then go into Transformus and talk about how they work and why they're amazing so for background on natural language processing NLP I'm gonna be talking just

LSTM is dead. Long Live Transformers!


about a subset of NLP which is the supervised learning a part of it so not structured prediction sequence prediction but where you're taking the document as some input and trying to predict some fairly straightforward output about it like is this document spam right and so what this means is that you need to somehow take your document and represent it as a fixed-size vector because I'm not aware of any linear algebra that works on vectors of variable dimensionality and the challenge with this is that documents are of variable length right

LSTM is dead. Long Live Transformers!


so you have to come up with some way of taking that document and meaningfully encoding it into a fixed size vector right so the classic way of doing this is the bag of words right where you have one dimension per unique word in your vocabulary so English has I don't know about a hundred thousand words in the vocabulary right and so you have a hundred thousand dimensional vector most of them are zero because most words are not present in your document and the ones that are have some value that's maybe account or

LSTM is dead. Long Live Transformers!


tf-idf score or something like that and that is your vector and this naturally leads to sparse data where again it's mostly zero so you don't store the zeros because that's computationally inefficient you store lists of a position value tuples or maybe just a list of positions and this makes the computation much cheaper and this works this works reasonably well a key limitation is that when you're looking an actual document order matters right these two documents mean completely different things right but a bag of words model will score them

LSTM is dead. Long Live Transformers!


identically every single time because they have the exact same vectors for for what words are present so the solution to that in this context is in grams you can have by grams which every pair of possible words or trigrams are for every combination of three words which would easily distinguish between those two but now you're up to what is that a quadrillion dimensional vector and you can do it but you know you start running into all sorts of problems when you walk down that path so a in neural network land it's the natural way to just solve

LSTM is dead. Long Live Transformers!


this problem is the R and n which is the recurrent neural network not the recursive neural network I've made that mistake but RN ends are a new approach to this which asked the question how do you calculate a function on a variable-length set of input and they answer it using a for loop in math where they recursively define the output at any stage as a function of the inputs at the previous stages and the previous output and then for the purpose of supervised learning the final output is just the final hidden state here and so

LSTM is dead. Long Live Transformers!


visually this looks like this activation which takes an input from the raw document X and also itself in the previous time you can unroll this and visualize it as a very deep neural network where there the final answer the number you're looking at the end is this and it's this deep neural network that processes every one of the inputs along the way alright and the problem with this classic vanilla all right on this plane recurrent neural network is vanishing an exploding gradients right so you take this recursive definition of

LSTM is dead. Long Live Transformers!


the hidden state and you imagine what happens just three points in right and so you're calling this a function this a transformation over and over and over again on your data and classically in the vanilla want a case this is just some matrix multiplication some learned matrix W times your input X and so when you go out and say a hundred words in you're taking that W vector W matrix and you're multiplying it a hundred times alright so in in simple math in in real number math we know that if you take any number less than one and raise it to a

LSTM is dead. Long Live Transformers!


very high dimensional value sorry very high exponent you get some incredibly small number and if your number is slightly larger than one then it blows up to something big and if you go if your X even higher if you have longer documents this gets even worse and in linear algebra this is about the same except you need to think about the eigenvalues of the matrix so the eigenvalues is say how much the matrix is going to grow or shrink vectors when the transformation is applied and if your eigenvalues are less than one in this transformation you're

LSTM is dead. Long Live Transformers!


going to get these gradients that go to zero as you use this matrix over and over again if they're greater than one then your gradients are going to explode all right and so this made vanilla RN ends extremely difficult to work with and basically just didn't work on anything but fairly short sequences all right so LST m to the rescue right so I wrote this document a few years ago called the rise and fall and rise and fall of LST M so at least Tim came around in the dark ages and then it went into the AI winter it came back again

LSTM is dead. Long Live Transformers!


for awhile but I think it's on its way out again now with with transformers so Ellis Tim to be clear is a kind of recurrent neural network it just houses more sophisticated cell inside and it was invented originally in the dark ages on stone tablet that has been recovered into a PDF that you can access on Sep hawk right there's a server III kid but seven and you're gonna both grade I enjoy they're both quite a bit but they did a bunch of amazing work in the 90s that was really well ahead of its time and and often get neglected and

LSTM is dead. Long Live Transformers!


forgotten as time goes on that's totally not fair because they did an amazing research so the LST emcell looks like this it actually has two hidden states and the the input coming along the bottom and the output up the top again and these two hidden states and I'm not going to go into it in detail and you should totally look at Christopher ollas blog post if you want to dive into it but the key point is that these these transformations these the matrix multiplies right and they are not applied recursively on the main hidden

LSTM is dead. Long Live Transformers!


vector all you're doing is you're adding in or the forget gate yeah you actually don't really need it but you're adding in some some new number and so the OS TM is actually a lot like a res net it's a lot like a CNN resonate in that you're adding new values on to the activation as you go through the layers right and so this solves the exploding and vanishing gradients problems however LST M is still pretty difficult to train because you still have these very long gradient paths even even without even with those residual connections you're

LSTM is dead. Long Live Transformers!


still propagating gradients from the end all the way through this transformation cell over at the beginning and for a long document this means very very deep networks that aren't just Toria Slee difficult to train and more importantly transfer learning never really worked on these LST M models right one of the great things about image net and cnn's is that you can train a convolutional net on millions of images in image net and take that neural network and fine-tune it for some new problem that you have and the the starting state of

LSTM is dead. Long Live Transformers!


the you mention at CNN gives you a great a great place to start from when you're looking for a new neural network and makes training on your own problem much he there was much less data that never really worked with Ellis Jim sometimes it did but it just wasn't very reliable which means that anytime you're using an LS TM you need a new label data set that's specific to your task and that's expensive okay so this this changed dramatically just about a year ago when the burp model was was released so you'll hear people talk about

LSTM is dead. Long Live Transformers!


Transformers and Muppets together and the reason for this is that the original paper on this technique that describes the network architecture it was called the transformer network and then the Bert paper is a muppet news and Elmo paper and you know researchers just run with the joke um so this is just context you understand what people are talking about if they say well use them up in network so this I think it was the natural progression of the sequence of document models and it was the transformer model was first described

LSTM is dead. Long Live Transformers!


about two and a half years ago in this paper attention is all you need and this paper was addressing machine translation so think about taking a document in in English and converting it into French right and so the classic way to do this in neural network is encoder/decoder here's the full structure there's a lot going on here right so we're just going to focus on the encoder part because that's all you need for these supervised learning problems the decoder is similar anyway so zooming in on the encoder part

LSTM is dead. Long Live Transformers!


of it there's still quite a bit going on and so we're but basically there's three parts there's we're gonna talk about first we're going to talk about this attention part then we'll talk about the part of the bottom of the positional coding the top parts just not that hard it's just a simple fully connected layer so the attention mechanism in the middle is the key to making this thing work on documents of variable lengths and the way they do that is by having an all-to-all comparison for every layer of the neural network it considers every

LSTM is dead. Long Live Transformers!


pause for every output of the next layer considers every plausible input from the previous layer in this N squared way and it does this weighted sum of the previous ones where the waiting is the learned function right and then it applies just a fully connected layer after it but it this is this is great for for a number of reasons one is that you can you can look at this thing and you can visually see what it's doing so here is this translation problem of converting from the English sentence the agreement on the European

LSTM is dead. Long Live Transformers!


Economic Area was signed in August 1992 and translate that into French my apologies la casa la zone economic European at this may and oh I forgot 1992 right and you can see the attention so as its generating lips as a generating the each token in the output it's it's starting with this whole thing's name button its generating is these output tokens one at a time and it says okay first you got to translate the the way I do that it translates into la and all I'm doing is looking at this next I'll put a color all I'm doing is

LSTM is dead. Long Live Transformers!


looking at agreement then sir is on la is the okay now interesting European Economic Area translates into zone economic European so the order is reversed right you can see the attention mechanism is reversed also or you can see very clearly what this thing is doing as it's running along and the way it works in the attention are setting the transformer model the way they describe it is with query and key vectors so for every output position you generate a query and for every input you're considering you generate a key

LSTM is dead. Long Live Transformers!


and then the relevant score is just the dot product of those two right and to visualize that you first you combine the key the query and the key values and that gives you the relevant scores you you use the softmax normalize them and then you do a weighted average of the values the third version of each token to get your output now to explain this in a little bit more detail I'm going to go through it in pseudocode so this looks like Python it wouldn't actually run but I think it's close enough to help people understand what's going on

LSTM is dead. Long Live Transformers!


so you've got this attention function right and it takes as input a list of tensors I know you don't need to do that a list of 10 serious one per token on the input and then the first thing it does it goes through each everything in the sequence and it computes the query the key and the value by multiplying the appropriate input vector by Q k and V which are these learned matrices right so it learns this transformation from the previous layer to whatever should be the query the key and the value at the at the next layer then it

LSTM is dead. Long Live Transformers!


goes through this double nested loop alright so for every output token it figures out okay this is the query I'm working with and then it goes through everything in the input and it multiplies that query with the the key from the possible key and it computes a whole bunch of relevant scores and then it normalizes these relevant scores using a soft Max which makes sure that they just all add up to one so you can sensibly can use that to compute a weighted sum of all of the values so you know you just go through for each output

LSTM is dead. Long Live Transformers!


you go through each of the each of the input tokens the value score which is calculated for them and you multiply it by the relevance this is just a floating point number from 0 to 1 and you get a weighted average which is the output and you return that so this is what's going on in the attention mechanism which can be which can be pretty confusing when you just look at it look at the diagram that like that but I hope this I hope this explains it a little bit I'm sure we'll get some questions on this so relevant scores are interpretable as I

LSTM is dead. Long Live Transformers!


say and and this is is super helpful right now the an innovation I think it was novel in the transformer paper is multi-headed attention and this is one of these really clever ID and important innovations that it's not actually all that complicated at all I you just do that same thing that same attention mechanism eight times whatever whatever value of 8 you want to use and that lets the network learn eight different things to pay attention to so in the translation case it can learn an attention mechanism for grammar one for

LSTM is dead. Long Live Transformers!


vocabulary one for gender one for kent's whatever it is right whatever the thing needs to it can look at different parts of the input document for different purposes and do this at each layer right so you can kind of intuitively see how this would be a really flexible mechanism for for processing a document or any sequence okay so that is when the key things that enables the transfer model that's the multi-headed attention part of it now let's look down here at the positional encoding which is which is critical and novel in a

LSTM is dead. Long Live Transformers!


critical innovation that I think is incredibly clever so without this positional encoding attention mechanisms are just bags of words right there's nothing seeing what the difference is between work to live or live to work right there they're just all positions they're all equivalent positions you're just going to compute some score for each of them so what they did is they took a lesson from Fourier theory and added in a bunch of sines and cosines as extra dimensions sorry not as extra dimensions but onto the the word embeddings so going back so

LSTM is dead. Long Live Transformers!


what they do is they take the inputs they use word Tyvek to calculate some vector for each input token and then onto that onto that embedding they add a bunch of sine and cosines of different frequencies starting at just pi and then stretching out longer and longer and longer and if you look at the whole thing it looks like this and what this does is it lets the model reason about the relative position of any tokens right so if you can kind of imagine that the model can say if the orange dimension is slightly higher than the

LSTM is dead. Long Live Transformers!


blue dimension on one word versus another then you can see how it knows that that token is to the left or right of the other and because it has this at all these different wavelengths it can look across the entire document at kind of arbitrary scales to see whether one idea is before or after another the key thing is that this is how the system understands position and isn't just a bag of words for Fortran when doing the attention okay so transformers there's the two key innovations as positional encoding and

LSTM is dead. Long Live Transformers!


multi-headed attention transformers are awesome even though there are N squared and the length of the document these all-to-all comparisons can be done almost for free in a modern GPU GPUs changed all sorts of things right you can do a thousand by thousand matrix multiply as fast as you can do a ten by two in a lot of cases because they have so much parallelism they have so much bandwidth that but a fixed latency for every operation so you can do these massive massive multiplies almost for free in a lot of cases so doing things

LSTM is dead. Long Live Transformers!


in M Squared is is not actually necessarily much more expensive whereas in an RNN like an L STM you can't do anything with token 11 until you're completely done processing token 10 all right so this is a key advantage of transformers they're much more computationally efficient also you don't need to use any of these sigmoid or tan h activation functions which are built into the LS TM model these things of scale your activations to 0 1 why are these things problematic so these were bread-and-butter in the old days of of

LSTM is dead. Long Live Transformers!


neural networks people would use these between layers all the time and they make sense there that kind of biologically inspired you take any activation you scale it from 0 to 1 or minus 1 to 1 but they're actually really really problematic because if you get a neuron which has a very high activation value then you've got this number up here which is 1 and you take the derivative of that and it's 0 or it's some very very small number and so your gradient descent can't tell the difference between an activation up here

LSTM is dead. Long Live Transformers!


and one way over on the other side so it's very easy for the trainer to get confused if your activations don't stay near this middle part all right and that's problematic compare that to rel U which is the standard these days and really you yes it does have this this very very large dead space but if you're not in the dead space then there's nothing stopping it from getting getting bigger and bigger and scaling off to infinity and one of the reasons why when the intuitions behind why this works better as Geoffrey Hinton puts it is

LSTM is dead. Long Live Transformers!


that this allows each neuron it to express a stronger opinion right in an LS sorry in a sigmoid there is really no difference between the activation being three or eight or twenty or a hundred the output is the same right it all I can say is kind of yes no maybe right but in with our Lu it can say the inactivation of five or a hundred or a thousand and these are all meaningfully different values that can be used for different purposes down the line right so each neuron it can express more information also the gradient doesn't

LSTM is dead. Long Live Transformers!


saturate we talked about that and very critically and I think this is really underappreciated values are really insensitive to random initialization if you're working with a bunch of sigmoid layers you need to pick those random values at the beginning of your training to make sure that your activation values are in that middle part where you're going to get reasonable gradients and people used to worry a lot about what initialization to use for your neural network you don't hear people worrying about that much at all anymore and rail

LSTM is dead. Long Live Transformers!


users are really the key reason why that is also really runs great on low precision Hardware those those floating of the smooth activation functions they need 32-bit float maybe you can get it to work in 16-bit float sometimes but you're not going to be running it an 8-bit int without a ton of careful work and that is the kind of things are really easy to do with a rel u based network and a lot of hardware is going in that direction because it takes vastly fewer transistors and a lot less power to do 8-bit integer math versus

LSTM is dead. Long Live Transformers!


32-bit float it's also stupidly easy to compute the gradient it's one or at zero right you just take that top bit and you're done so the derivatives ridiculously usually rel you have some downsides it does have those dead neurons on on the left side you can fix that with a leaky rail you there's this discontinuity in the gradient of the origin you can fix that with Gail u which burnt uses and so this brings me to a little aside about general deep learning wisdom if you're designing a new network for whatever reason don't

LSTM is dead. Long Live Transformers!


bother messing with different kinds of activations don't bother trying sigmoid or tant they're they're probably not going to work out very well but different optimizers do matter atom is a great place to start it's super fast it tends to give pretty good results it has a bit of a tendency to overfit if you really are trying to squeeze the juice out of your system and you want the best results SGD is likely to get you a better result but it's going to take quite a bit more time to Verge sometimes rmsprop beats the pants

LSTM is dead. Long Live Transformers!


off both of them it's worth playing around with these with these things I told you about why I think SWA is great there's this system called attitude my old team at Amazon released where you don't even need to take a learning rate it dynamically calculates the ideal learning rate scheduled at every point during training for you it's kind of magical so it's worth playing around with different optimizers but don't mess with the with the activation functions okay let's pop out right there's a bunch of a bunch of Theory bunch of math and and

LSTM is dead. Long Live Transformers!


ideas in there how do we actually apply this stuff in code so if you want to use a transformer I strongly recommend hopping over to the the fine folks at hugging face and using their transformer package they have both pi torch and tensor flow implementations pre-trained models ready to fine tune and I'll show you how easy it is here's how to fine tune a Bert model in just 12 lines of code you just pick what kind of Bert you want the base model that's a paying attention upper and lower case you get the tokenizer to convert your string

LSTM is dead. Long Live Transformers!


into tokens you download the pre trained model in one line of code pick your data set for your own problem process the data set with the tokenizer to get training validation splits shuffle one batch um four more lines of code another four lines of code to instantiate your optimizer define your loss function pick a metric it's tensorflow so you got to compile it and then you call fit and that's it that's all you need to do use - all you need to do to fine-tune a state-of-the-art language model on your specific problem

LSTM is dead. Long Live Transformers!


and the fact you can do this on some pre trained model that's that's seen tons and tons of data that easily is really amazing and there's even bigger models out there right so Nvidia made this bottle called megatron with eight billion parameters they ran a hundreds of GPUs for over a week spent vast quantities of cash well I mean they own the stuffs but so not really but they they put a ton of energy into training this I've heard people a lot of people complaining about how much greenhouse gas comes from training model like

LSTM is dead. Long Live Transformers!


Megatron I think that's totally the wrong way of looking at this because they only need to do this once in the history of the world and everybody in this room can do it without having to burn those GPUs again right these things are reusable and fine tunable I don't think they've actually released this yet but but they might and somebody else will right so you don't need to do that that expensive work over and over again write this thing learns a base model really well the folks at Facebook trained this Roberta model on two and a half

LSTM is dead. Long Live Transformers!


terabytes of data across over a hundred languages and this thing understands low resource languages like Swahili and an Urdu in ways that the it's just vastly better than what's been done before and again these are reusable if you need a model that understands all the world's languages this is accessible to you by leveraging other people's work and before Bert and transformers and the Muppets this just was not possible now you can leverage other people's work in this way and I think that's really amazing so to sum up the key advantages

LSTM is dead. Long Live Transformers!


of these transforming networks yes they're easier to train they're more efficient all that yada yada yada but more importantly transfer learning actually works with them right you can take a pre trained model fine-tune it for your task without a specific data set and another really critical point which I didn't get a chance to go into is that these things are originally trained on large quantities of unsupervised text you can just take all of the world's text data and use this as training data the way it works very very

LSTM is dead. Long Live Transformers!


quickly is kind of comparable to how word Tyvek works where the language model tries to predict them some missing words from a document and in that's enough for it to understand how to build a supervised model using vast quantities of text without any effort to label them Ellis team still has its place in particular if the sequence length is very long or infinite you can't do n squared right and that happens if you're doing real time control like for a robot or a thermostat or something like that you can't have the entire sequence and for

LSTM is dead. Long Live Transformers!


some reason you can't pre train on some large corpus LS TM seems to outperform transformers when your dataset size is is relatively small and fixed and with that I will take questions well you yes yeah would CNN how do you compare words CNN transformer so when when I wrote this paper the rise and fall and rise and fall of LST M I predicted that time that word CNN's were going to be the thing that replaced LST M I did not I did not see this this transformer thing coming so a word CNN has a lot of the advantages in terms of

LSTM is dead. Long Live Transformers!


parallelism and the ability to use rel you and the key difference is that it only looks at a fixed size window fixed size part of the document instead of looking at the entire document at once and so it's it's got a fair amount fundamentally in common word CNN's have an easier task easier time identifying diagrams trigrams things like that because it's got those direct comparisons right it doesn't need this positional encoding trick to try to infer with with fourier waves what where things are relative to each other so

LSTM is dead. Long Live Transformers!


hi everyone I hope everyone here okay so I'm pleased to be here with you today to share with you some of my experience in Catalan during several years so the title is called is easy so it's quite different from other talks where it was more technical about how to win competitions here I will keep it rather simple and I will talk about what I was doing before and what I started doing to improve myself so I will start by giving some general advices that everyone should follow and I will talk about some technical sir

Gold is easy: Kaggle tips and tricks | by Chahhou Mohamed | Kaggle Days Dubai | Kaggle


they me about how to improve or to check for improvements in your models and then I will talk quickly about some case studies and see that you will see that getting a gold medal is not sometimes well most of the times not that difficult so starting with the advices first advice positive mind if others can do it you can also do it that is how I started when I started kaggle I said if those people can do it I can do it I compete a lot and I managed to become again my second advice understand the problem I try to find you IDs never starts


fine-tuning your put your model during the first week while you are still doing some feature engineering steps you are still looking for a good architectures you will just waste most of your time doing by training spend this time trying different approaches different architectures to come up with different models for your final example turtle bites don't use caramels when you start a competition you will mostly end up using a slice variation of that kernel so don't look at caramels when you start try doing that data analysis


by yourself try to come up with your own models with your or Peter engineering you will end up with models that are different from doors that you will find on Kegel later on you can use ideas you can borrow ideas from cattle camels to help your model improve even better if you have the chance to work in teams don't share anything except some important insights so when I work in team with my teammates we don't share the architectures of our models we don't share the features that we create we only share important insights that we


find by doing the analysis of the data if something doesn't work don't stick to it you will mainly waste your time find a different approach try different modeling approaches to your problem you will end up with different solutions let me help you another advice is you should always keep it simple we saw previously in different competitions some awesome solutions that use hundreds of models which stuck in and as a as beginners or even at some advanced levels you may say that I can will never be able to do something like


this stuck in is very an advanced topic so try to to make simple models simple models will also work and to help you to get a gold medal don't understa mage the power of neural networks even on tabular data mostly on tabular data people try to use only gradient boosting decision trees because they are so powerful moral networks can be as good as present boosting decision trees if you are able to come up with a good architectures and it will also help you your final ensemble if you are working with two models right inputs


boosting decision trees and or immensely another point is throws for IDs that's how a college is try to work on different fields to image images in AP time series suppler don't mean classification regression try to work on as many different topics as possible ideas that you may use in in images for example can be used on tabular data if you know how to model your problem in order to use such ideas and the last idea and the last point is thinking outside of the box don't think don't try to do what other people does do because


we'll end up with the same solution and you will end up with the same idea try to do it things that people don't think about trying new thoughts new new ideas most of the time there will be crazy ideas that may not work but you have to try them and if one of those ideas work it will get you the gold medal so checking for improvement I would start talking talking about Peter importance how to assess the importance of so one thing that's all so what does I have no ties among calculus is that they are always trying


to eliminate features looking for the lorikeets filters trying to eliminate them in order to improve the performance of the model and to make it faster to trade so I never do that when I start a competition but let's say I never do that it's during the first modelling approach but later on you have to do it so why I don't use it I thought eliminates ranking filters when I thought because you may have correlated features that they explain the low ranking of some filters when we will do feature engineering new filters that will you


will end up coming with may have some good interactions with the lower ranking filters and you will see an increase if they're in their importance and also the ranking of the filters is dependent of the complexity of the models some low ranking filters may have they become really important if you increase the complexity or if you decrease the complexity some some high important filters may decrease so here is an example using the Titanic exam so on the left you can see the original row features and on the right I just added a


new feature that I called noise filter that I made from a normal distribution and as you can see I made a grid search trying to find the best parameters for the two models and you can see here that the noisy feature is ranking third which doesn't make sense this is just random noise so if you start looking at the lorikeets features you will try to eliminate some of those filters and will completely forget about the noisy filter so this is something I have noticed when I was working on the first silicon petition I made up


hundreds of meters and I tried to use the the noise Peter trick which is just made a noisy filter and see the ranking and if anything is rank it below the noisy filter it just means that it it's just noise you can eliminate this but it happens that in practice doesn't work and my noisy Peter even though I have hundreds of meters was always ranking in the top five filters so don't look at the low ranking Peters but start looking at those that are ranking very very high your model is probably overfitting on some noisy


filters and by eliminating those features first because the low rugged Peters may become more important because your model will try to use them instead of using the noisy Peters so the main purpose is to detect the overfitting meters you have two choices you remove them if it helps your cross-validation and if it hurts your cross-validation then just try to apply some transformation on those filters to have a better generalization so the second part of checking the improvements is about Twitter engineering and this is


quite tricky to to find what feature is working or not so when you come up with a new Peter and you want to check if this feature is helpful in your model or not well if you see very important increase in your score you know for sure that the Peter is helpful but most of the time the filters just give you some little improvements from literally trees of maybe decrease in your first validation spot so how will you assess that for example the third change happens from using this filter or is it just from some random seeds that you are


you into your model so first approach that you can use is doing bucket so what is bunion is just use the same model with different seats and try to to average the predictions out of those models and see the final your final cross-validation spot so instead of using a five fold cross validation this will allow you to use just one one random seat it's just one model I prefer using three bucks or 3/4 is almost as fast as doing a 5 first foundation because you have less training data and which just refers instead of 5 volts


oils almost as fast that it allows me to use 3 different models with 3 differences and by beginners blending those models I somehow gets rid of the randomness of somebody my final score is not really dependent of the seat that have been used in my model and I can be quite sure that the improvements that I see is not coming from randomness but really from the improvement so this is one way to do it the second way and this is how I do things when I I mean I do Peter engineering if I never use or I never test only one feature at once I create


what I call or I will be back to this line later on I always do what I call a bunch of features so I create let's say for example five features and instead of trying those features one by one I try to fill the full bunch of features at once and if improvement there is it will be not very small improvement but rather a quite important improvement and this works very very well so why does it work so well well let's imagine that you have two features a future one which is just a mean computed on some feature X grouped by


future while and some other filter to that use the feature one but you just remove the mean from your filter types you may end up if you try those features one by one you will may end up but not see but not seeing any improvement in your model but using them but in your model you may come up with some interesting improvements why because those two features are quite related between them one is the mean of the group and the other is just the difference of the filter X to the two this mean and maybe this feature two is


not working quite good by itself unless it sees to have some kind of interaction with the mean of the group so by creating a bunch a bunch of features you are allowing for more interactions then trying feature by filter and when you are done with this bunch of feature you make for example five features and you see improvements at that point you can start looking if you have any overfitting features and you can sometimes but say what I do is starting doing feature selection during my bunch of future creation so I create like 10


meters I see improvement I check for overfitting features I remove them and see if all the filters are useful so in some users may not be used for into these bands by just remove it one of the most powerful features that I have found so far and almost always work is this veto this kind of features you provide some feature wide and you compute some statistics using another feature and then you just remove that beam from the original row so you can apply you apply this on categorical filter features you can also do it on


discrete amiracle meters and you can also apply it on continuous features by doing some discretization just before also tabular data can have special special and temporal information in them so you may group by different filters with different granularities and they also derive your own the regularities from - - to make your blue pipe working so we in the two sigma competition within that end up in the second place just by using those those kind of features so we have some technical indicators and some fundamental


indicators and we we did this on some features and our our global our while Peter was the day so for each day we just compute the mean and look at the deviation of each sample of each observation to the mean a touch even day and we end up with different models that we went together and that was enough to win the competition just with those kind of the turtle third parts for improvement is about to be on fire so here I'm going to talk about our pliers in the targets that you are trying to predict so our pliers can be


due to different reasons so for example the home pricing you may contain some typos maybe you have some under priced sales because this happened between family members or maybe there was some fraud that happens so you know something that you can not predict because it's under price or maybe you have something over priced some cells that's big houses on some big island but you have very few samples of them so you can not find some interesting patterns not enough samples to learn anything interesting from London so how to deal with this kind of


our piles so during the trading pains you can start by keeping the outliers and have some kind of deadlines for then you can apply some transformation of the target and see if there is any improvements this alone you can do what we call insulation so winds organization and we've talked about it later this is the second thing that you should try and the third thing that you should try is removing the outliers so winterization is just you need to cut the target so if you see some values that are higher than a given threshold


you just put any higher value to that threshold okay or you can also cut the error so joining for addiction during the learning phase you have to write your own loss and what you do is you compute the error and for big errors you need to cap them so that your model will not will not be biased towards those predators those outliers especially if you are using some metrics like the mean squared error which encases the importance of the outliers so after trying the winterization will have a new spot that you can compare with your


baseline and the third approach is the removal of the outliers so how can you do the removal you just build a model and predict on the train dataset and you have to use a cross validation approach it's mandatory otherwise it's not going to work and you will predict and using those prediction you will get an error and based on this error you will find an optimal threshold and you will remove from your training data all the samples that have an error higher than the threshold that you have fixed during the testing you should not apply the


winterization or the test but regarding the removal of the outliers in general you can keep the outliers in your testing data set unless you are sure 100% that the outliers can make a hug in no case they can they can be predicted so you can remove them from from your from your test also when you see improvements when you are doing feature engineering and you see some improvement with a new feature how do you know that this feature is important because if you keep the outliers in your testing data maybe with some luck you are predicting


a good out player that will improve your score but the feature by itself is meaningless it's just some luck that happened on some of my own so you should always check on test data with and without our players to see if the improvement is from some kind of general pattern that your model has have discovered or it's just luck some outliers okay some additional tips that I can give you here always try blending your models so what I mean here is doing when working on tabular data you will do a lot of feature engineering


you will sometimes end up with hundreds of feet off so instead of using all those features in just one model you can create make different models with different subsets of features and try to blend together all those models and among with with the the feature engineer that we did in the two sigma competition we tried five different models I mean five same models but with different subsets of features and did how about a lot because what we have no ties we are we were trying to predict in the future so that was the time series time series


problem and we were trying to predict in the future on a large time sample and we noticed that sometimes adding one feature in our model did improve our prediction for some modes but decreased our score for other mounts so each feature was not working on the whole time or the whole unseen future so they were working on some periods I'm not working on some periods and so what we did is we clustered somehow those features in two different spots though the features that were working on some periods we put them together and those


that were working quite well or some other periods we put them on another model and by blending all those model what else we end up with something that was quite effective on the whole period also another advice that you should always write here is to try to when using begin so you have the same model and you are bugging with different seats try to let your models over fits a little bit by religion them over fits a little bit sometimes the packing will be will increase your score more than trying to find the best number of epochs for


example if you are using neural network when you are using a lot of features try a very small feature fraction so this is one of the parameters of the gradient boosting decision tree this is the number of filters that will be used for each decision tree so most of the time we are using something like 80% or 90% but if you are using a lot of features you should use something or pill or rather try something like 10% or even 5% of the filters so your trees will be really very low correlated they will be almost independent from each other


this will increase significantly your finance ecology some some tips about the neural networks neural networks are doing feature engineering by 5 in 10 so most of the time and I will fill neural networks I don't do any feature engineering I just try to find to come up with a good architecture platform so when you are doing this and in order to increase the feature engineering parts of the neural network try to apply the impedance also on discrete numerical Peter so most of the time we know we only do impedance on categorically


categorical features this help helps in fitter engineering for the knowledge report but to doing that on the street needle computers you will embed those features and you will probably encode more information in those impedance than the initial value such that you are provided with so I have noticed that those impedance of this place inverter filters always increase by final spot in this in order always to another network even if you are working on tabular days they work great in ensembles and they have they are almost as good as great


important decision so when you are doing feature engineering use it for example I GPM and you have lots of teachers that may be used to do feature engineering it's very difficult to tribe or combinations with all teachers you may end up with thousands and thousands of pizza before finding boss that may help you so one thing that's I usually don't I find it quite useful is that I make a neural network and the gradient boosting decision tree I have two scores to baseline scores I remove the same feature from both the


models and I check the difference in the scores if I see that the spot of the neural network have decreased much more than the graduate within decision tree then I know for sure that the neural network while doing some kind of fitter engineering using the Twitter so that helped me decide which feature should I focus on when doing little engineering in writing poster decisions another trick does may use may be used to improve the generalization of your models especially when working with neural network histor and only replaced


some of the filters with newer values okay during the training phase you have some some generator and but for each epoch you replace some values randomly with the newer values most of the times doesn't work but sometimes is may be helpful for the generalization also when working with neural networks and this is this always help when you are working with neural networks when you have a filter a numerical filter with null values always create an additional feature which is just a boolean filter indicating if the corresponding value


doesn't know or not because you are replacing the null values into the original filter with some kind of mean or median or something like that and you need to help your neural network by this additional information so back to the outliers so when you are doing the removal of the outliers from your training set you know that later on you will do some feature engineering and you can you can ask yourself well if I remove those outliers now maybe filter engineering may help me later to predict those outliers color correctly so how


can I decide if the removal of the outlier should be done before or alternate feature engineering so first approach just keep the outliers until you are done with the feat of engineering so you are sure that nothing can help you in predicting the vampires or you can do something that I usually do and that is that is what I did in the second stage of the zero competition is that I made neural network because neural networks are doing feature engineering implicitly okay so if I see that I'm not able to predict with my neural network


my based knowledge network I'm not able to predict suppliers and knowing that neural networks are orange is already doing the feature engineering part that I have some insights that probably feature engineering with rotten garage in person decision tree will probably not be useful to predict guitar players so this helped me to decide if I should remove to our players soon or Lenten okay so I will show you here some case studies some example of competitions and you will see how it's maybe sometimes easy to get a gold medal or even a top


three position so we start with the mercury competition that was a capital carrier competition so you have limited hardware just one hour to run the training and the inference phase of your model so we we were provided with those eyes with those features the item description that was the most important feature the name which was just this short description of the the item category condition shipping and brand name so when we started this competition I was in the team and we were doing the team and we focused on Elijah be a model


because they they were so good we tried some neural networks some rich models and the light GBM was the best model that we can that we could make and even though it was our best model we could not make it in the top ten of the leaderboard so we tried some pencils with neural network they didn't work we even tried to add some impedance coming from neural networks as new features in Nigeria and it didn't work and the running time for alleged GBM was so high that we couldn't add any importance other models because


we are we were limited to one hour of training so if something doesn't work change your approach we were so sad to throw dancing that water that took us so much time to eat to make but we have no choice but dropping that model and come up continue so we tried different models that were pretty fast to train and we used to neural networks and one rich model so one model one neural networks used embeddings of the the words as as input and another model just were used to the one hot encoding as input so we have a sparse


input and some kind of dense in H which work or in vengeance so if you look at the architectures of our robots one model is just some dense layers it straightforward feed-forward neural network with a sparse input which is one half hour one hot in fighting so the count vectorizer here one of the parameters that I forget to put here was binary you can equal as through so the world is seen with the item description Swann otherwise it's you know so it's somehow some kind of one not encoder the second model was using the embedding and the


importance see that had to be done is you know is to use a shared embedding because the name the item description were using the same vocabulary the words have the same meaning so you have to use the same embedding photons details and then we did something that that you can that have been used in the past X which is just try to capture the meaning of the sentence using an average of different different impedance of your sentence of your item description and as you can see there is no competition here just every agent of


embeddings and putting forward in these two dense layer was enough to have a good spot that was our second model and the third model was a simpler each model taken as he put the description in the name filters so in quran 2.1 and we made a cat sound and grin with n equal to 2 and here instead of using the tradition to classical vibrant with a sliding window we use all possible combinations of by grams for the name because the name was so short that we were we could authorize or sell to use those kind of eyebrows and using classical background


and his kind of migrants make the difference in the final concern so why we went for this kind of approach when we notice that the score of each model by itself is not important so even though our light JPM will have the best score well it was not helpful in any ensemble and we note iced by that by Insane plane or light GBM with a rich model the score of the rich model by itself has no importance so we could let out reach over feet or not over feet the score of the rich model give give us the same final performance so we decided


that instead of looking for the performance of each model we should optimize for the performance of the ensemble by Infirmary and we tried to to art and models that were improving the final in fact another thing that have versus let's let in the models of repeat a bit but have have that helped us in the finally movements we also because we have only one hour to run the car locks so instead of decreasing the learning rate after each epoch to epoch we were increasing the batch size there is a nice paper about skis that you can find on the internet


and also removing the mean from the target before the training was quite help second competition this was a solo competition I did just for fun because I was waiting for the zero results so here we have just one Peter which is you have a question and you should predict if the question sensor or instance so this is an NLP problem I started training from scratch I have a baseline score [Music] sorry little when I and you cannot question so training from scotch give your baseline it's obvious that it's not gonna work because the data set


was too small so next I used some embeddings with fine tuning it was helpful but not enough to to get an eye sore again so didn't work what did work is to use the ambient the impedance and don't fine-tune your model so the layers were not trainable for the impedance and this did work so I list the model train for some epochs and fine tuning just the last book was I was helpful in improving the score and then by began back in two different models with different different seats the despoiling moved much more so I think that almost


everyone on capital used this approach so what makes the difference in getting that gold medal so here are the few things that I made and were quite different from other controls so because we are we are not changing our embeddings we are the layers are not trainable so you have to find the maximum impedance of every world you cannot learn impedance for words for new words for rare words so you have to find those embedded so what I did is an iterative pre-processing step if I don't find the word in an embedding I do some


pre-processing and check again if if a building is available and I did this so for example tried lower case and upper case in remove the accents etc and check again if the embedding is available and the last pre-processing step this Damon this is the last thing that I do and if I don't find the the stem of the world Indian bed and then I considered the world as a rare world the nice and this was the most important tips that I have used in the pre-processing and it improved significantly the performance of the model the next thing that I did


is because the impedance of the rail words are not available and I'm not gonna train I'm not gonna learn them because my layers are not trainable so I cannot use them then when I will do some global enumeration or global max so I always multiply those entities by zero otherwise I will use some random values that will not change during the trader this part this was quite helpful in to improving the scroll and later I didn't use the global average chain because I have a lot of zeros embedded in my in my my inputs


because of the padding and because of the rare words that I multiply by zero so I have a lot of zeros and averages would not work so what I need is instead of mid narration I just sum up the values of the intervals then I apply some budget augmentation and here is by our total that's the full network and the spread is straightforward to understand so I have an input which is which is the question by itself and another input which is just a binary indicator telling me if the end if the the world is in the middle or if the


world is a rare world which is not in the ended then I use the the globe in bed in here so I don't initialize my impedance with some randomness that I use a transfer learning to initialize my impedance and then I multiplied this is the tricky part here I'm not supplied by embeddings with the mask so rail words will be multiplied by zero impedance with the equal to zero otherwise I will use the transfer learning then there are two parts here this is the first X approach you just try to capture some kind of meaning of the sentence so usually what


we do is average all the images but the Brazil was not working because I mostly have zeros my sentences so what I did is just some the impedance apply much normalization with my second part of the network is just use an STM or on my input and applying a global maximum and again I cannot use the global average plane I just used this found concatenate everything and straight forward turn the neural network as you can see quite simple but those three little tricks make the difference and I end up just in a few days of work in the 14 place I was


ragged 14 and I got about middle which is which something as simple as that the last competition I will talk about is a recommendation system to or to music so the problem is the following we have a user who is listening to some song but sometimes type t1 and we want to predict if the same user went to read listen to the same song at some time same cheap g2 and t2 minus t1 is less than one mouth so we are trying to predict if there is any releasing of the Sun for each user we have some statistic here but the


important thing to note here is that the contract problem was was extreme here we have almost 26 percent of new songs in the testing data so we have to deal with this false start problem those are the features that were provided and the main the most important feature where the user ID the song ID the suicide and the artist day so does work those are the features I will focus on so what we did so we used here the light GBM water so we computed different statistical features based on the user the song and the artist and we noticed that using


statistic using the target was not working and you will see next okay so the data set didn't include any time stamp or session information but we noticed by doing some analysis that it was coronary ordered so what we need is we approximate sessions of the users by using some little trick just look at the indexes shifted by one try to check the difference if the values are small this means that those samples are in the same session if the value is high this means that this happening in some other sessions this is the first trick that we