that the absence of P chimeric acid stops the up regulation of the cytochrome p450 pathway bees only have 47 cpy genes whereas most insects have 80 and the absence of P chimeric acid coming from fungi turns off their mono oxygenase pathway and so they can't detoxify this accumulation of all these toxins that are become resident as they for a out into the farmer's fields


sprayed with pesticides fungicides and herbicides okay interesting and so then it turns out that when the EPA license many of these fungicides and insecticides and herbicides they didn't look at the consortium of them all coming together and turn out the sub-lethal doses of these toxins defeat the microbiome in the gut of the bee so you have another problem happening here

Mushrooms as Medicine with Paul Stamets at Exponential Medicine


not only is this my oxidase pathway the cytochrome p450s turned off but the microbiome now is being damaged by glycolysis by the way there's one of the big cult culprits some craig Venter hope you're listening so then fungicide contamination in the fields is harming the resident fungi and now we don't have rotting logs we've got agricultural crops may the species of


which are not native okay so beekeepers feed bees sugar water and up to 50 these a 50 percent water 50 percent sugar this is because they need to have the sugar obviously for food and then the bees are truck hundreds of miles in this case the almond and walnut orchards for pollination so the bees are now being fed pure Sugar's as opposed to the complex carbohydrates and polysaccharides was


coming from the sweat of the Weisse ilium so I had an epiphany why don't we take our mycelium and my research team you know it gets credit for this and we came up with Michael honey this is totally made from mycelium it's like 90% sugars but they're complex sugars and guess what it has P chimeric acid in it as the antiviral agents in it as the antibacterial agents so we contacted


Washington State University working with dr. Steve Sheppard and and Brandon Hopkins and we started doing a series of experiments by feeding extracts of the mycelium to bees at different concentrations this is called a stress test they're in captivity they only live 30 days when the worker bees fly out and they're doing a pollination if they don't come back nurse bees are prematurely recruited to


become worker bees and they fly out they abandon the brood so it's a doubling down every time the fewer and fewer worker bees come back nurse bees now have to go out and get pollen and food for the hive so the the larvae and abandoned mites then proliferate mites are injecting viruses into the larva okay so the mushrooms that we're talking about clean the one of my hat Amadou


reishi and chaga are polypore mushrooms and birch forests worldwide apis mellifera the honeybee is from europe it's not native to North America but it produced a four-digit amounts of honey so there's chaga there's Amadou and there's red reishi we provided the bees with twelve different species these are the three ones I'm going to talk about I've never shown this before this this


information just came came in the sugar control in one week's time the virus is increased by 63% when the bees started sipping on the mycelium the viral about pathogen payload plummeted across these three different species in week one versus a week two the sugar control the viruses you know increased dramatically and with with the bees that were taking sips of the mushroom mycelium extract


the vibe is plummeted they went up here and they plummeted down on a dose-dependent basis and so it also occurred with a red reishi so now we're trying to get the right concentrations and the often is obvious now that if we use a combination of these you know what benefit what a benefit will this be but we don't know the way of the B tomorrow the rosetta spaceship


lands on a comet 300 million years 300 million miles out in space well we can find a comet we don't know the way of the B now I've spoken to entomologists about this they spoke through their friends no one's ever mentioned this I spoke at a national mycological Congress I said any mycologist out there this 500 I call just has anyone ever heard of this B's go


no one has bees go to rotted logs because of the immunological benefit increasing their hosts defensive resistance they're complex sugars of nutrition and the antiviral properties I'm the first one to have discovered this really how is that possible we grew up with Winnie the Pooh reading it to our kids they're going after the rotted logs and and and we don't know the way


of the bee I think this says a lot so doctor have Steve Shepherd it was so impressed he provided this wonderful quote as entomologist 39 years of experience I'm unaware of any reports extended the lives of worker bees this is this spread is incredibly important this is a period of high pollen acquisition and so if you increase the workers lifespans by 20% they had a


tremendous effect on a tipping point a tipping point in favor of the colony survival so I suggest to you let's be friendly let's be mushroom the scientists cross disciplines need to work together biodiversity is our biosecurity now think of the bigger picture here we were forced people bees evolved in the forest the mycelium is confirmed confirming an immunological


benefit to to animals but it's unprecedented as far as I know that there is an antiviral agent that is duly active in helping bees and also helping humans and these are from polypore mushrooms are resident in the forest that our ancestors were dependent upon so I want to conclude that humans trees bears mushrooms are all terrestrial organisms that evolved to be interconnected within the mycelial


web of life Earth's natural Internet and I think the way of the future is using mycelial scaffolding with a mutualistic organisms in the bacteria using epigenesis and then being able to have the quorum sensing and there's response and being able to up regulate gene expressions that otherwise may not be present or up regulated with one organism but quorum sensing can give up


regulation of multiple gene sequences otherwise hidden in nature this is the way of life so as much as many of you are ultra specialized I want you to think about the implications of what I'm showing you today we were once forest people for hundreds of millions of years we had force that we're in contact with we're losing the perspective of the synergism and the


[Music] yeah hello everybody my name is Daniel I work for a company called odd phone and we are here today to talk about efficient developers so the first thing that I want to do is define what we mean by efficient to be efficient means achieving maximum productivity without wasting resources and in our case as developers that resource is usually time which means to be efficient is to do

Habits of Efficient Developers


things fast but this definition is missing one key element for us that we can see in this quote by Peter Drucker to be efficient is to do the things right well to be effective is to do the right things so the be efficient is not that we have to be very very fast it's the things that we do we have to do them in a proper way so the decisions that we take today don't slow us down in the future

Habits of Efficient Developers


now that we have this quote here you maybe wonder well what is more important to be efficient or to be effective obviously doesn't make any sense to go really really fast if you end up in the wrong place but equally if we know where we want to go but we never achieve that place it doesn't make sense either in fact there is some synergy between to the between the two if you

Habits of Efficient Developers


are efficient it means that you spend less time takes you less time to do things which means that you have more time in your hands to stop look around and make sure that you're going in the right direction so to be efficient allows you to be more effective now in this talk we are just going to focus on what it takes what it makes us efficient and we're going to stop stop talking

Habits of Efficient Developers


about focus there are plenty of studies they tried to quantify what is the cost of an interruption for us developers and it seems that the cost is around ten or fifteen minutes so every time that somebody comes and interrupts you it take us between ten and fifteen minutes to load the context reload the context of the tasks that we were doing to be able to be productive

Habits of Efficient Developers


so it's fair amount to be efficient to minimize the number of interruptions that we get so we have long periods of time where we can focus on the task at hand there are basically two types of interruptions the ones that you control and the ones that other people control the ones that you control this one's email notifications I probably the worst offenders if you think that that little

Habits of Efficient Developers


pop-up doesn't nothing for your concentration the truth is that for your brain it looks more something like this you cannot just stop looking at it millions of years of evolution have made our brain really sensible to any unexpected movement mostly because the fear of being eaten so when that little pop-up shows up on your screen you have to focus on it you don't have an option so

Habits of Efficient Developers


efficient developers the first thing that they do they disable all notification not just email notifications but absolutely all notifications in fact you don't even want to see that little number with the number of in that the number of unread notifications on your screen because as soon as that little number changes your brain is going to pick up the change and

Habits of Efficient Developers


you're going to start thinking about well what could send my an email or what do I need to do so you should always you can always deal with emails whenever you want whenever you have the time nobody should expect that you'd reply immediately to their emails there are other means of of communication that are more appropriate if something is really really urgent emails are asynchronous

Habits of Efficient Developers


and it's more efficient to deal with them in batches the only notification that you want to say is the one that tells you that you broke the build and please never be one of these guys that they type an email and they want to tell you just to make sure that you received an email this is really really annoying and this brings us to the other types of interruptions

Habits of Efficient Developers


they want that you don't control what can you do with somebody comes to your desk and interrupt your flow I know three possible options the first one is to wear some headphones really really big so when somebody comes you pretend that you didn't see him and you hope that he will just walk a row we'll walk away the second option is that you have a very good team late

Habits of Efficient Developers


somebody like this guy somebody that is able to tackle any interruption before it reaches the team the third option is to do peripheral grameen if you are doing pair programming when somebody comes to interrupt you what should happen is that one of the two developers in the pair so stand up walk away a couple of meters and deal with the interruption and when he's done he goes

Habits of Efficient Developers


back to the other pair and that all the guy that was able to keep focus it works like a really fast cash to get him into a productive State a lot faster the additional benefit of pair programming is that because you have another developer looking at what you are doing all the time you are not going to check the news or your phone or or your email as often so that peer pressure is just

Habits of Efficient Developers


going to cause you to be more efficient I don't think I need to explain this right we all know this and I don't like to sound like your mom when she tells you to get your greens so let's move to the next one efficient developers just do one thing at a time and the reason is exactly the same why we hate interruptions yeah because of the context switches here we see that doing

Habits of Efficient Developers


the blue tasks after doing finis in the green tasks it just takes less time in fact it is not just that it takes less time when you try to do multiple things at the same time the quality of your work usually suffers we all know that the definition of multitasking is just screwing several things at the same time so you should always focus on one thing if you finish and then you move to the

Habits of Efficient Developers


next task you may we are all going to spend thousands upon thousands of hours in front of your ID it's one of our main tools so you really need to know it inside out because any efficiency that you can use in your IDE is going to be multiplied but those are thousands of hours that you are going to spend front of it yes basically need to know two things its functionality and its or

Habits of Efficient Developers


cuts now just because you are sitting already in front of it for six hours a day it doesn't mean that you are going to master it to master your ID you have to make a conscious and deliberate effort to learn it to find what functionality you don't know about it you can't read the release notes you can follow blogs or YouTube channels of people using it or you can just do pair

Habits of Efficient Developers


programming when you do pair programming each of your partners is going to show you how they use the ID and it's going to show you functionality that you didn't know about or it's going to show you more efficient ways of doing some tasks also you can teach him your tips and tricks so you make the whole team more efficient I'm always surprised about the amount of manual work that

Habits of Efficient Developers


with developers can put up with and I found it very paradoxical given that we are paid to automate all these people's job manual work is not just slower to do but it's dull boring and it's error problem so I'll always wonder well why we keep doing it in one of the main reason is that we sometimes for God that we are developers and as developers we have this very rare and powerful skill

Habits of Efficient Developers


that allow us to create an army of minions that will do as we say they will not complain it they never get tired and they do it really really fast and I think we don't use this skill often enough sometimes it may be because we just end up with this kind of minions but that's a different talk just to make sure that these things I'm going to repeat it again you are developers you

Habits of Efficient Developers


don't do things that a computer that your computer can do for you so efficient developers before starting any task they always think well can I write a program to do the task or at least to help me to do the task and I'm not just talking about automating some work that can take you hours I'm also talking about automating tasks that can take you five seconds but you

Habits of Efficient Developers


do several times a day and I am so talking about driving programs for one of tasks that you're never going to do again first because maybe is more efficient you can do it faster but second because I hope that writing programs is fun this is fun for me more finalists and doing things manually to write simple programs there is nothing like all good but she'll have

Habits of Efficient Developers


been a developer for eighteen years and during those 18 years I have changed operating systems I have changed programming languages I have changed IDs I have changed my mind up about all kinds of ideas and practices the only thing that has been constant during those 18 years has been bash so I want to show a little demo about what bash can do for you the beginning of picking off in the fall

Habits of Efficient Developers


I didn't cope so let's say that your business manager tells you that some other some other team has created a program and that program is collecting news and he wants to know how many news per country we are we have so you jump into the box and the first thing that you are going to do is write a program this is a program that is going to tell you how many news we have this is a program

Habits of Efficient Developers


right from 4000 so now let's see what we have in one of those news so it seems that it's a piece of JSON right but kind of hard to read so we format it so we see here some news with an ID some social stats the site that we come from and we have here the country yeah so what we are going to do is write a little program to extract that country so country code quote single quote one

Habits of Efficient Developers


single code okay let's see if that works yep so we see always make mouse here the country so now let's try to extract it we start doing grab - oh we get something and now we want to get the whole line so so you don't get too lost here it is our result country and then we have the u.s. so what we are going to do is now split the line with a split line with a delimiter of quote and one day feel I

Habits of Efficient Developers


think is the threat of the fourth is the fourth feel cool so well now what we have it's a program to strike the news they the country for news so we just want to do the same for all the news and now we just need to sort it and count it this because we want to sort it again so we sit in order so we get that for this country there are around three thousand years but now that you have this program

Habits of Efficient Developers


let's say that instead of the country we're interested on the side yeah we see there number of news per side but now your business manager comes inside like well I want to know what are the top three website for each country so we're just going to modify this program a little bit we grab country let's try for example I L and if I didn't get that wrong does that all the countries so we

Habits of Efficient Developers


just want we just want to sort them and take the top three that's it now we are going to format it so it all have chosen one line so now we have a program that for one given country it tell us the top three so we just need to do a for loop right we just need to find all the countries that we can get basically from our previous program and we just need to write a for loop for in this two we echo

Habits of Efficient Developers


the name of the country and we finished the for-loop Papa Papa back instead of IL is the country that we want to know about and because this is bash I need to quote quote quote quote unquote and it got something wrong did I miss anything mmm it's a new error site what what it doesn't matter so I'm going to then move to the next thing that is whenever you are writing a

Habits of Efficient Developers


program you should always time limit the amount of time that you are expected to use on it this is a good example right because if I keep going I will spend the next 25 minutes just trying to get your work in so whenever you are trying to write a program or Eltham it a task the first thing that you should do it's time limited and if after paint that time limit you are not able to finish it just

Habits of Efficient Developers


move and do it things manually right now even if you think that that's a waste of time right I tried to for five minutes to get this thing working and it didn't get it well the truth is that you have learned a little bit that's not waste time that's invested time on you learning and getting better okay so whatever yep there is a nice table from a CD that tells you how much time you can expend

Habits of Efficient Developers


to automate some tasks so have a look at that and if we are talking about pro writing programs what you should always do it's about you will want about graphical user interfaces why because you cannot put a UI inside a for loop you guys don't compost they just live in their little world now I'm not saying that you should never use them because they are extremely useful

Habits of Efficient Developers


when you are getting started when you are learning something new but once you are past that phase of beginner you we'll actually want to do more complex stuff and you guys just constrain what you what you can do and if we're talking about a body in your eyes the first way that you want to avoid is your own applications you are right there is nothing less efficient than starting

Habits of Efficient Developers


application clicking things around and filling up forms to know if you're featured the new feature is working or if you broke anything a part of making this a more efficient automated test also give us the give us the confidence to refactor and change code because it's going to catch bugs and bugs are the worst time waste of all first you need to write that the bug then somebody has

Habits of Efficient Developers


to review the back then you need to put the back into production and then by the time some user notice the bug you have gone through this massive context which because you probably wrote the bug several weeks ago and so even if you wrote the code that has debug the code is already alien for you and you have to dig into it and then you need to fix it you need to go review it

Habits of Efficient Developers


you need to explain it to your boss you need to fill some JIRA issues and then you need to go again through all the release process so bugs are just a big waste of thing but worse than a bug it's having the same bike twice right so whenever you go and fix a bug the first thing that you should do is write a test to prove that you are able to reproduce the bug you see it fail and then you fix

Habits of Efficient Developers


it and the last thing that you want to avoid to do manually is setting up the development environment right this is not going to make it just more efficient but the whole team more efficient this is how distractions for any project that I joined look like from my point of view and the only thing clear on them about them is they're going to they are not going to work

Habits of Efficient Developers


maybe they are missing some step or they're not precise enough or maybe I will make some silly mistake when I try to follow them and the result is always the same two three four days of wasted time what you want to achieve its instructions as close as possible to this just one command and that one command should bring all the tools and configure them to be able to build run

Habits of Efficient Developers


and test your application if you need a database it should install the database and configure it and seed it with some data if you need any bill to maven NPM whatever it will download the correct version of maven and install it and configure any SDK that you need as you can see my tool of choice right now to do this it's docker compose which is part of the toker suite if you are not familiar with

Habits of Efficient Developers


it it looks this an example and here we are think that our development environment in its three containers Postgres DP Redis DP and our own application this has multiple benefits right first thing it takes use minutes for somebody new to get started but also if something stops working on your development environment you can just easily just wipe the whole thing and

Habits of Efficient Developers


start again if there any change on the development environment you know share immediately with the whole team and these structures never get out of date also because docker is running things in isolated environments it means that if two projects that you're working on they use completely different versions of a database or a JD SDK well they're going to be completely isolated so it doesn't

Habits of Efficient Developers


bother one and the other and also because it's so easy to make changes it allows you it encourage you to experiment if you want to try a new JDK or SDK or a new version of the database just make the changes started and if you don't like it you just completely what the whole environment and the last section that we are going to talk about feedback doesn't matter what you were what you

Habits of Efficient Developers


are working on you should always try to find the shortest and tightest feedback loop possible feedback is what it tell us if we are going in the right direction feedback make us at the same time more efficient and more effective you want feedback often and early to make sure that you don't wander on the wrong path for too long with the consequent waste of time

Habits of Efficient Developers


an energy if we talk about the benefits of automated test yeah we save time give us save it with it catches bugs allow us to the factor one is the best moment to try test well my opinion is before you start doing any coding if you're not familiar with the TDD workflow it's basically this going to go really fast through it you first write one test and only one test you run it you see it fail

Habits of Efficient Developers


you see it right and then just write enough code to make that test pass and then you refactor you clean up your code running the test just to make sure that you didn't break anything there are least four reasons why you want to use the this workflow the first one is the fast feed but that gives you as you are building the new feature to note that your code is doing what you

Habits of Efficient Developers


spread it to do the second reason is that if you truly believe that automated test saves you time you want that benefit as soon as possible as you are developing the new feature the third reason is organizational I have here years too many times the phrase I don't have time to write this or I'm not given the time to write this and for me just actually means that well I always write

Habits of Efficient Developers


my code I finish my feature and once I finish my features when I do try my test and if there is any time pressure well you know I'm not going to I don't get time to write those tests and because you don't write tests it means that you don't refactor your code because to refactor code you need a very good automated persuade and because you don't know factor code it means that your code

Habits of Efficient Developers


starts to accumulate garbage and because your code starts to accumulate garbage it takes you a little more time to actually build new features and because it takes you more time to build features you get more time pressure and you add more time pressure so you have less time to write test closing a vicious circle cycle that always end up with the same with us developers crying for our right

Habits of Efficient Developers


and the four reason why you want to write your test first it's one of a mechanical reason because seen a test fail is the test that tests that the test test what is supposed to test or in simple words how do you know that your test doesn't have any bug if you write a test and you see it right there is a strong indication that is some piece of production code some logic

Habits of Efficient Developers


that is not there if you write the test and you never say bread what you don't know if it is because you're ready implemented a feature or because you forget on a certain in your test or the setup code is not correct now when you present this idea to a lot of people they always come up with this phrase I can't write a test first because I don't know what I'm going to

Habits of Efficient Developers


build and this can mean different things it can mean that you don't understand what business is asking you to do right and in this case it's true you cannot write any test but you cannot write any production code either what you have to do is go back to business and ask for clarification what do you want to do the other case is that you actually understand business and you're too early

Habits of Efficient Developers


understand the logic that you need to build but you don't know if you are going to write one class or ten classes or if you are going to put an if statement or a switch or a factory factory factory you don't know what you're going to do right but you know understand the logic and you know understand the mechanics of the side effects so you know which database you

Habits of Efficient Developers


are going to use you have juicy ten thousand times already you know the table you know everything in all these cases you can actually run a test first but it's true that sometimes we actually don't know what how to do the side effects that we are asked for we for example maybe the logic for your new application functionality in its it needs to cause some restful endpoint to

Habits of Efficient Developers


get some for exchange and you have never used it and you don't know the end point and you don't know what you need to give to it and you don't know what it's going to give you about or maybe you need to consume some messages from a message queue and you have never done that so you don't know which libraries to use you don't know how they work in all those cases you don't really know what

Habits of Efficient Developers


you need to what you are going to do there is always this face of exploration that we have in our in our job that is that we used to fill up those gaps to convert known side-effects into known side-effects and that's something that TDD doesn't help you with what you want to do it first read the documentation to see if you are able to fill those gaps and the second thing you

Habits of Efficient Developers


want to write a lot of little programs to test to play around with that technology for this the best tool that I know it's a rebel rebel stands for read eval print loop and it's just basically a fancy way of saying that you have like a common line interface inside your application starts off trying to explain it let's see it in action if it works this time so have already start an

Habits of Efficient Developers


application with a rebel inside and what I'm going to do it from my IDE I'm going to connect to that rebel so let's say that you didn't know how the plus function works so I'd write a piece of code in my IDE and I'm not should using a shortcut to send that piece of code to the application and the application tells me that 2 plus 3 is 5 yep so it writes the code on the top

Habits of Efficient Developers


screen and I get the result on the bottom screen so as as I was saying this allows you to experiment with the library so maybe what happens if I pass three parameters seems to work what happens if I pass a very big number I get an exception what happens if I just pass one parameter it works no parameters it works so this is just understanding how they are how the library works and this

Habits of Efficient Developers


could be an HCP library messaging library some concurrency library your were just writing little programs and executing them to see what's the result let's do something slightly more fancy let's say that your business manager it tells you that you have to build a new feature and you know it some for exchange for that feature and one of your mates told you that there is a

Habits of Efficient Developers


restful endpoint to do that and it gives you the URL so we're all an HTP we make an estimate request and we see that we are getting some exception let try to format that sketched exception try/catch exception okay try it again so it tell us it's a 400 which means it's our fault and we see here somebody so what we are going to do is let's get the body okay that seems some piece of JSON so let's

Habits of Efficient Developers


parse the JSON there it is so it seems that we're missing some date some query parameters query parameters yeah we get these change rates what happens if I pass all and older what happens if I pass something in the future I'll still returns data that sorry to be worried what happens if I pass a string I get that error so what we are doing is proving how the real world works and how

Habits of Efficient Developers


we are doing it we write a little program we run it we see the result so it's a very very fast feedback cycle now you may be when the world why don't you use something like postman to do this right it's just xdp restful must be postman well there's somebody feats of doing it like this way the first thing is you have a full language this is a production language the one that they

Habits of Efficient Developers


use in production which means that I can do four loops and if statements if I want to mix this data with something from the database well I know how to make database calls from the JVM and also if I now go and I grab this exploratory code inside a function this that you see here this is production code as you see it there is going to go to production I'm making the changes in

Habits of Efficient Developers


the project this is not a different tool that then I need to get what I got from the tool and translated to Java or.net or whatever you are using this is production code it's ready to go also because the repple is running inside your running application you can actually go and poke at the state you can look at the state of your running application and what we are doing here if you notice

Habits of Efficient Developers


we are modifying or running application and we are doing all of this without having to compile or restart anything that's a very very quick feedback loop and I don't know if I mention it but we are connecting to this rebel through a socket and because we are connecting through a circuit it means that we don't really need to be running this process in my local box it can be running tests

Habits of Efficient Developers


or production so you could be suspecting modifying adding log statements into production code as without stopping the application this is extremely power powerful and you know with great power power comes great responsibility so use it with care the last thing that we are going to talk about it's code reviews code reviews tell us if the design of the code that

Habits of Efficient Developers


we are doing if it fits the application it allows other other one of your your teammates to tell you if you have any bugs and it also we can use it to share knowledge right it's a way of sharing knowledge so efficient developers want that code to be code review now there is something very truth behind code reviews when we are presented with this huge massive changes I don't know what your

Habits of Efficient Developers


reaction but my reaction is something like oh my god yep when we get those but when we get small changes we are able to give useful feedback to the alpha of all the of the change right because we are able to understand the change also even if you are very disciplined developer and you go through that really painful review process in my experience what does it

Habits of Efficient Developers


happen when you go and tell the other like well you know I think your design is your no no no your sorry it's gonna improve your design or we could use a different library that will save us some time or some resources or whatever what usually happens tell whoever say like yeah I think you are right but you know I have already spent like several days or weeks working on

Habits of Efficient Developers


this and the end of the Sprint is tomorrow so even if I think you're right I don't think I'm going to have time to do your change what you're suggesting because it's going to take me several more days to do it also you know it's already working so let's do something different let's just commit the change as it is and we are going to ask the product owner to create a refactoring story I'm sure he

Habits of Efficient Developers


will be delighted to put it on top of the priority queue I will know that those things never happen so you end up again with words code that leads to this lower implemented feature with plaplapla blah blah blah so efficient developers they don't want just code reviews they want small and early code reviews so what they actually want discontinues co reviews this practice consists on on

Habits of Efficient Developers


getting one of your teammates to sit just beside you and as you are implementing the future this this developer sitting beside you is going to suggest improvements on your code it's going to be catching bugs that you are doing and for their for the reviewer the changes are really really small yeah as you type them he see those those changes and for you as the author you

Habits of Efficient Developers


can get feedback even before you start writing any code additionally if for whatever reason you cannot able to finish the feature this other developer it's able to pick up that feature without any effort because he has been behind each of your decisions so you avoid those knowledge silos within the team also this this other developer can work as your personal stack overflow

Habits of Efficient Developers


because maybe he has already found that similar issue and he already knows how to fix it and sometimes you don't even need to ask the question because he sees what you are doing some people call also this program so that's all that I have very briefly focus master your ID your tools are both manual work and find yourself the fastest feedback loop possible and last words you should always find

Habits of Efficient Developers


time to stop reflect on how you are working and never ever stop learning thank you very much thank you so much for your tips I will definitely start with the notifications part tomorrow we got a lot of questions during your talk so let's start with the first one how do we balance avoiding interruption with work in small dynamic teams these need rapid feedback loops

Habits of Efficient Developers


and frequent communication okay if your work is if your team is really small and if you are doing pair programming your team becomes tiny yeah and because the thing becomes tiny it means that that need of communication the number of not ages on day on the communication graph it just reduces so try to pair programming Thanks where can we find your slides everybody

Habits of Efficient Developers


wants Terry my laptop if somebody wants to grab my laptop the other can somebody come and pick up I would polish them in my personal blog that probably none of you great I don't know if they I will tweet it I will treat them now and would put them in in someplace that you can find it that would be great additionally there are recordings of all sessions so you can watch them later or

Habits of Efficient Developers


sent the links to your colleagues another question how can we efficiently automate ourselves out of the job it depends if you work for yourself that this is the best thing that you can do yeah this is it free money I think it's it depends on your ethics right if you can actually if you think that you can actually automate your work why not write you finish people that are using

Habits of Efficient Developers


that tool or your business will be very thankful and you know we have plenty of jobs around the world so don't worry about your job there is a better world job working for you okay I think it's time for our last question what's the worst distraction for a developer and it's it's like well why is it slick I don't think it is luck to be honest I think the worst distraction if

Habits of Efficient Developers


you have a two-year-old that is knocking on your door and you work from home I think that worse but I don't think I'm a remote worker most of the advice that I give you it's when I was not a remote worker and we used to slack a lot and I just mute everybody but they know that is a really neat Mian they'd really need to reach me they know how to get hold of me so I don't know what we feel offended

Habits of Efficient Developers


transformers are quickly coming for your favorite models yesterday they replaced lstms in nlp they used to be good at nlp but blah we now have transformers think again today we're going to see that maybe in the near future transformers will replace convolutions in image processing so this paper is a step in toward towards this direction you just wonder what is it going to be

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (Paper Explained)


tomorrow maybe linear regression is going to be replaced just by giant transformers trained on 5 000 tpus uh who knows we'll see in any case we're looking at axial deep lab stand-alone axial attention for pan-optic segmentation by hui wang yukon chu bradley green heart week adam alan yul and liang chia chen of john hopkins university and google research so this paper


combines a bunch of techniques that have been introduced recently uh to deal with attention in problems where you would traditionally use a convolution so in this particular case they deal with this problem of pan-optic segmentation which basically you'll see you'll get an image and there's a bunch of stuff on the image like a cat here and a house right here and you're supposed to color


the pixels of the same object the same so you see you see all these pixels here are house and then all these pixels these pixels right here are cat and so on and then there's also the background so all these pixels right here i know beautiful beautiful beautiful our background so for this problem um it's kind of important that there you you you're very precise


first of all so you can look at you know pixels or clusters of pixels and also that you take long-range dependencies into account because if you for example recognize that this is a house and you recognize that here's a wall right here um you might be able to much better classify what is wall over here and what isn't okay so the kind of long-range dependencies play


a role in these problems across images and usually attention mechanisms are pretty good for these long-range dependencies but they're also expensive and that's what this paper deals with so they use this axial attention that has been introduced for exactly resolving this problem in types of data like images or higher order tensors and they also combine this together with


learned positional encodings which we've also seen um time and time again throughout the kind of transformer and attention literature so the combination of axial attention these learn positional embeddings allows them to replace the resnet backbone that usually is found in panoptix segmentation models with the with a standalone attention so they build models that are partial


replace the convolutions with attention modules or replace them entirely so the entire model is going to be just an attention model so no more convolutions in it and they perform pretty well in classic tasks like they they test on imagenet classification they perform pretty well and they achieve state-of-the-art on some of these segmentation tasks so we'll go through the model right here


this is a very very extensive paper in terms of experimental evaluation what i want to get into is mainly how the method works um and show you what their model looks like so we'll go through it and as always let me know what you think in the comments and tell me if you liked it or not uh share it out if you did all right so they go over a very long list of prior work which is you know


pretty pretty cool and here they say their contributions so their contributions are four fold first of all the proposed method is the first attempt to build standalone attention models with larger large or a global receptive field and we'll see what that means we propose position sensitive attention layer that makes better use of positional information without adding


much computational cost we show that axial attention works well not only as a standalone model on image classification but also as a backbone on pan-optic segmentation instant segmentation and semantic segmentation maybe what i did before described before was instance or semantic segmentation and not panoptix segmentation excuse me if that's the case as you can


see it can be used for various various image tasks lastly our axial deep lab improves significantly over bottom-up state-of-the-art on cocoa achieving comparable performance of two-stage methods we also surpassed the previous state-of-the-art methods on mapillary vistas and cityscapes so these are various tasks as i said and also what they don't mention here is


that they perform fairly well on imagenet in fact in the abstract they formulate this as um in particular our model outperforms all existing standalone self attention models on imagenet like that's you know that's a way to phrase it uh you just exclude all of the other models until you're the best outperforms all existing standalone self-attention models on imagenet yeah


i mean that's good i i mean there's something to be said of comparing apples to apples but you can also you can also go overboard if you want to make your work look as good as possible of course you know everyone everyone does that and there's no particular shame in it okay so if a we're going to build up our model right here and the basic element of this um


model is going to be this self-attention mechanism now quickly because i know you all know what it is but very quickly you want to perform this action right here over a region right here so there is always a query and now the subscripts here are going to be important in this paper okay so the query is at a given position position o and you can see that's the o


right here that's the i'm going to call it the output i guess that's what they said as well so the output position you want to go over all of the input positions and you want to aggregate data from all of the input positions so that's right here and how do you aggregate data by this softmax operator right here and you can see the key also has a p right here and the softmax is over the


axis of p so in particular case of the images what does that mean if you have an image right here it's made into pixels okay so you have pixels now a transformer or gen in generally these attention models what you can imagine is they always transform a data point into a data point of the same dimensions now this doesn't have to be actually and i think one of the


developments that is going to come in coming years or months or weeks maybe someone's already doing it is in fact to play more with this with this arbitrary constraint that we're imposing on ourselves because it's not really clear that this is the best thing but for now an attention layer is always transforming a data point here a four by four image into a data point of the same size also


a four by four image right here now this is as i said this is quite simplified but it is true in nlp where we always transform our whatever 512 sequence token sequence into a 512 token sequence and it is true here now the output is is going to be here on the right and the question always is okay so i'll go over these um these these pixels right here and for every pixel


let's say for this pixel i'm going to ask what data goes there what's the output of the layer at that particular pixel and the output of the layer is going to be somehow dependent on on the input right here now if you know classic convolutional models what the classic convolutional model says the output of this is going to be dependent on this region right here if it's like a


three by three filter okay so you have this convolutional filter and that means that blue dot on the right is going to pay attention to you know its own location in the input plus everything around it okay and then every single uh data point here is going to do that so for example this green data point is going to pay attention to this region right here now there's a


border um so there's maybe some padding but the question is always where does the information come from and how is it aggregated okay in a convolutional layer what happens in a convolution layer in a convolution layer you simply have your filter right you have your filter and the filter has numbers in it like three and five and eight and so on and what you're going to do is you're


going to take this region right here this blue region of the lower layer and that's maybe that's also you know filled with numbers like seven what's a good number zero zero is a purple is a good nice number and you're going to multiply those and then you're going to sum them up and then you're going to put that on where the blue dot is okay so where does the information come


from in the convolution from around the location from around the output location but in the input okay so you go to the input at the same location as where you want the output to be you take the neighborhood and there is a fixed a fixed scheme of aggregating the neighborhood okay and then you sum you multiply and you sum across it in contrast to this in a uh fully attentional


model where does the information come from let's again look at the blue dot and let's consider it fully attentional okay where does the information come from everywhere anywhere anywhere at all okay the information comes from everywhere now how how do i know um how to aggregate the information so it's no longer in a neighborhood how do i know how to aggregate the information


that's also different so two things are different um now in a convolution i would have another four by four grid here that's pre-specified but in the attention model this here is basically all filled with question marks question mark question mark where how what what number goes here how do i in the end i also do this multiply and i sum it up and i put it right here


okay but how do these numbers come to be well these numbers also come these are dynamically computed also from um from the input it's a bit special but this is how attention works okay so every pixel gets to decide where information comes from and how it is aggregated it basically it comes from anywhere and how it is aggregated is dynamic depending on the pixel


if you don't still don't understand it maybe pay out to watch a video on attention itself i happen to have made one but you can watch any one when you understand that you will understand the um the extension here to the image is the exact same thing as with the sequence except uh the pixels are basically one long sequence in the image okay so this would be a fully


attentional model down here now what's the problem here the problem is that pictures are pretty large so even even something like mnist which is like 28 by 28 is like 700 pixels plus i don't remember exactly but it's like about 700 pixels and our big transformers now so bert a very famous transformer takes inputs uh that are like 512 in length and you already need pretty decent hardware


to run this and the requirements on memory and compute scale quadratically with the input length so already with mnist you're in pretty pretty shady territory um if you go up to something like imagenet which is like 225 by 225 you're that's bad right that's not good um so you have to come up with something else so people have been playing around the reason why i introduced it this way


is people have been playing around a bit with sort of coming up with an intermediate with a compromise between the two so the compromise that this paper here focuses on is going to be is going to be a compromise where we you remember when i said where does the information for a given pixel come from and we said okay it can come from anywhere in the attention framework and that's


good because that allows us to make super long-range connections so any pixel can aggregate information from any other pixel and not even in a fixed way but in a dynamic way so depending on the pixel value itself and the other values it can it decide how it wants to aggregate information that turns out to be expensive right every pixel together with every pixel


well that's quadratic okay so what do we do we make a third method that's going to be a compromise and the compromise is going to be the following the compromise is going to be all right we still do the dynamic aggregation which means that we still do the attention thing however however we're going to restrict back to this neighborhood region of the convolution


so in this model where this information for the blue dot come from it again comes from this neighborhood right here and this number the size here is going to be called m so it still comes from that m by m neighborhood so a pixel can only aggregate information from its neighbors but contrary to a convolution how it aggregates the information like this what in convolution would be a


kernel the kernel is made dynamically by the attention module and it's made dynamically on a case-by-case basis okay so we restrict it to a neighborhood multiply sum it up and then put it into the output and we do that for every pixel now it resembles much more a convolution simply a convolution with this dynamic with this dynamic matrix right here and that's the starting point


for this paper so this paper does two things to this it says okay um we can augment this by so-called positional embeddings uh a positional embedding you might know from the sequence transformers so if i have a sequence my cat is tall i don't even know what that means for a cat but okay what in a positional encoding so if you use a transformer and you transform this as we said into a


sequence of equal length and then transform is basically information routing the transformer simply sees the lower layer sequence as a set not as a sequence it has no notion of what's neighboring to what what comes from where so it pays to tell the transformer by the way this is word one this is word two this is word three this is word four there are various


ways to do it um transformers usually have fairly complicated kind of sine wave based positional encodings that bring many advantages with them um in this case they say well it might pay a payoff to learn where actually these things are in this neighborhood so they experiment with relative positional encoding which means they annotate this neighborhood with something like


look here in the middle it's a zero zero here it's like zero one here it's zero negative one negative one zero and so on so they annotate it with uh these positional encodings now this is this would be the easy way what they actually do is they simply they give the model a matrix like this and they learn that matrix by heart let's say um so the positional encodings are relative


positional encodings and they are learned okay so you can do that you can learn position on coding so if you don't want to do the one two three four right here you simply say well here is a vector here is a vector here is a vector and here is also a vector now model you're already learning like all the ways to make this thing here happen and you're already learning your


output weights up here right using back propagation why don't you learn yourself what you would like for position one like what kind of information you would like to be to have there using back propagation right so the model you provide them you always provide the same vector so this is the same vector for position one and you have a different vector for position two and


you have a different vector for position three right so um but across all of the data points these vectors are going to be the same so the vector one is always going to be that same vector for all of the data points so the model somehow must learn independent of the data point what it means to be in position one so the model must learn how it wants to fill that vector that's called a learned


positional embeddings we've seen this in many models so far it usually works pretty well and i guess here it works especially well if you have these relative positional encodings and so this thing here is not going to be an actual matrix filled with these numbers it's going to be a learned matrix a trainable matrix that is filled that the network is allowed to fill


with numbers right like three five eight and you might be you might notice that we've seen this before right um so ultimately the information in this blue thing right here is going to depend on this dynamically created aggregating of information through the neighborhood and this statically learned aggregation of information throughout the neighborhood which is a con which is sort of a


convolution right um because in the convolution you've already seen here this is a statically learned map of how to aggregate information from the neighborhood of a pixel so i think even though there are slight differences um they for example say this these are the same across attention heads and so on um however i suspect that you you can think of these learned positional embeddings


to be um to be kind of like what you learn in a convolution not exactly so um no i i think i made a mistake and we'll see it in the formula we'll see it in the formula yeah okay so here they introduce these positional embeddings okay so you see that we previously we had the soft max previously we had this and this okay so this is the lower layer this is the information


that comes into the layer and now it's it's transformed into values by a linear matrix but essentially this is the lower layer and for each of the output locations you want to know how should i aggregate information from that lower layer and you do this by this thing here this thing here is this dynamically constructed attention matrix using also the softmax


okay so how should you aggregate information this comes from this query at the output position and the keys at the input position and now you add to that this method this thing right here which is again an inner product between the carrier query and the positional encodings okay so the positional encodings are going to be learned and hard coded but they still are


modified by the queries so the query can still pay attention the difference is the keys depend on the input while the positional encoding does not depend on the input so the queries can decide i want to gather information from this and this and this type of information so that would be the key or it can decide i would like very much to look at pixels that are


somehow on the bottom right of the pixel that i am now that would be the um positional encodings and that's that's the mistake i made when i said it's equivalent to a convolution it is not because the query can still it's still modulated by that query vector um of how to aggregate information otherwise you would have this to be a standalone multiplied by the input right here but it sort of


pays off to think of it like what you do in the convolution so in the convolution you learn how to aggregate information basically based on on position um relative position to the position that you want to output and here you do a similar thing you learn static position embeddings that you then can attend to with your queries all right so these are the position


embeddings and they make use of those position embeddings in fact they attend them to the following in this work we enable the output to retrieve relative positions beside the content based on query key affinities formally so the problem up here is that okay you have these position embeddings um and here are the outputs but if you do this in multiple layers right if you do let's let's go


with 1d sequences if you do this in multiple layers and here you annotate the position let's just go one two three four um and okay this layer can make use of that right we gather stuff from here but then when this layer when this layer gathers information from here the where the information comes from in the layer below is some is how somehow getting lost right so it cannot kind of pull


through this information to here or at least it's very complicated this model extends this positional embeddings in order to pull through that information so as you can see there are two new things right here the biggest important new thing is that right here we don't so here is how we aggregate information okay and here is the information that we aggregate over


now you can see previously this was just this value vector and now it is extended to the position to positional embeddings learned positional embeddings okay so the this with this you're able to route the positional embeddings to the output and also here you can see the attention gets fairly complex so you have query key attention which is classic attention the queries can attend to positional


codings but also the keys can attend to positional encodings so not only can uh not only can the the node on top say i would like to attend to position three um position three can also say well together with me uh positions two and four are are fairly important i guess that's what that's what that is maybe i'm mistaken here but you can see right here there is an interaction


between the keys and the positional encoding right here now these position encodings they are different for the queries keys and values but um ultimately we don't it doesn't make too much of a difference so here is a contrast between what a traditional attention layer would do and what they would do so a traditional attention layer gets the input x and transforms it by means of these linear


transformations right here into the queries these are the queries let's call them q into the keys and into the values okay then it does a matrix multiplication with the keys and the queries and puts that through a softmax so this here is going to be our attention matrix this is the attention matrix and the attention matrix is multiplied here by the values and that determines our


output okay again the attention matrix defines how we aggregate information and the values is what information do we aggregate you know for the output in contrast when we introduce these positional encodings you can see right here again we have query key and value now it gets a little bit more more more complex right here namely we do this query key multiplication right here but


we also multiply the query by these uh positional embeddings for q we also multiply the keys by the positional embeddings for k and all of this together so this is a big plus right here all of this together is routed through the softmax okay and now the diagram is a little bit complicated uh now you can see the softmax aggregates information from here and from this learn position


embeddings i would rather have they would just use it like they did in the formula uh do v plus r and say that's going to be the information that we are aggregating and the soft max here the output of the softmax is going to be how we aggregate information this is the attention all right i hope that's sort of clear you introduce these positional embeddings for queries keys and values


and that allows the model to have a sense of where the information is coming from basically what positions which if you drop the convolutions so the convolution had this intrinsically because in your convolutional kernel right uh can i i'm i'm dumb if in your convolutional kernel the number right here if there was a seven right here that meant that wherever you are


whatever is on the bottom right is seven important okay so that's that was the the convolution had this intrinsically here if you just do attention we as humans we see it in a in this kind of grid form but the machine doesn't the machine simply sees a set of pixels it simply sees you can this is to the attention mechanism this is exactly the same as a long list of pixels or a


discontinued set it doesn't matter to the machine so it's like the problems a feed forward network has so we need to annotate it we have to give it positional information and learned positional information seems to work very well right here though you could think of static positional information okay this is the first thing the positional embeddings um that now help the attention mechanism


see where the information is coming from that's really important in pictures uh so we add that the second thing they do is this so-called axial attention now axial attention is a sort of a let's say a trick in order to reduce the [Music] load on a the load on an attention mechanism so what does it mean we've already we've already seen in sequences right if i have a sequence


a sequence layer that's going to be n squared connections between the two now there are various ways to restrict that so instead of having all of these connections let's say from onenote we've already seen wait if we just restrict it to let's say only this thing right here only this stuff that can be that is lower right that is lower in complexity and this in this case it would be just a


neighborhood so that's what we've done that's this this m thing right here however we can also do it in different ways since this is a set anyway we can simply say uh maybe we should just always skip one we could like do attention like this and that would be just fine too right that would also leave away some of the information but you gain in computational efficiency there are


various trade-offs now in a picture you have the same options right so you can do the neighborhood thing as we did or you can say where should the green pixel pay attention to axial attention says the green pixel should pay attention to only the row where it is in okay that's it should ignore the rest of the input it should only pay attention to that row where it is in and then in


the next layer we'll flip it then the green pixel the same green pixel will pay attention to only the column it is in okay so that's that's called axial attention but don't think like don't don't there is nothing special about this being an axis or whatnot you could also define and it would not be called axial attention but you could define it it makes the same sense to say well that green pixel


just depends on this diagonal right here just in the in this layer it just does this diagonal and then in the next layer it does like the anti-diagonal um you can say uh i just choose five random pixels in this layer and five random pixels in the next layer and that would work as well we've already seen this in this paper called big bird right the big big big bird but big


bird so big bird explicitly used random connections in the attention mechanism and their argument was well if we use different random connections in each layer then information can travel pretty fast through the network so what's the problem with these neighborhoods right here what's the problem with neighborhood attention like this the problem is that you break the long range


dependencies so let's see what happens if information needs to go from this pixel or to this pixel or this node to this node but if information needs to travel from this node to this node in a classic attention mechanism everything's connected to everything so that node in the next layer can simply aggregate information from here well that's not possible if you do this


kind of neighborhood attention as we've done here if i do neighborhood attention then at most right because the neighborhood is three long at most this node right here can aggregate information from this node and then again it's three long in the next step so now this node can aggregate information from this node okay because the in the neighborhood is three long


and you can only attend to within your neighborhood this means that if i want to send information to something that's really far away i need to um i need to go many many layers right i need to go layer layer layer layer and this has been well known this has already been a like a problem this has already been a property of convolutional neural networks so convolutions


specifically traded off the fully connectedness of fully connected layers two local connections convolutions but that means that you have to go very deep in order to make long range connections you can't just make them in one step the same problem right here now this paper big bird argued that if you have random connections instead of neighborhood connections


just the property of random graphs mean that um you you are pretty fast in sending information around so because in a random graph of size n you on average all two nodes are connected by path lengths of log n this is much faster because in this neighborhood thing two nodes are connected in a path length of order of n right you can you can pretty easily see that if i make


the sequence longer i need that many more steps in order to send it around in fact it's like something like n divided by m this neighborhood size in a random graph it's log n and in this axial attention that's why i introduced it it's two okay every uh every two nodes are connected by two steps if if node if this node right here needs to send information to this node right here


in a classic attention mechanism you could do some one step because every pixel attends to every other pixel however right now we have to um we have to see so this node attends in this layer sorry i have to think so how do we send information between the two we select this node right here in the first layer this node pays attention to this row okay which includes the red


dot so the red dot can send information to the x in this layer in the next layer we select this node right here which is our target node where the information should go to it pays attention to all of this column which includes that x that before right this this x right here where we send information to so it takes two layers two steps to send information from any node to any


other node that's pretty good so this um axial attention if you stack them on top of each other you sacrifice a little bit of uh of being able to send information from anywhere to anywhere for the pleasure of not having this quadratic attention anymore as you can see your attention mechanism is now as long or as big as your column or is wide or your row is high again this


isn't this isn't specific to rows or columns you could do this as i said with these kind of uh diagonals you could do it with any other sort of sub pattern where you can sort of guarantee that the overlap between the layers is enough so you can send information around pretty efficiently and they use this right here so this axial attention you can see the formula is exactly the


same the only change from before is this part right here you can see that the neighborhood that they aggregate over is no longer m by m it is now 1 by m so we've seen them going from if this is the the full input image and you wanna you wanna see where to attend what this paper does is it says a classic sorry a convolutional neural network would be attending to some sub part


right this is convolution an attention mechanism pure attention would attend to everything right this is attention then what we are doing sorry that was a mistake what other people were doing were reverting back this attention um to a subpart this kind of neighborhood attention okay but that was still you know you still have m squared you still have o of m squared


because of the attention mechanism now what we are doing is we are going even lower we're actually going one by m okay this this is with with axial attention so in general it's one by m and then in the next layer we can go one by m in this direction and have that property um and because it's so cheap now right because it's now o of m to compute this we might as well


make m as long as the row itself okay so their last step is going to be to say okay we have one by m right here and that's going to be the row itself now you can see right here that they say axial attention reduces the complexity to hwm this enables global receptive field which is achieved by setting the span m directly to the whole input features optionally one could also


use a fixed m value in order to reduce memory footprint on huge feature maps which is something that they're going to do later on imagenet i believe so when they have big inputs or big outputs they actually do use a smaller m what you can see right here is that i wasn't really that wasn't really correct of me to say that it's now o of m because you you still have the entire query space so you


multiply query by by keys now even if you make the keys to be 1 by m yes you reduce definitely you reduce this from height times width to times height times width to this but then you can see on this thing right here if you take it and let's say we have this kind of row pattern and we replace m by the width then we have width squared so again the square appears however


it's smaller than the original attention the original attention was h squared w squared right because hw is the image and you need that squared in order to do the attention mechanism now we've basically reduced one of the factors it is still an attention mechanism so there's still a tension going but we've basically transformed the the image we've reduced it to one column


now the one column is still a tension so this is still a tension like here so this now reduces to the attention that you see in a in a single sequence okay if you see the image as a long stretch of pixels what this does is basically it's up it simply subdivides that into neighborhoods so we're back to neighborhoods basically um but we shift the neighborhoods


from layer to layer so in the next layer the neighborhoods are going to be just alternating right the neighborhoods is going to be this is one neighborhood connected to this neighborhood connected to this neighborhood i hope this makes sense so it's going to be it's basically a mix between if you if you were to do this in convolution you could do one layer where it's neighborhood


convolution and then one layer where it's like convolution with holes in it i think they're called actress convolutions or something like this with like giant holes in it that are exact is exactly the anti-pattern of the neighborhood convolution from before that's what this is so you see their axial attention block right here their axial attention block replaces the resnet block so if you know


resnet i've done a paper on resnet resnet basically takes the input pipes it through straight and adds to it whatever comes out of this operation okay that's a residual block now usually this thing here would be convolutions and convolutions and they are now replaced by these multi-head axial attention you can see there is a multi-headed tension in the height


and there is a multi-head attention in the width and that gives us the property that every node can send around information to every other node in two steps i don't like the fact that there is only two because um well this i guess this gives a significant bias to one or the other direction depending on the order that you do them in if if i had done this i maybe would have


used three of them because it depends on how you want to aggregate information right like here you train the network specifically to aggregate information first in this direction and then in this direction which might work and it will give you that sending around information anywhere so maybe they've actually tried and it it just performed the same so i i just might


have a dumb suggestion right here in any case they simply replace in we've come a long way right we've gone to like neighborhoods and blah blah blah ultimately take a res net replace the convolutions with the height axis attention and the width access attention and we're good and then we come to results so that's it you have these positional embeddings you have the axial attention


and it turns out that on imagenet they perform fairly fairly well so you can see that models like a resnet 50 model will get a 76.9 on imagenet which is not state of the art but it's also not it's not bad right the resnet 50 is pretty good model um you can see the full axial attention right here uh achieves a 78.1 also not state-of-the-art but still pretty good and as they say it's the


best fully attentional model on imagenet as our standalone attention model on imagenet so where this model really shines is where you really have to make long-range connections between pixels and that's these kind of segmentation tasks and i want to skip the tables right here yeah their best and everything and go to the appendix where they have some examples of this so here you can


see specifically uh this is the original image you have a ground truth and you have the differences between their model this axial deep lab and the panoptic deep lab um that is a baseline for them and you can see that the the failure cases here are are pretty you know show how show how the axial deep lab is better i don't know if they are cherry picked or not but


at least you can see that at some point so it handles occlusions better it handles instances better so here you see that the ground truth separates the person from the tie and the axial attention is able to do this but the the baseline is not able to do this correctly because it labels part of that white shirt also as and you can see why there's kind of a delimiter line here


here here here but if you have long range dependencies right if you have long range dependencies in the model the model will recognize wait wait that's that must be the same thing as this thing here and this thing here and this thing here so that must be the same object um it's simply that the shirt was occluded by the tie and goes beneath it and now appears


again it's not a different it's not part of the tie and it's not part of the um of a different object it's actually part of the shirt so the long range attention you can see at these examples sometimes here okay this might not be an instance of super duper long range dependencies this is simply where the model performs better so you can see here the ground roof has that surfboard


segmented and the baseline does not um that this can also just be you know there are a lot of tricks to make this work of course and you throw a lot of compute at it and sometimes you just get better numbers or part of the better numbers because of the additional compute right here what do we have so you can see occlusions it appears to handle occlusions uh in a better way


and this might be due to this axial attention it might be due to the positional embeddings but you can see that the ground truth here has the laptop between the person's hands segmented the baseline cannot do that but the axial tension does do that and i don't know what this is honestly this is um uh you can you can see though the axial attention also misses the fact that it


should segment this in the background and if this occlusion handling you can see best in this example where the person in the back reappears on both sides of that person so you can see that the axial attention manages to segment that where that is just a mutant person right here though the ground truth is equally shaky i think there is might be some ambiguity of how you can


segment these images obviously but you can see the fact that there are long range dependencies probably helped with this saying that wait in this image there's this white stuff right here and there's this white stuff right here and um connecting these two regions with attention probably helped in segmenting uh these to be the same object even though you can see


there is a break in the object so there is a break no at no point is the object on the left uh touching or the segment on the left touching the segment on the right and still the model manages to put those into the same label category there is the last um last thing where they they want to research what their heads learn and usually you can do this right you can kind of


visualize what the attention has learned so in this case right here in the column heads the way you have to read this is that this particular head right here um aggregates information from its column so everywhere where it lights up it there's a lot of information being routed you can see specifically in this here uh the heads of the people or the heads of the


persons in the picture light up fairly well so for example this head right here is probably aggregating information a lot from this position right here and this head here is aggregating information from this position so you can deduce that that particular attention head probably deals with people's faces uh whereas that particular attention head probably deals you can see the


attention is mostly on the grass right here and you can see the same with the for the row heads now their description here is that we notice that column head one corresponds to human heads while calm at four course correlates with the field only which you know you can interpret it as this this seemed pretty clear but then they say something like row head six focuses on relatively large


relatively local regions where column head five pools all over the image so row head six which is this thing right here you can see that okay it maybe focuses on small regions though you can see okay what like here you can get it that's a person but in other places um i don't know where column head five pools over the whole image and this i don't know maybe they just


needed something more to say because they put these pictures here they were like oh okay the the column heads are really nice because we couldn't like these this one's really nice because it you know just pays attention to the people and this one looks really nice because it pays attention to the field but we can't really put the column head attention without putting the row head


attention but then none of the row heads really are like super distinctive on the particular thing in the image so we need to come up with something that we can say and then he's like ah this one this is there's not a lot of attention so we need to contrast this with something then you would think that they contrast it with another row head but then there's no row head that does


this whole image so there's like ah column at five yeah i'm i'm not sure if there's there's a bit of there's a bit of uh tactical writing going on here i suspect i mean it's still you know it's doing something uh cool but yeah there's there's a definitely an element of sales in when you do when you write research papers and just um not to this data but just props to the lines in front of the


histograms makes it so much easier to read how big the stupid bars are why does everyone put the lines behind the histogram i probably do that myself and now i'm just i'm realizing how much easier that is all right there is a big big big experimental section right here and there's a big appendix where you can read up all of the different numbers comparisons ablations what not um ultimately i just


wanted to go over the method basically putting this into context with other things like putting this into context with stuff like big bird axial attention other positional encodings uh how it co how it relates to convolutions how it relates to feed forward networks and what convolutions did to feed forward networks and so on i hope you at least a little bit gain an understanding of


[Music] Tocantins the way human whip [Music] [Music] with the powers code we are able to create technology that solves some of the most fundamental issues with humans in the first place we can cure diseases we can solve world problems we can do so much that we never put down without the power of technology feels like home in terms of conferencing I like the format a lot it's very the

WeAreDevelopers Congress Vienna 2019 Aftermovie


audience is more technical than usual so I definitely appreciate this conference as a sort of contrast to what usually takes place in many venues [Music] I'm a programmer myself you know so basically talking to other developers is the next experience just to see you know how other people are dealing with the same maybe constraints that you are dealing to [Music]


but I was very amazing I really like the location so it's really really cool to be at Hope work [Music] [Applause] both changes the way people think and people find in Kord impacts the way that different social challenges can be solved [Music] [Applause] and the cause changes the way we consume information technology can help us to create fake news but they can also help


[Music] in this video we'll be discussing the concept of semi-supervised learning semi-supervised learning kind of takes a middle ground between supervised learning and unsupervised learning as a quick refresher recall from previous videos that supervised learning is the learning that occurs during training of an artificial neural network when the data in our training set is labeled

Semi-supervised Learning explained


unsupervised learning on the other hand is the learning that occurs when the data in our training set is not labeled so now onto semi-supervised learning semi-supervised learning uses a combination of supervised and unsupervised learning techniques and that's because in a scenario where we'd make use of semi-supervised learning we'd have a combination of both labeled

Semi-supervised Learning explained


and unlabeled data let's expand on this idea with an example say we have access to a large unlabeled data set that we'd like to train a model on and that manually labeling all of the state ourselves is just not practical well we could go through and manually label some portion of this large data set ourselves and use that portion to train our model and this is fine in fact this is how a

Semi-supervised Learning explained


lot of data use for neural networks becomes labeled but you know if we have access to large amounts of data and we've only labeled some small portion of the data then what a waste it would be to just leave all the other unlabeled data on the table I mean after all we know the more data we have to train a model on the better and more robust our model will be so what can we do to make

Semi-supervised Learning explained


use of the remaining unlabeled data in our data set well one thing we can do is implement a technique that falls under the category of semi-supervised learning called pseudo labeling this is how pseudo labeling works so as just mentioned we've already labeled some portion of our data set now we're going to use this label data as the training set for our model we're then going to

Semi-supervised Learning explained


train our model just as we would with any other labelled data set okay and then just through the regular training process we get our model performing pretty well so everything we've done up to this point has been just regular old supervised learning in practice now here's where the unsupervised learning piece comes into play after we've trained our model on the labeled portion

Semi-supervised Learning explained


of the data set we then use our model to predict on the remaining unlabeled portion of data we then take these predictions and label each piece of unlabeled data with the individual outputs that were predicted for them this process of labeling the unlabeled data with the output that was predicted by our neural network is the very essence of pseudo labeling now

Semi-supervised Learning explained


after labeling the unlabeled data through this pseudo labeling process we then train our model on the full data set which is now comprised of both the data that was actually truly labeled along with the data that was pseudo labeled through the use of pseudo labeling were able to train on a vastly larger data set we're also able to train on data that otherwise may have

Semi-supervised Learning explained


potentially taken many tedious hours of human labor to manually label the data as you can imagine sometimes the cost of acquiring or generating a fully label data set is just too high or the pure act of generating all the labels itself is just not feasible so through this process we can see how this approach makes use of both supervised learning with the labeled data and unsupervised

Semi-supervised Learning explained


learning with the unlabeled data which together give us the practice of semi-supervised learning so hopefully now you have an understanding of what semi-supervised learning is and how you may apply it and practice through the use of pseudo labeling and I hope you found this video helpful if you did please like the video subscribe suggest and comment and thanks for watching

Semi-supervised Learning explained


hi there if you play chess you'll probably recognize the following moves as illegal in the top row pawns move two squares at a time while they are not on their home row in the bottom row you'll see a pawn moving backwards and another one moving sidewards even so in classical chess these moves are illegal but there are variants of chess where these moves aren't illegal where

Assessing Game Balance with AlphaZero: Exploring Alternative Rule Sets in Chess (Paper Explained)


they are actually explicitly part of the rules these are alternate chess rules and this paper is about exploring those rules what happens if you implement those rules how does the game play change and what can we learn for general games so the paper here is called assessing game balance with alpha zero exploring alternative rule sets in chess by nina thomasev ulrich paquette


demis hospice and vladimir kramnik uh the former three of deepmind and the latter is was the world chess champion for these eight years depicted so the paper tries to bring together two different worlds first it is the chess world so a lot of this paper is explicitly about the game of chess if you don't play chess or if you occasionally play chess like myself


this might not be the most interesting paper though it contains some really interesting kind of bits the other world is the reinforcement learning world which you'll see in the alpha zero name right here so the reasoning behind this is the following chess is a really really old game and rules have evolved over time and have sort of consolidated on the rules we have today


but also strategy has evolved over time and lots and lots of thinking and theory has gone into the strategy of chess and to change the rules around um you can change the rules of chess however you can't really assess how the game would be played by humans uh if the rules were changed because you don't have a thousand years of the entire humanity studying these


new rule sets and therefore you're kind of stuck with assessing the games from the perspective of someone who has learned the old rules but reinforcement learning to the rescue so consider the following rule changes no castling this is a really simple rule change no castling castling is disallowed throughout the game if you don't know what castling is castling is like a special move


where there is this rook and the king is right here i don't know how to the king and if there's nothing in between they can sort of swap positions it's called castling uh it's a special move that you can do and it allows you to bring the king to the outside where the king is safe and to bring the rook to the inside where it can potentially cause a lot of damage


so it's a very very favored move by a lot of players and no castling the rule change probably alters the game a lot because if you think of the chess board kings start about here they can only move one square at a time so to get them to safety will require like four or five um steps for them while you have to move everything else out of the way including the rook that stands here so


players might elect to just leave their kings where they are but then they can't really open up in the middle as much because that would leave their kings exposed so it is fair to assume that just introducing this one rule might change the games around quite a bit how the game is played but as we said we don't know this is from someone who has learned classic chess and all


the grandmasters that we have have played and learned classic chess so how do we assess this this paper says that alpha zero can be used to assess these new rules so alpha zero is a reinforcement learning algorithm that can learn these board games very very quickly in within one day or so and it can learn them so well it can beat humans at the game easily in fact modern


modern grand masters and so on use these algorithms in order to learn and to better their play in order to expand their theory their knowledge of the game to play better against other humans so alpha zero imagine alpha 0 can solve a game to perfection what we could do is we could simply give this rule to alpha 0 together with the all the other chess rules and then let alpha 0 solve the game give


it a day and 50 billion gpus solve the game to perfection and then look at what alpha zero came up with kind of look at the games how they turn out and um whether or not they are more interesting less interesting longer shorter and so on so that's that's what this paper does so there's the implicit assumption which you need to believe in order to believe anything in this paper


is that alpha zero actually has this ability there is pretty good evidence that it does because of zero cans of classical chess and go and shogi and a bunch of other board games um all with the same hyper parameters it can solve them such that it is easily at superhuman power so but you need to recognize that this is an assumption so what is alpha zero if you don't know what alpha zero is


alpha zero is a reinforcement learning algorithm but not in the kind of base reinforcement learning sense it is a reinforcement algorithm that has a planner included what do i mean by this so if you are in a let's consider the game tic-tac-toe so alpha zero for tic-tac-toe in tic-tac-toe you have this board and you have a situation where let's say you play your opponent plays this and


now you're tasked of playing something you wonder should i play maybe here or here or here where should i play so what you can do is you can train a reinforcement learning algorithm you can do q learning what not okay that will maybe work what's better to do is you can plan so in planning what you want to do is you want to build a tree of possibilities so we're going to consider all your


possibilities and in this case you have eight possibilities so we want to consider all the eight possibilities and i'm going to draw just some of them so up here you're going to consider the possibility that you place here and here you're gonna consider the possibility that you place in a different spot right here okay and you can see how this goes so if you want to plan


and here you have your opponent has seven possibilities and here your opponent also has seven possibilities and so on so you get this entire tree of play but if you could do that and if you could do that to the end then you could easily simply choose the path here where you win okay where um no matter what your opponent does you win you can find such a path if it is


possible at all to win which is not in tic-tac-toe right if everyone plays optimally it results in a draw but let's say you could win you could choose the path that gives you the best result and that's it there's no learning involved okay so alpha zero works with a planner and planners usually construct a tree so in an abstract way you are in a situation and you consider


all your options and with all your options you consider again all your options and so on and you do a tree search now this tree in tic-tac-toe it's already huge as you can see um in something like chess it is way way huger okay and therefore it's not possible to actually search the entire tree because you need to consider every single possible future situation from the board position where


you're in right this here is the board position where you're in and this is the future the entire future of the game so every single possibility so alpha zero uses this thing called a monte carlo tree search it has several components so it's first component and they right here they have a description and it's very short alpha zero this is alpha zero this is what it does it's


like this is almost comically short so what you do is you put your state so s is your state okay s is it's the board as you have it right now okay this here that's this is s okay you put this into a neural network and the neural network gives you two things first of all it gives you p and and v so that's the second thing so v will simply give you a number v will tell you that this thing right here


is about a plus 0.5 maybe so it says so plus one is winning and minus one is losing uh and it is this is called the value so maybe it says well this position i'm going to expect you to win uh roughly 75 of the time right which in expectation would be a value of positive 0.5 here because 75 percent of the time you win and the rest you lose let's say there is no draw in tic-tac-toe


so there's this value function and the second thing is this p and the p is a policy function so the p will and i've drawn this a little bit maybe not super super duper too large but the p will tell you for every possible move you could make which one should you consider even okay so it maybe it assigns this here a point three and this here a point four but this here is like a point zero


zero zero one and so on so for every possible move that you could do it will assign a number and it's a distribution so these numbers add up to one but that's not important it tells you which moves you should even consider going forward right so p in this case is a distribution over the next moves and with those two things together we can reduce our tree search quite a bit so now


instead of expanding all the tree let's go back to the tree right here you can ask your p hey p which one of these three should i even consider and maybe p says you should only consider those two okay and then you go down and again you ask your p hey p which one should you consider and p maybe says well here you should consider those two here you should only consider that this


one and this three over here we've we've already discarded this from the beginning okay so this p right here it guides your search it tells you at each point which moves should you consider and this as you can see reduces your tree dramatically in fact what alpha zero does is it simply says you have one second of time now expand as much as you can in this tree uh given this one second


uh of of of time budget and the second thing is the value so what you would have to do expanding the tree is always to go to the end right so you always go to the end where at the end you have a fully filled board i don't know here x so you consider every possible situation okay here maybe this this player wins as you can see you always have to go to the end