having labels so having the audio file and the accompanying labels it's just so difficult to come up with all of that so that's the problem with supervised learning is it's not always possible to get this kind of labeled data but then there are times that the housing prices that is definitely possible because you would have things like that on record and the good thing about modern times is that there's a ton of data being collected all the time by different apps and things like that and in those circumstances you often come up with a


lot of labeled data naturally so unsupervised learning then is we have training data and it doesn't have labels so the goal here is to identify interesting structure in the data because we don't know the answer is necessarily but you can see in the graph they have associated here because I'm supervising something people don't grasp as easily because you're kind of thinking well what would be the point of this but one example is anomaly detection so that's when you're trying to something that's a complete outlier and

Supervised vs Unsupervised vs Semi / Self Supervised vs Reinforcement Learning | Machine Learning


it's either it's something new or it's just you know a total outlier so you can see we have in the data we have these clusters that we can see and obviously in a 2d data set like this it's very easy to see these clusters with your eyes not necessarily using any kind of algorithm so we can see we have these clusters of blue spots that are normal and then we have these orange ones that are being classified as noise because they don't fit into these clusters so those are the kinds of things you might want to look up for in a data set and


obviously you're saying you know why do we need unsupervised algorithms if we can just see this but in a multi dimensional problem so here we've just got two variables if we have like 50 you can't see this with your own eyes when there's outliers like that but you can use an algorithm to cluster your data and then if we have data points that are so far away from the center of a cluster and I cluster is a group here so you can see them in the graph as well if we have something that's so far away from a cluster then we know it's probably an


outlier or something like that so that's an example now we're gonna go into this sort of sub categories here of supervised and unsupervised so hopefully you have a general understanding of the difference here supervised we have labels unsupervised we don't and so the first example we're talking about is self supervised learning so we have an unlabeled data set but actually what people do in these self supervised learning is they actually turn an unsupervised they turn an unlabeled data set into a label data set by


manipulating the data in some way so the basic task here is to find the missing part given and one part so to explain that better want to imagine if we have like what we had before was a set of sentences that was the data we were given what the um what the researcher does is they take as one part of and they generate this training data set with labels so we have the information is the trick the sentence with a piece missing and the answer is the piece that's missing so they train this way and this way we have labels because we have these


sentence with the piece missing and we have the missing piece so they generate this label data set from unlabeled data so it works in these kinds of situations so in speech recognition well in sentence like if you're trying to understand the semantics of sentences you can take a sentence remove one piece and see if the system can learn to fill in the blanks and similarly with images we can take out a small part of the image and see kind of a system learn to spot fill in that missing part of the image and actually it has been


relatively like pretty successfully be surprised if you want to look up self supervised learning it is pretty interesting to see how a system can learn these things and that's usually done with deep learning because it's not something a traditional machine learning algorithm can necessarily do myself like a linear model is not going to do that and so yeah basically in this sort of learning you want to learn about the underlying properties of the data and this would have required a lot more data if you don't do it this way so this is


kind of like the system's being thrown into the deep end being expected to generate this missing piece from basic examples that they've seen before so moving on we have semi-supervised learning so I mentioned before that labels are very hard to obtain and especially in cases like speech recognition and web content classification so if you've got a webpage and you want to add tags to it you know it's hard to it's very time consuming for a person to sit down and read through that entire webpage and manually assign tags and then as well


you probably have to have a couple of different people doing it so that they can it's not like it's one person's bias it's a few people who are collectively doing so it's very time-consuming it takes a lot of Human Resources to do all of that labeling so semi-supervised learning algorithms have been developed for the cases where we only have a small amount of labeled data and I'm a very small amount when there is a huge amount of data that could potentially be there so what I said about sentences there's unlimited possibilities for sentences


and we might have a small number of labeled audio to sentence and files and a ton of audio data that haven't been labeled so basically this is used in any task that requires a big amount of human resources to do the labeling and when there's a big amount of possibilities so the characteristics here are that we have a huge amount of unlabeled data otherwise we would just use a supervised learning algorithm and there's an input/output proximity symmetry so what does that mean it basically means that the underlying requirement here is that


if two inputs are similar their output should be similar and that's a very simple thing to understand but it's if it's not present this won't work because you can't make things you can't get the system to learn that to some inputs that are similar should be very different you know this kind of algorithm wouldn't make sense in that way so that's one other requirement and another requirement is that the labeling should be relatively simple so we have we need we need the simpler labeling to be not more difficult if we have this middle


step so trying to get the labels from the small set of label data should not be more difficult than it would be if we had humans doing it as well like generally we'd have this is a low dimension problem meaning that there's not a ton of features and a ton of input variables and things like that so the last one we're going to talk about is reinforcement learning and this definitely could do with this whole other a whole other video all about reinforcement learning but I just wanted to give a basic introduction to it


because I think that's something people are usually trying to figure out what's the difference between supervisor unsupervised and reinforcement learning but basically this is a form of supervised learning the the main factor here is the system focuses on maximizing a reward signal so the reward signal is supervision that that is a convenient so we have an agent and it's said that into the feature space so all of the input variable options are there and it was it has to learn what actions it can take in this


feature space in order to maximize the reward signal so one example would be AI chess or any form of AI game playing it's generally done with reinforcement learning so it's if you example if you think about the chess board the the thought I guess that's playing chess it has all of these options of the different moves they can take and the reward they're trying to maximize is winning the game and that's all that really matters and this is whether they win the game or not so it'll play this game hundreds and hundreds of times


until it learns what exact moves in both scenarios so that the environment also plays a role here so you can see in the graph we've got the agent actions environment and so here the agent would be the ball playing chess the action will be a move that the bot takes the environment is in this case we have somebody else playing chess so the move that they take is the sort of environment that they need to relate to as well as the chess board that results from these actions so when the bot makes a move their pieces move so they need to


update their understanding of the moves that are available based on the move that they've taken and then again by the moves that the other person has taken and then so the state will be the game the board game the layout of the board essentially and the reward again will just be the whether they're winning or not their probability of winning will probably be included in the some way based on the layer of the board and they learned by trial and error what is the best move to make in what situation and there's also usually a


delayed rewards obviously in this case we have that there is the winning of the game and that can be delayed in the sense that some moves that you take will have a knock-on effect on your probability of winning so like if this game is tracking the probability of winning your next move might not effect that necessarily but you'll give you access to the move after and that could increase by a lot another example is things like a but that has to travel around and collect things obviously every move that it takes won't necessarily collect


something but a few moves from there if we continue along the right track they'll pick up something and that will be a reward so the delayed ward is a big characteristic there and then because that's how it learns how this set of moves are important as not just a single move but like a series of moves and then the trial by error is how it learns and yeah usually there'll be like hundreds and hundreds and thousands of instances of this and then that's how they learn so again reinforcement learning can do its whole other video I just wanted to


give a basic introduction to all of these things so I really hope that that was helpful for you guys I hope you enjoyed this video and if you want to see more videos like this again do be sure to give it a thumbs up onto um comment down below because this is a new style of video on this channel and when we're trying get new things here I need to know they're working so buying you guys giving the video a thumbs up on commenting below it then that is way for me to understand whether it's working or not and then in future that means I can


okay so hi everyone so huge thanks to Alexander hyung you and Sammy for organizing this incredible workshop and excited to be here and thanks everyone for coming I know I'm between you and lunch so I'll try and keep things on time so today I'm really excited to talk a bit about transfer learning in the context of deep learning so transfer learning is this incredibly popular technique it's used almost everywhere that we apply deep neural networks but there's also this challenge that we really don't understand many aspects of

Towards Understanding Transfer Learning with Applications to Medical Imaging


it all that all that well and so by the end of this talk I hope that you know you have some sense of all the many different ways it comes up and also some of the interesting open questions there are in the field so a lot of this talk is going to be based off of a paper understanding transfer learning for medical imaging that's joint work with my collaborators chauhan jeong john Kleinberg and sami Benjo okay so let's dive right in what is transfer learning very basic so in the settings that we're gonna really be


studying what you do is you first you learn classifier on some tasks let's call it Tosca and sometimes we call this pre-training then having learned Koski you continue training this classifier on a new task task B and the goal is really to get good performance on task B so the goal is to get good performance on toss B you might ask why train on task a at all and the general belief in the community is that if you know task a is sort of complex and diverse and very general then by going through this process of training on task a hopefully


you've learned useful things that you can sort of transfer over when you start retraining on on task B so this high-level framework actually has connections to a lot of interesting work that's come out from the the theoretical perspective I think Zac mentioned some in his talk earlier today and I wanted to kind of give a quick point out to shy and Devine who is maybe here no I don't see him oh you're here you're here okay in the front where there's been a lot of interesting work done in in this very related field of


domain adaptation to try and understand this from a more formal framework and I think the times really come to revisit some of these ideas and see how we can use them to give us better insights for transfer learning in in the deep learning context so speaking of transfer learning and deep learning what does that look like well it's very similar of sorry very simple you sort of just replace classifier with deep network so you have this deep network it's gonna be your classifier you randomly initialize it and you train on toski and this will


be known as pre training then you take this network and and it's sort of converged to some set of parameters on taos k and then you train it again this time on task B and then voila you have your final model that you're going to deploy and it's it's hopefully going to do great on on task B so this paradigm is pretty simple but it's been extremely successful and it's probably the computer vision community that sort of really showed us how successful this could be in various applications so in sort of specifically the current setting


what people do is pre train a large convolutional neural network on some data set of large images and you know image net gets a special shout-out here it's extremely popular for for pre training so much so that there are sort of entire papers saying why image net is good for transfer learning but besides that there are a couple of other data sets so ms cocoa is a very popular big computer vision benchmark for object detection that sometimes used for pre training and companies also tend to have their own internal data sets which they


like using for pre training so a really big one that Google likes using is jf th has sort of three hundred million images so absolutely enormous what's been really interesting to see in the past couple of years though is that transfer learning has also become very popular in applications in natural language processing so in the past people were able to transfer word embeddings so you've probably seen these diagrams of taking a word in your vocabulary getting a vector representation and then these vector representations of all of these


nice properties we could do that for a while but it's only more recently that sort of all of these a neural networks that are named after Muppets for some reason have have have been developed and that lets us transfer much more complex representations of language and that's shown to be very very successful in a lot of standard natural language tasks and now I have to mention the most important part of transfer learning which is github for transfer learning so been mentioned in a talk in his talk earlier that you know github can be this


very useful research resource and that's definitely very true in transfer learning so in transfer learning kind of applications nobody actually bothers with the pre-training stuff instead you go to github and you sort of find the model you're interested in and find all of its pre trained weights and then you just download it and then after you download it you just perform the fine-tuning for whatever tells you're interested in and this is really important because what this is enabled is it's enabled people who totally


aren't working on sort of core machine learning to apply transfer learning to all of their problems and nowhere is this more true than in medical imaging where where it's sort of the entire community has almost universally adopted transfer learning as this paradigm and what's the setup here well the setup here is that you take this sort of standard pre-trained imagenet model something large and complex like Inception v3 and then you and then you have sort of these pre trained weights on imagenet that you sort of downloaded from somewhere so


imagenet has you know amongst other things a whole bunch of different dog breeds and then bizarrely you sort of find two in this model to do all kinds of medical predictions so you find unit to predict and diseases on chest x-rays diseases like retinal diseases PET scans for early detection of Alzheimer's and sort of the most exotic application was even sort of screening human embryos for IVF treatments so people are just sort of going out there and doing this and and when you think about this this is kind of bizarre because the reason the


community um you know wanted to do transfer learning is this belief that you sort of learned features on sort of this this source dataset and then you can kind of transfer all of this to to your target tasks but of course medical images and natural images are extremely different to each other so it's kind of interesting to understand what's what's on here one final thing is these aren't just sort of sort of turning into papers where you see accuracies but they're actually being deployed in clinic so this is the example of one company


called idx which literally states that it takes inception v3 pre turndown imagenet and is using these to diagnose retinal diseases and it's sort of out in clinic right now so understanding their examples the inception model mean for this problem adversarial examples where for medical images you mean I'm just asking you're saying this is being deployed and it's natural to think about at the state examples for the infection not that I mean so for example you know I'm not talking about transferability I mean that's that's an interesting


question sort of like how much can you like kind of do like these are completely different datasets so I think that's also like a very interesting thing to look at um but okay so while we're kind of going ahead and sort of deploying all of these there's kind of this real challenge because even in the natural image setting we actually don't understand the effects of transfer learning that well so I'm gonna review some results which have just come out in literally the last year that have really challenged the common assumptions people


have on transfer learning so this is a picture of Ms Coco and I mentioned it earlier it's a very popular computer vision benchmark for for doing object detection so this is kind of what it looks like you're trying to learn these balance boxes and in almost all of the competition entries the standard thing to do is you pre train on imagenet and then you sort of fine-tune your image net a model on this this ms koko tusk but then last year we got this paper rethinking image Annette pre training which basically showed just by kind of


being up maybe a little bit more careful about how you pick your learning rate you actually get exactly the same results from random initialization as you do with Swiss pre-training now part of the reason people really like pre-training on imagenet is again this belief that it's kind of this big diverse task and sort of if you train on there you're gonna learn lots of interesting features that you can kind of reuse in lots of places so sort of this underlying assumption is more data is is great but then there was there's


another paper also just last year which looked at pre training on jft which is sort of this even bigger data set sort of 300 million images and what they find is that more true more pre training data is not always better which is this sort of very closely held assumption in the community so in particular like they sort of try training from random initialization versus the entire data set and in a bunch of places the performances are actually really pretty comparable other criterion if you fix the number of iteration or training data


more source domain data is not better I suspect that that won't be true right it's like fitting more to the training distribution it's not necessarily better to the story features not necessarily better so they're they're not fixing the number of training iterations I think I think they're literally just training to convergence and see what it's late right yeah right like you're saying maybe you just sort of like dues or barely stopping and then you just fix it whatever the number is that you got from the first amount of examples you do


the same number of updates for more data or data points you know I'm not I'm not sure it'll make a difference I see your point and I'd have to check to see exactly what they did but we can discuss this offline yeah but yeah so so one interesting point to kind of make here and I think this is actually connected to some of the theoretical work in this that's that sort of come up in related topics is if you do a better job of actually picking the subsets of data that you train on you actually do see significant performance gains and I


think again this is a place where we can kind of revisit some of the theoretical ideas and see if we can do something better this process is like relatively ad-hoc and then finally one other paper that came out just earlier this year do better image net models transfer a better has this kind of makes us implicitly makes this very interesting observation that when you decide to do transfer learning you're not just taking the the features but you're also committing to an architecture it because you download them both together and if


your task looks very different to imagenet you know this is something you should be be aware of and thinking about and so do better image net models transfer better well it's complicated there's this nuance relationship that depends on sort of how you regularize during the training process the data set the specifics of the data set and size etc and so you sort of actually see a lot of variability based off of all of these conditions and sometimes you actually get pretty similar performance as usual optimal weight optimal where um oh here


um yeah so this is optimal for transfer so so this is sort of like standard ways in which people can perform regularization and this is how you tend to download your pre trained models but these are actually sort of not as good for for doing transfer learning on and.and here if you kind of train in a slightly different way you actually end up with better features for transfer yeah really thorough so I highly recommend reading it so they tried they tried pretty much everything so they tried this sort of like fixed feature


extractor setting where you sort of freeze things and then just retrain some of the top they also tried left fine-tuning setting where you sort of trained everything and they were also of course comparing to training from from scratch in these settings and so yeah the kind of exact results vary a little bit I think your fine-tuning maybe you have like slightly less sensitivity to some of these these settings particularly for larger target datasets but yeah they try everything so definitely worth reading so this is all


in the natural image setting what about in the medical image setting actually hardly anything is explored in the medical image setting which i think is sort of something we should we could really try and address from sort of two angles firstly of course I think it's important to study it because we're actually deploying these in a lot of places and it's important to understand what's going on particularly is this is sort of a very counterintuitive thing to do secondly I think this medical imaging setting also captures a very interesting


part of captures an interesting sort of regime in of performing transfer learning so here your source and your target tasks are extremely different to each other like the data is different the actual tasks you're going for is different and as we'll see later there are still some benefits that that come up so sort of understanding why that happens I think is sort of very interesting to to explore from a purely principled angle okay so in this talk so first we'll do a quick sort of performance evaluation of transfer just


to sort of set things up and then we're going to go into a little bit more detail and try how is pre-training affecting the actual features we're learning in our model and for this I'll also touch on some work we've been looking at on studying representational similarity of networks using canonical correlation analysis and then finally and very interestingly and also somewhat paradoxical we'll look at some feature independent properties of transfer that we see okay so the first yeah so I'm fine um so I think the


question is sort of is there a precise definition of what fine-tuning means in this setting is there as in like you sort of stop after some amount of time or yeah is there something very specified um the answer is like not really you're really just sort of training to convergence or pretty much you're mostly stopping once you see that like validation losses converged yeah but okay so okay so so let's let's take out let's breeze through the first part um so first part we're gonna just try and evaluate the actual performance


gains of transfer and to do this the way people tend to evaluate transfer learning because you're sort of downloading these datasets from at least sorry these models and these weights from github is you just tend to evaluate it on on standard imagenet architectures so like some big complicated thing like this but as I mentioned if your task looks really different to imagenet its kind of important to think about the fact that you're making this implicit or contextual choice and so in our evaluation we also evaluated this much


smaller family of architectures that we call CBR s they're really just vanilla convolutional neural networks they're called CBR s because the most sort of popular and successful way of having a vanilla Convenant these days is to have a convolution followed by batch norm followed by a rel u activation and these things are really tiny there may be sort of like one eighth to maybe one twentieth the size of your full-fledged two image net architecture and then in terms of tasks we looked at sort of two large-scale medical imaging


tasks one of them is diagnosing different diseases from chest x-rays and another one is diagnosing a certain kind of retinal disease diabetic retinopathy from scans of the the back of your eye so we sort of run these experiments across all these different architectures random initialization and transfer learning there was a lot of experiments but we saw some clear takeaways so firstly perhaps you know sort of falling on from some of the results we've seen in the natural image settings transfer and random initialization actually


performed pretty comparably so here's a sort of complicated results table we got from our chest x-ray experiments and if we look at where transfer and random initialization perform comparably it's actually for most of the table and in some cases transfer even sorry random initialization even outperforms transfer secondly and and interestingly we observed that these simple vanilla kind of networks actually performed about as well as these standard imagenet architectures so we weren't really trying to optimize for performance we


just wanted to try some simple things to see what they look like compared to image net like image net architecture is because those architectures are really pretty different to to what you might want for the for the medical data that's it knows the simple architecture did you also free trainer when this yes yes so we pre trained them on imagenet and then we and then fine-tuned on the medical data sets yeah and then sort of did those comparisons so the data set sites so we actually buried this so like kind of variations of this


or in the paper um the full data set is around 200k images I think which is kind of like reasonable size but um sort of smallish compared to image net which is like in the millions so it's sort of interesting to see that these sort of perform comparably and then finally we also saw that image net performance was not actually indicative of how these architectures would perform on a medical task so what do I mean so let's look at another results table so here's where I at 50 and here are these two architectures and here's what we saw


when we train them on image net these architectures actually perform horribly they're not designed for image net in a way that I can sort of explain offline but when we look at how they do on the medical talk they're actually really within sort of ballpark performance of each other oh these architectures they're just um they're just like a family of simple vanilla convolutional networks yeah we kind of just made them up just because like we wanted something extremely simple and I can explain why sort of like you're sort of seeing this


difference I guess offline if you want but there are kind of clear ways in which image net is not the right way to design your architecture for some of these tasks and so we were able to take advantage of that you mean the data set size so yes it's something we've buried but I'm full dataset sizes maybe two hundred thousand images for both of them yeah so I mean this last point is interesting because other papers so this paper undo better image net models transfer better and even the papers that been Rick's group has been working on on


this distribution shift and sort of seeing if how performance car lights across distribution shift does show that there's this correlation but here here we don't see this okay so that was kind of a quick sort of overview of what the the performance evaluations look like but we like to go beyond just the performance evaluation we kind of really want to understand what is transfer learning doing to our architecture it's like what are we gaining from from applying transfer learning if anything at all and you know


I mean what we saw is like at a performance level things are performing about the same and so a really fundamental question is well random initialization and pre train weights don't really look anything like each other so we have one thing sitting in one part of the space another set of parameters sitting in another part of the space what's happening during this fine-tuning process is it just that it doesn't really matter how you initialize and then after you fine-tune you just sort of change dramatically and


everything you initialize with is erased so we could just kind of do whatever we liked or is something else going on and to alter that question what we really want to do is we want to look at some of the the latent representations of these models and and take a measurement to see how similar they are the problem with trying to do this kind of an analysis is that comparing representations from different neural networks is is really difficult there's this alignment problem it's not like one neuron and a layer of


one network is going to correspond nicely to to another neuron in the layer of another network in fact there's no reason that one neuron Maps so another neuron at all it could be like a group of neurons that are having the same function as as a single neuron and another network or one group mapping to to another group so it's pretty complicated but in another line of work we've been looking at doing exactly these kinds of comparisons using canonical correlation analysis and the basic framing is as follows we have some


data set of interest and we're going to think of the neurons representation what it's learned as what we call an activation vector so we feed in this kind of data set of interest and this neuron is going to emit a scalar value across all of these these input points and we can literally just sort of stack all of these and that'll form a vector that we call the the activation vector of this neuron and so there's this sort of nice framework where we kind of think of these neurons as these activation vectors and because layers are linearly


combining their neurons there are sort of these subspaces that are sort of spanned by their neurons so I'm so so this is going to be at a very high level the details of like all the the the mathematical details of CCA are in their relevant papers but so at a very high level what we do is we take in two sets of these neuron activity vectors and they're typically going to be layers so layer from one network a layer from another network and then CCA will find the linear combination of these neurons that maximizes correlation and by


iteratively applying this process we can basically get something like a similarity score between these layers so the score just tells us well you know how how similar are the representations learned by these layers sort of up to sort of scaled linear transforms so that's the way in which it addresses this disalignment issue and so previously we've kind of used this to study various properties of convolutional networks lately it's been become quite popular in studying various different kinds of language models in


NLP and more broadly this this kind of entire area of studying similarity between deep representations has quite a lot of people who have been thinking about it so we're I'm mostly gonna be talking about using CCA but the first paper here was probably this paper called convergent learning by Lee Osinski Kuhn Lipson and Hopcroft in ic LR 2016 where instead of dealing with the distributed problem they just tried to find nice one-to-one mappings between between neurons and different networks then we kind of followed up with some of


the CCI work then there was a more recent paper that's really pushing this framework of having activation vectors for neurons and and sort of comparing similarities between subspaces and then most recently there's been this paper similarity of neural networks representations revisited by Kornbluth naruse Ely and Hinton that's broadly proposing a kernel based similarity measure and one one quick note about sort of all of these is that I think performing these sort of similarity comparisons is a very interesting way to try and get at what


your are neural networks are doing it's kind of useful for interpretability and it has interesting consequences for things like compression and for model Ensemble Inc and sort of all these papers including our own are interesting but I think there's a lot of scope for doing things in a more formal and a more principled way so although many of these methods are built off of sort of principle techniques a lot of the ways in which we apply them are indeed heuristic and we I don't think we can claim we fully understand their


limitations or or where best to use them so I think there are a whole bunch of interesting questions in this space but okay so going back to transfer learning what are we going to do well we're gonna do a very simple experiment we're gonna train a bunch of networks from pre train weights we're gonna train a bunch of networks from random initialization then we're going to apply CCA to just see how similar they are to each other we want a baseline so CCA is gonna give us these similarity scores but we want some kind


of a baseline to compare these two and so we're also going to look at the similarity scores we get when we train a population of networks from different random initializations and apply CCA there so here what the results look like so along the x-axis are sort of different architectures these blue points are what you get from doing this comparison from networks trained from different random initializations and these yellow points are what you get when comparing pre train networks the network's train from random


initialization yeah so so you do see different architectures here but ah good point no we didn't do that and that would be kind of an interesting thing to study definitely so-so but yeah so we just kind of stayed within an architecture but kind of the the takeaway is that these blue points are sort of higher up than these yellow points what does that mean well it means that models train from random initializations seem to be more similar to each other representational e than then models trained from from pre


trained weights and transfer learning so even though we're seeing the same performance there is something different happening at the the representational level so yep the different circles actually compare a correspond to different networks so we train multiple networks and then we just sort of performed we put on these comparisons yeah so it's it's it's averaged over the layers um multiple I mean I can tell you offline but sort of slightly different layers for different networks because their architectures are


a bit different but yeah taking a few layers at different stages in the network performing this comparison and then averaging we have two distributions and I get a bunch of samples Ramon popular some examples of the other you know like basically just be something I'm wondering like any statistic is going to be kind of you know but I think this pertaining to a population but you know this number of like thirty four and a half verse 36 is this enough to tell me something about I think it is hard to compare across architectures at least


the way we did this experiment just because like we looked at different layers and things and we'd have one point see see a similarity actually telling us a lot about of transferability I think well not about transferability but I think it is actually telling us something about what's similar versus what's not similar like I think that was the whole point of like kind of having this sort of baseline comparison I guess in blue and I think the fact that we see and nobody tried like multiple networks for that so


I think the fact that we are seeing that the blue things are higher like in most cases is telling us that there is more similarity there we train on the same training data exactly yeah yeah because we're interested in lots of questions okay we're interested we're interested in yeah sort of seeing what this looks like once we've trained on the medical data yes oh yeah so like we're not done making the conclusion yet but sort of the first thing we saw was that like performance is similar and so like hypothesis one is like it kind of


doesn't matter how you initialize and like they're actually all doing the same thing all the way through so on top they're clearly similar because performance is similar but like we don't know what's happening in between so so then we try this analysis and then and then it looks like different things are happening in between but there's kind of more coming yep CCA assumes that the inputs are like linear you take a linear combination I was wondering if you try like deep CCA where you take a nonlinear or why did you assume make the linearity


assumption yeah I mean that's a good question you could definitely try deep sea CA for CCA like I mean we have to have some kind of I guess place where we want to say something is like you know like where we kind of conclude that things are not that's similar to each other representational e and we thought linear is like kind of a good proxy because you know layers kind of operate linearly and so like you know things are sorted within linear transforms of each other it seems like a reasonable kind of call to say okay that's sort of somewhat


similar whereas yeah when you come to like not like I think the nonlinear comparison could also be interesting but yeah you need to know exactly when to call things similar and not somewhere oh man so many questions okay I'll take one more question and then maybe move on did you have a question me yep okay Jack maybe we'll chat more offline okay yep similarity r/t layers yeah so it's that's an interesting question and what I'm about to get to so so the altar is like I think for like some models you see that but not for not


for not for others and so that is about to come up yeah so so so okay so like kind of so we saw performance is similar and then a hypothesis says maybe they're just all doing the same things that doesn't seem to quite be the case and now like kind of let's look further into that and to look further into that let's do something really simple so this are the actual filters from the first convolutional layer of of ResNet that's kind of initialize with pre-trained weights so you trick take your rest net you pre train it on image net and this


is what the filters look like and sort of you know true to kind of the community's expectations you see all of these really nice Gabor filters come up so yeah so I don't think the weights OCC I only operates on like activations like I think it's like I think it's difficult to sometimes make direct comparisons between weights because your weights can look really different but I think functionally is like what we what's like more interesting to us like I mean there's like the standard experiment where you have like a ground truth


neural network and then you train something to mimic that but like even they're like the weights are not going to look like that's similar but like kind of out with wise is yeah what we're looking at well yeah so we're interested like whether there are kind of functional similarities in these they're like like the actual outputs and so that's what we we study by making your your similarity measure parameterize by the examples not by the weights it factors out all these in variances and permutations and stuff like that inside


of a network and like you mentioned there are lots of papers legend sincere that find all these relation between the example mapping or the ground matrix it something yeah like I think almost all of these like even even the paper those doing kernel based measures yeah like kind of thinking about it in terms of examples I think is kind of really helpful for doing these sort of similarity measures but okay so so we're gonna do something even simpler let's just look at like let's just look at the filters okay so these


are the filters from conv one that we've initialized from having pre-trained freshly off of imagenet look at all these gorgeous Gabor filters what happens when we when we train this network so we train it on this medical data and well after training it actually looks kind of similar so so maybe that just means that you know these like these kind of Gabor filters are perfect for this medical data okay now let's see what happens when we do the same thing from random initialization so this is what our our network looks like when we


randomly initialize it and again we're gonna train it on this medical data so we do that and oh man this actually also looks somewhat similar so so so what is going on because so over here like maybe you could say oh there's interesting feature reuse happening but but you're also seeing the same stuff for random initialization okay so that's like our that's that's ResNet now we've trained a whole bunch of other architectures and some of them are much smaller so what happens here well here's one of our very small architectures it's maybe one-tenth


the size of the resonant so here it is initialize with it's nice imagenet weights and what happens after training well it actually changes dramatically and then again if we look at it at random initialization and then look at it off your training again it changes significantly your connections a few dozen skip connections activation I mean we should interpret them by adding you know having you across layers or something like the real representation is and it depend on all the layers not so this is just yeah so so because the


architectures are different we looked at Khan Wan specifically for that because it's sort of below all of that now you are getting feedback from kind of the skip connections that you aren't getting but I think this this is also true for for say like something like Inception as well I'm just sort of showing showing resonate here and I think I think what's really going on is to do with sort of to do with the size of these models so yeah so two quick points one point that's kind of interesting is that everyone loves Gabor filters but these models are


not actually the smaller model which is changing a lot does it actually really seem to be learning at least the classical Gabor filters and so here are like places where there's a good more filter and it's actually erased the Gabor filter so like here's another place where it's kind of good more filter erase cover or filter erase and then the other point is relates to what my tousle already brought up which is that like what we observe through kind of further experiments is that the size I think the size of these architectures


is actually kind of really impacting what you see during this fine tuning process so like the kind of picture we have in our head it's sort of maybe so random initialization and pre-trained weights take us to sort of very different parts of the space but sort of somehow when your model is kind of large and these imagenet architectures are indeed in some sense seem to be large for these medical tusks maybe you just don't sort of move as much and then and then whereas when you have sort of smaller models you sort of change a lot


more now we've you know we've throughout this workshop we've seen lots and lots of interesting work on the neural tangent kernel and sort of thinking about these infinite with limits in the kernel regime versus the sort of the not the deep regime but like at least it to my kind of high level understanding it's not a direct mapping to what we're seeing here so I think it's kind of interesting to to sort of study this further and try and understand try and make this connection because there's probably some kind of a connection and


sort of understanding why this is happening would be really interesting okay so um final point is that in the paper we have a lot more kind of work on sort of broadly thinking about similarity and reuse there are other interesting things we see like we can store for our larger models our similarity at initialization can be pretty predictive sort of similarity post training we can also like kind of look at how much feature reused is happening and this interesting co-adaptation problem which happy to chat about more offline but you know i


think time is running out and I don't want to I want to make sure we get to lunch on time so I wanted to end with I think one of the most interesting observations we saw during this entire set of experiments so one thing we observe again and again across different architectures and and across our different setups is that when you train with pre-trained whites versus training from random initialization there is a huge difference in convergence speed so this yellow line here is what you get when you convert in


sort of your training your training curve with with pre trained weights and this blue line is what you see with random initialization if you sort of extend these far enough out they basically converge to the same value but there's this sort of huge difference in in how quickly they converge and you know when you when you first see this plot you might think oh well this means that transfer learning is doing its job like you know you've learned some useful features and you're sort of reusing it and that's why you're converging foster


but we've also seen a lot of counterintuitive results on the things that larger models are sort of maybe a bit lazier and just don't move as much and it's it's still kind of not fully clear exactly how much feature use is happening in this in this process and so we tried an experiment to try and understand why we see this difference in convergence speeds and the experiment is very simple so we decided to initialize by drawing weights IID from from sort of the same distribution as random initialization but rescaled to match the


pre trained weights so sort of here's here's what this concretely would look like if you initialize those pre trained weights you'd look something like this if you initialize with random initialization you of course destroyed all the features and it looks something like this and then this thing which we call the mean var in it I mean it looks exactly like random initialization except that it's scaling is different because you sort of rescaled so how does this do if we initialize what this and train instead turns out that it actually


helps a lot with convergence speed and we sort of students across different architectures and across our different tasks and so what's really interesting here you didn't very and grains from the imagenet transfer the pre train weights exactly and and it was per later so not across the entire architecture that wouldn't make sense but sort of perlier take this and then initialize and so what's really interesting is that yeah this is this is a feature independent property because we're sampling i idea we've kind of destroyed all of the


features and we're only kind of keeping the scaling but these are these are just both for the the two large-scale medical imaging tells me like that and when we already know from and the relative order of magnitude of the amount of data on sorts first target times because uh a dramatic impact here like did you look at like Burt did you do all the stuff we were doing the exact same things in 2014 but instead of training on the billion or a dataset we were training on like penn treebank or something like this and


that difference like the discrepancy like and I've seen I know at least internally at Amazon I have some colleagues we're doing some stuff that was like how many so many unsupervised examples what I like look the fire fiber choosing it by the cost of like how much would I pay to get a million unsupervised examples versus a hundred extra labeled ones or something like that and these numbered skin you know could be it could be a very big difference so I wonder with 1 million versus two hundred thousand is are sort


of on the same order of magnitude and that's why you see yeah so actually this specific experiment we also so like I'm not covering this here but we tried a bunch of things where we varied the data and so it's interesting but this is actually pretty robust to varying the data so you see the same sort of like convergence like you actually see a speed-up even when your data is much smaller one thing you do see is that with these really large image that architectures if you have something as small as like five thousand data points


that's when you see like a little bit more of a gap between transfer learning versus random initialization maybe it's like two percent but then by the time you've gotten to fifty thousand examples that gap is like almost gone and and like there's really no reason why you'd a priori you want to use like kind of imagenet sized architecture on like 5000 examples like I think that's sort of like also like kind of yeah merits sort of further study we've been talking about over parametrizations I think that's like an interesting


related question but yep the main point here is that this is like this is just purely a feature like a property of scaling and it's kind of a purely feature independent property and so I think there are a whole bunch of open questions here especially related to this kind of scaling part so specifically is there sort of maybe some scaling rule that explains this convergence speed up we looked at this a little bit but sort of not extensively and there are differences but I think it seems like it should be possible to


maybe pin this down and then I think we also did some very preliminary experiments on natural images and I think we're seeing sort of similar effects and there are kind of interesting questions we can all skier like you know if we train and then sort of reinitialize but just preserve the scale like you know do you see a difference in convergence speed that's like one of the basic questions we could try and try and answer and then I think there are sort of other questions that also came up through this process sort


of like kind of maybe really kind of getting at sort of similarity of representations at initialization versus after training and sort of how I do things vary between large and small models because at least I mean looking at the weight certainly they're learning very different filters so sort of understanding that better would be really interesting and then I think there's also kind of scope here to actually formalize things a little more so I'm sort of seeing medical imaging partially also as this way of seeing a


place where transfer learning does provide some benefits some convergence benefits but in a scheme where like the the source and target distribution are extremely different from each other and so just understanding better what might be happening there and maybe saying something formal there could also be very interesting and with that thanks for thanks for coming [Applause] oh yeah absolutely I mean like honestly if you look at how much how long you train rent from random initialization versus training on imagenet plus


training on the medical data like training from random initialization is going to be way faster like this but this question makes scent like sort of the reason this question is very important is because people are just downloading their models from github and then sort of just doing the fine-tuning so at that point you're like oh well you know this thing is readily available so do I want to do this or do I want to do do something else yeah yep now we didn't do 200,000 so I can chat with you offline about that but you don't need to


do 200,000 - to kind of get a good similarity measure you can do something smaller than that the trade-off there is between sort of like the number of data points you're using and serve the actual number of like kind of vectors you're trying to find similarities over this is why people so so okay so I think there are two parts to this firstly like people I think there's some outs of people like using transfer learning simply because they've seen other people do it and kind of you train it and then you can do this process and it sort of


works and you're like oh great but I think I think in sort of settings where people are doing extensive experimentation like I know we think Google like part of the reason transfer learning is popular is I mean they have resources but you're still trying to run a lot of experiments and part of the reason transfer learning is popular it's because of this speed up you see in convergence I think like you have to be a little careful about this because um part of the reason I think you also see this fee up and convergence is because


you're also committed to this imagenet architecture so like in the paper we sort of studied this further but you know I think like we're meaningful feature reuse is happening if it is happening is really in the lower layers and so one way to get sort of similar speed ups but maybe have a better architecture is like you kind of just reuse some of the weights and then you sort of reinitialize stuff and sort of train that away and this ties into like kind of all kinds of other interesting questions so like there's been this


interesting point in in deep learning about this co-adaptation problem which is suppose I have an initialization but I only keep part of it and then I kind of reset everything else how will those work together and and it's kind of interesting because for some settings that's been a problem for this it doesn't appear to be a problem so that's also another interesting question to study I think yeah yep yeah absolutely so I mean I guess I guess Zack and I were we like I guess this is a discussion we mentioned briefly earlier so if you have a very


very small amount of data you will see a bit of a difference so I think we had to get to five thousand data points on the image net architectures at least and so there we saw maybe like a two percent difference instead of like a fraction of a percent difference but then by the time we got up to like fifty thousand data points that kind of that difference was really gone and then when we tried on a much smaller architecture so bear in mind before we were training this enormous image and net architecture so we tried on a much smaller architecture


hi there if you play chess you'll probably recognize the following moves as illegal in the top row pawns move two squares at a time while they are not on their home row in the bottom row you'll see a pawn moving backwards and another one moving sidewards even so in classical chess these moves are illegal but there are variants of chess where these moves aren't illegal where they are actually explicitly part of the rules these are alternate chess rules and this paper is about exploring those rules what happens if you

Assessing Game Balance with AlphaZero: Exploring Alternative Rule Sets in Chess (Paper Explained)


implement those rules how does the game play change and what can we learn for general games so the paper here is called assessing game balance with alpha zero exploring alternative rule sets in chess by nina thomasev ulrich paquette demis hospice and vladimir kramnik uh the former three of deepmind and the latter is was the world chess champion for these eight years depicted so the paper tries to bring together two different worlds first it is the chess world so a lot of this paper is explicitly about the game of chess if you don't play


chess or if you occasionally play chess like myself this might not be the most interesting paper though it contains some really interesting kind of bits the other world is the reinforcement learning world which you'll see in the alpha zero name right here so the reasoning behind this is the following chess is a really really old game and rules have evolved over time and have sort of consolidated on the rules we have today but also strategy has evolved over time and lots and lots of thinking and theory has gone into the strategy of chess


and to change the rules around um you can change the rules of chess however you can't really assess how the game would be played by humans uh if the rules were changed because you don't have a thousand years of the entire humanity studying these new rule sets and therefore you're kind of stuck with assessing the games from the perspective of someone who has learned the old rules but reinforcement learning to the rescue so consider the following rule changes no castling this is a really simple rule change no castling castling is disallowed


throughout the game if you don't know what castling is castling is like a special move where there is this rook and the king is right here i don't know how to the king and if there's nothing in between they can sort of swap positions it's called castling uh it's a special move that you can do and it allows you to bring the king to the outside where the king is safe and to bring the rook to the inside where it can potentially cause a lot of damage so it's a very very favored move by a lot of players and no castling the rule change


probably alters the game a lot because if you think of the chess board kings start about here they can only move one square at a time so to get them to safety will require like four or five um steps for them while you have to move everything else out of the way including the rook that stands here so players might elect to just leave their kings where they are but then they can't really open up in the middle as much because that would leave their kings exposed so it is fair to assume that just introducing this one rule might


change the games around quite a bit how the game is played but as we said we don't know this is from someone who has learned classic chess and all the grandmasters that we have have played and learned classic chess so how do we assess this this paper says that alpha zero can be used to assess these new rules so alpha zero is a reinforcement learning algorithm that can learn these board games very very quickly in within one day or so and it can learn them so well it can beat humans at the game easily in fact modern


modern grand masters and so on use these algorithms in order to learn and to better their play in order to expand their theory their knowledge of the game to play better against other humans so alpha zero imagine alpha 0 can solve a game to perfection what we could do is we could simply give this rule to alpha 0 together with the all the other chess rules and then let alpha 0 solve the game give it a day and 50 billion gpus solve the game to perfection and then look at what alpha zero came up with kind of look at the games how they turn


out and um whether or not they are more interesting less interesting longer shorter and so on so that's that's what this paper does so there's the implicit assumption which you need to believe in order to believe anything in this paper is that alpha zero actually has this ability there is pretty good evidence that it does because of zero cans of classical chess and go and shogi and a bunch of other board games um all with the same hyper parameters it can solve them such that it is easily at superhuman power so but you need to recognize that


this is an assumption so what is alpha zero if you don't know what alpha zero is alpha zero is a reinforcement learning algorithm but not in the kind of base reinforcement learning sense it is a reinforcement algorithm that has a planner included what do i mean by this so if you are in a let's consider the game tic-tac-toe so alpha zero for tic-tac-toe in tic-tac-toe you have this board and you have a situation where let's say you play your opponent plays this and now you're tasked of playing something you wonder should i play maybe here or


here or here where should i play so what you can do is you can train a reinforcement learning algorithm you can do q learning what not okay that will maybe work what's better to do is you can plan so in planning what you want to do is you want to build a tree of possibilities so we're going to consider all your possibilities and in this case you have eight possibilities so we want to consider all the eight possibilities and i'm going to draw just some of them so up here you're going to consider the possibility that


you place here and here you're gonna consider the possibility that you place in a different spot right here okay and you can see how this goes so if you want to plan and here you have your opponent has seven possibilities and here your opponent also has seven possibilities and so on so you get this entire tree of play but if you could do that and if you could do that to the end then you could easily simply choose the path here where you win okay where um no matter what your opponent does you win you can find such a path if it is


possible at all to win which is not in tic-tac-toe right if everyone plays optimally it results in a draw but let's say you could win you could choose the path that gives you the best result and that's it there's no learning involved okay so alpha zero works with a planner and planners usually construct a tree so in an abstract way you are in a situation and you consider all your options and with all your options you consider again all your options and so on and you do a tree search now this tree in tic-tac-toe it's


already huge as you can see um in something like chess it is way way huger okay and therefore it's not possible to actually search the entire tree because you need to consider every single possible future situation from the board position where you're in right this here is the board position where you're in and this is the future the entire future of the game so every single possibility so alpha zero uses this thing called a monte carlo tree search it has several components so it's first component and they right here they have


a description and it's very short alpha zero this is alpha zero this is what it does it's like this is almost comically short so what you do is you put your state so s is your state okay s is it's the board as you have it right now okay this here that's this is s okay you put this into a neural network and the neural network gives you two things first of all it gives you p and and v so that's the second thing so v will simply give you a number v will tell you that this thing right here is about a plus 0.5 maybe so it says


so plus one is winning and minus one is losing uh and it is this is called the value so maybe it says well this position i'm going to expect you to win uh roughly 75 of the time right which in expectation would be a value of positive 0.5 here because 75 percent of the time you win and the rest you lose let's say there is no draw in tic-tac-toe so there's this value function and the second thing is this p and the p is a policy function so the p will and i've drawn this a little bit maybe not super super duper


too large but the p will tell you for every possible move you could make which one should you consider even okay so it maybe it assigns this here a point three and this here a point four but this here is like a point zero zero zero one and so on so for every possible move that you could do it will assign a number and it's a distribution so these numbers add up to one but that's not important it tells you which moves you should even consider going forward right so p in this case is a distribution over the next moves


and with those two things together we can reduce our tree search quite a bit so now instead of expanding all the tree let's go back to the tree right here you can ask your p hey p which one of these three should i even consider and maybe p says you should only consider those two okay and then you go down and again you ask your p hey p which one should you consider and p maybe says well here you should consider those two here you should only consider that this one and this three over here we've we've already discarded this


from the beginning okay so this p right here it guides your search it tells you at each point which moves should you consider and this as you can see reduces your tree dramatically in fact what alpha zero does is it simply says you have one second of time now expand as much as you can in this tree uh given this one second uh of of of time budget and the second thing is the value so what you would have to do expanding the tree is always to go to the end right so you always go to the end where at the end you have a fully filled


board i don't know here x so you consider every possible situation okay here maybe this this player wins as you can see you always have to go to the end but in our case we don't want to always go to the end we'd rather explore more into like more branches than always go to the end and this is where the value comes in so at some point you simply say now i'm deep enough and now i'm going to ask my value v now there are slight differences with respect to alpha go and alpha 0 and so on but they all have in common that they


estimate the value of the intermediate nodes using this v model from over here um i have v s v was green so they use this v model from over here to estimate at a certain depth so v learns to look into the future so everything that can happen from here and it estimates and it says well from here you maybe have a you know a 0.5 value or maybe a negative 0.7 and so on so v learns to assign these values to situations to states which are these nodes right here and p learns to suggest things to expand that's alpha zero


and then at the end if you've expanded the tree enough and estimated well then you have a pretty good idea what's going to happen in each of the branches that you considered right in each of these branches you look into the future um from you here you look into the future here look into the future by doing this pv play and after one second after you've done you know a couple of hundred or thousand or however many uh looks into the future then you have a pretty good idea for each of the top level actions what's going to happen


in the future and you can simply pick the one that has the best future for you according to your own model so that's what alpha zero does not so this is how you combine planning and neural networks you want to do planning but you can't because you can only go so deep so you use neural networks to first of all reduce the number of branches you consider because the neural network will tell you which ones are worthy to even look at and second of all you don't always have to plan to the end because you can simply


ask your neural network how much an intermediate state is worth in expectation and this turns out to be pretty good why don't we do this for every single problem well we do for this we do need a simulator so you may recognize that right here i said we consider all the possible actions that we have and for each action we know exactly what's going to happen this is only possible like in a board game it's not even possible in like a board game where you have a a die to roll or a card to draw anything that is random there there is a way to


include this right here but in this simple formulation we need to know exactly with 100 certainty what is going to happen if we take a particular action so this is only really applicable for the types of full information board games where we can write simulators that are pretty fast right and even then um even though chess you know has lots of available actions and complications it's nowhere near the complexity of like a let's say a modern video game or even or the real world is is completely out of scope for now for


these types of things all right so that was alphago sorry alpha zero uh which builds on alphago of course and uh the rules of chess that we're going to consider using alpha zero are the following so there's no castling no castling for ten moves pawns can only move by one square forcing a stalemate is a win rather than a draw so you may know this in chess if you do not um checkmate the opponent's king but only put them put the king in a situation where it cannot move that's called that's considered a draw and i think even in the chess community


some people want to consider this a win there is torpedo where pawns can move by one or two squares anywhere on the board and semi torpedo where it's the same but only from the second and the third rank pawn back where pawns can move backwards and pawn sideways where pawns can move laterally by one squares but captures are unchanged diagonally upwards and there is self capture where it's possible to capture one's own pieces so um there are you know slight slight details here with respect to the 50 move rule and so on


but if you if you don't play chess simply consider these are changes minor in a lot of cases minor changes to the chess rules that make the new rules either a superset or a subset of the original rules but they are going to have quite some changes in for the play and we're going to look at what happens so that's the entire research setup as you've seen it's alpha 0 applied to these new rule sets and under the assumption that alpha 0 will solve these will become master at these games which we can't verify we can verify in


chess because right alpha zero can beat people that have trained chess for all their life we can't verify it here so again this is an assumption so the first thing i want to look at here and this is going to play a little bit into my criticism of this paper it's a pretty cool paper but i do have some concerns right here is the following uh the following charts so they do they do we don't consider how you train alpha zero let's just say you can train it you know to whatever pretty good performance here is how they evaluate so they evaluate for


each variant they do 10 000 games played at one second per move for each different chess current so if you remember as we do our tree search right we expand the tree according to our p and we estimate the values according to our v and we do this for one second in this first thing so in one second maybe this here is the tree so we have some sort of an understanding of what's going to happen in the future you can imagine if we have more time then we can expand this tree more and get a much more accurate picture of what


happens in the future okay so they do 10 000 games at one second per move but they also in addition to 1000 games played at one minute per move so there's 60 times more time and you can imagine that we'll add quite a number of nodes here and you know if if your p and v would be perfect then it wouldn't matter as much how much time you have as long as you sort of have enough time but since they're not going to be perfect since they're only neural networks they're not uh god or schmidt hoover um they cannot accurately extremely


accurately predict the future so this planning the the more you plan the more you actually look into the future the bigger your tree becomes the better moves you make so on the left you see the distributions of winds losses and draws for one second per move and on the right for one minute per move so both white and black pieces here are played by alpha zero so it's not alpha zero against something else this is playing against itself and you can see in uh in classic chess it's it's quite it's quite saddening actually um


that this game which is is so famous you can see that in of 10 000 plays 8 820 end in a draw which means that if both players are super duper good and uh and and play you know play against each other it most likely is going to be a draw and this i think is the criticism even in human chess is that it's not really a decisive game in that it ends a lot of times in a draw so one of the motivations here would be can we find a rule set that is maybe more decisive so that's one of the investigations they do in the paper but


you can see that there are actually so if you consider this torpedo chess right here um there it is more decisive as you can see in more times either white or black winds right here um and there are others which are even less decisive like pawn back so when pawns can move back then uh players may just camp they like move a pawn forward and move it back again and that will lead to a lot of closed plays and so on whereas torpedo makes you move much faster you can advance your pawns much faster and that will probably lead


to the end much faster so if you consider this on the right so what changed the rules didn't change alpha 0 didn't change it simply changed that we now let alpha 0 think for longer and you can see that the decisiveness reduces dramatically so whereas 88 resulted in a draw with one second per move now 98 result in a draw with one minute per move and this is a trend throughout these games and that's also what they say in the text it is to assume that if you let alpha zero plan for even longer that this trend will continue and


ultimately whatever rule set you make the result is going to be a draw um if two two let's say perfect players play against each other which is a bit which is a bit saddening right because um yeah that ultimately ultimately means that all of these rules aren't decisive it's only they're only decisive due to the fact that either um one or the other players is way better or or that in general that they are not they are not perfect um which is an appeal of the game but there are certainly games that are decisive even though both players


are pretty high level i mean think of every every competitive video game um so yes so that's a bit of my criticism all of this all of this needs to be analyzed in the background that what's actually happening here is that we're dealing with imperfect decision making due to a limit in resources okay and this assumption now is already a little bit invalid right the assumption we made at the beginning why i pointed this out is that alpha zero can solve these games let's say to perfection and here when we analyze the


decisiveness and so on it seems to be purely or largely a factor of um how much time alpha zero has to think about the moves and these two things to me they don't really go go together because we don't know if for a different rule set um you know the training is harder or might take longer and so on or that this exact one second makes a difference or not it's it's just um there are so many variables here and when you're dealing with let's say imperfect systems that are not trained to the end or evaluated in their


full potential you're always dealing with the fact that you stopped each thing at some intermediate point and that intermediate where that intermediate point is can influence the results drastically now here it seems at least the ordering isn't changed by much but um yeah this is one let's say one criticism the other criticism here uh that that i would have again is the fact that if you consider something like torpedo where you can move much much faster then yes of course uh let's say i don't know is it more interesting


that's that's the question right here so they look at a lot of things like decisiveness diversity and so on but the question is is it more or less interesting to play and i think that's what humans are really after and they're sort of trying to find proxies to this um i would argue if you play something like torpedo the game's maybe much faster and um so you you get to the end faster but also maybe might not be as interesting even though it's it's faster uh because your the complexity is is less and with respect to the decisiveness


here so if you have a game that's faster um you also need to take this to into account because here is another thing that is sort of an arbitrary choice as moves are determined in a deterministic fashion given the same condition diversity was enforced by sampling the first 20 plies in each game proportional to their mcts visit counts so what does that mean that means that if you run alpha 0 on the same situation on the same tree sorry on the same board position it will always come up with the same move except for parallelism inconsistencies


and so on but it will in you know in in a lot of times it will come up with the same move so how do you play 10 000 games because you can just play one game because each game will be the same because you simply tell alpha zero give me your best move right so it will just play its optimal strategy and all the games will be exactly the same so there's no reason why these should come out different so they enforce diversity by saying okay okay in the first 20 moves of a game we don't actually take the best move right usually you have


you have this distribution at the end of the tree search you have a distribution where you say okay this move right here is clearly the best move i'm going to play this however if this is one of the first 20 moves of the game they say no we need a bit of diversity uh so we're going to sample according to this distribution rather than just play the best one now this number 20. it's just sort of decided arbitrary right and if you consider something like torpedo it's a faster game so you're faster in opening faster make


and you're faster to the end game maybe even though they say well the game length isn't affected this much it could just be that um you're faster in a situation where um you're kind of forced to do certain moves and maybe the difference in decisiveness here is simply a result of the combination of the faster uh moves in torpedo together with this the fact that they just keep the 20 plies for each game again this is something that you need to consider when analyzing these results right here and there are a number of these choices


um right here like the one second or one minute per move we sample for the first 20 plies before we play the max move that where i think the the results of the study right here they have rather limited interpretability if you if you ask me because um because of these of these choices now of course they're still the results are quite plausible believable and the idea is really cool to explore these rule sets but this was this is just my criticism right here so we'll go through the rest of the results pretty pretty


quickly because a lot of people aren't chess enthusiasts and we'll just pick out kind of the core messages that the paper is trying to get across so here the table again with respect to decisiveness and you can see even uh for so for classic chess it's a white has a 50 this is the empirical score for white under different game conditions so 50.8 percent means most of the time it's a draw so white wins uh with a probability of 50.8 uh most of the time it's a draw and you see even like the most decisive variant torpedo right here


is a 54 only um so they they analyze different defenses and how the decisiveness is with respect to different defenses that are not really popular under classical chess and the results are interesting if you play chess but i would say they're rather they're kind of aha okay if you do not play chess because they consider individual moves and so on what is an interesting part is um this right here where they look at they look at one move that in classical chess so e4 is a very very um popular opening where you move your e


pawn twice for white and nf3 is not a super popular opening and here they compare this in classic chess and in no castling gesso this thing right here is a histogram and the histogram shows you the log probability of opening sequences when you play the individual moves so what does this mean right here if you play e4 then the distribution is something like this which means that you have some sequences that have no entropy at all which means that once you play e4 and maybe one move more then it's almost it's almost determined


what you have to do according to alpha zero you have like no choice except play these few next moves um however if you play nf3 then alpha zero says look this distribution is much more to the right which means that you have a lot more options here now again this could be because the move is actually less decisive because the move leads to more balanced more interesting situations where you can continue however you know with many choices it could also be because it's simply alpha zero simply doesn't know as well what to do because


it leads to more complicated games and you get to give each move one minute to evaluate alpha zero might just not be as good in those situations because it leads to more complicated situations if it could search for longer maybe this distribution would shift over here just as well again we don't know because you only give this one second or one minute each time for both um and again this goes under the assumption of alpha zero as this perfect player however back to what they want to say here if you do this in no castling chess you


can see that uh this spike right here are all the these berlin defense variants and castling this 0 right here is a big part of that line if you do this in no castling chest you can see that these two moves now the histograms overlap much more which means that and in fact you can see in the in this number of possible moves right here that they come closer together so not only does the blue shift to the right the orange actually shifts to the left and it basically means that whether you open with e4 or knight


f f3 you are going to have about the same complexity of game the same number of moves available to you going from there as you can see right here these lines are the moves available for white and black under the different rule sets so in e4 here especially as black you do not have many moves available as white a little bit more but also not more um whereas in no castling you do so again small rule change uh big effect on the possible moves that you have can consider and this is the type of this is the type of information


that you would want to have when you design a game and they allude to this also at the end here in their conclusions so the last thing is they also compare the material values of the pieces here in the different rule sets as you might imagine so some pieces become much more or less valuable i find it particularly interesting that if you do something like pawn sideways or then where the pawns are much more powerful of course all the other pieces drop in value again these results are pretty plausible so i don't want to trash the


paper right here because it seems like it seems like the the results are as i say plausible and can give some cool insights so the chess master also gives um gives his opinions on these different strategies that alpha zero comes up with for the different rules and let's go through the conclusions real quickly so they say assessing the consequences of rule change in the game design process demonstrate on chess where we've trained alpha zero to evaluate nine different variants representing atomic changes to the rules


of the game training alpha zero model on these rules changes helps us effectively simulate decades of human play in a matter of hours and answer the what if question what the play would potentially look like under developed theory in each chess variant we believe that a similar approach could be used for auto balancing game mechanics in other types of games including computer games in cases when a sufficiently performant reinforcement learning system is available and yes this is i mean this the application here would


be for something like this if you design a new game then you want to know what you have some choice with how you can make the rules and you don't want to let humans become really good at each of the rules and then compare you can simply give this to the algorithm and the algorithm will tell you what kind of plays result from each rule set and then you can choose the one that you find most interesting or most uh maybe commercially viable and what not i actually see this much i see this bigger than just games and this alludes a bit to the


salesforce paper on this ai economist i think we can let ai you know get tell us what happens if we change for example uh things like tax policy or any any sort of policy i know humanity is very complex to model and so on and you're never going to have a perfect simulator which probably makes alpha zero not good but in limited situations like maybe also stock trading rules and so on you could definitely have situations where the rule set is too complicated to solve analytically but you could give it to an rl algorithm and see


what happens and whether or not you like the outcome and whether or not there are any like obvious exploits that uh you did not see so this i find you know pretty it's it's a pretty cool approach and and we should think of this in the future as we build systems that have rules in whatever capacity be this games or policy so the they say okay yada yada yada we showed that there are several chess variants among those considering the study that are even more decisive than classical chess meaning torpedo chess semia torpedo


chess no castling chess and stalemate equals winches we quantified a rising diversity of opening play and the intersection of opening trees between chess variations showing how different the opening theory is for each of the rule changes yeah they again this this diversity of opening play it really rests on this assumption that alpha zero is a is a good player and any sort of an equally good player in all of these variants right because if it's worse in a variant it might not be as sure about the moves and that would just look like


oh you have many possibilities but in fact alpha zero is just worse at it and it doesn't know so they also look at the intersection of opening trees like if you change a rule how does this change um change the the kind of how does this change the the initial game so a lot of these grandmasters they learn by heart all of these opening trees the initial moves of a game how much would they have to relearn there is a negative correlation between the overall opening diversity and decisiveness as decisive variants


likely require more precise play with fewer plausible choices per move again this is one view right the other view is that there are rule sets that are just make it into a harder game and then alpha zero given the same amount of compute is a worse player and therefore it can't play as well therefore the games are less decisive and also the opening diversity is higher because it doesn't know if the game could be as decisive it might just be an effect of alpha zero for each of the chess variants we estimated yada yada okay no castling chess being


the first variant that we analyzed has already been tried in experimental blitz grand master tournament in chennai as well as a couple of longer grand master games our assessment suggests that several of the assessed chess variants might be quite appealing to interested players and we hope that this study will prove to be a valuable resource for the wider chess community i yeah i don't know is is the chess community flourishing or going under recently because it seems to me like it once once a game is solved that hard


by computers i mean it's still fun but um yeah i just i just i guess counter strike is also solved by bots real hard uh it's just impressive when humans play or so um yeah i don't know all of this is again if you're into chess look into this paper they have a lot of really interesting results that are not interesting to go into for the general community but i believe this should give you a good impression of what you could do if you design a system that is built on rules all right so this was it for this paper i hope you enjoyed this


next our each other I renege we'll talk talk about our fine spline insight into the attorney Oh Mike max it's great to be here how can everybody hear me at the back okay yeah it's great to be here really looking forward at this meeting that is part of a program on foundations of deep learning and and what I'd like to do is just talk a little bit about some of the progress we've been making trying to find a language a language we can use to describe what we're learning as we scratch away at these black box deep learning systems that have been

Mad Max: Affine Spline Insights into Deep Learning


promised to catapult us into going into the end of the future right and what I'm gonna argue is splines provide a very natural framework for both describing what we've learned but also providing us avenues for extending both the design and analysis of a whole host of different deep learning systems and so I'm going to talk a little bit about a particular kind of spline today and then I'm gonna give you a whole bunch of apples of how we've been using it to describe and extend and of course we're not the first people to think about the


relationship with deep nets or neural nets and splines this goes way back to pre the the previous deep net winter but I think that we have you know particularly we've identified a combat a collection of particularly useful splines for modern deep nets okay so just let's jump in and talk about the basic set up so we all know that deep nets solve a function approximation problem we're trying to use training data to approximate the prediction function from data to some prediction might be a regression problem might be a classification problem and we


do this in a hierarchical way through a number of layers in a network and what I'm going to argue is that deep net do this in a very particular kind of way using a spline approximation so show of hands how many people here know about splines okay so there's two key parts to a spline approximation the first is a partition of the input space or the domain so if X is just a one-dimensional input variable then we have a partition Omega in this case we're splitting the domain up into four different regions we're now the second important thing is


there a local mapping that we use to approximate some in this case function so we have a blue function we want to approximate here and we approximate it we're gonna be interested in piecewise a fine or piecewise linear splines by just in this case four piecewise of fine mappings okay makes sense to everybody but really this yin-yang relationship between the partition and the mapping that that works the magic in splines there's two big classes of splines there's the really powerful splines for example free not splines this is where


you let the partition be arbitrary and this and then what you do is you jointly optimize both the partition and the local mapping these are the most allow you to have the highest quality approximation but it's important to note that they're computationally intractable in 1d in fact in hired even dimensions to an above it's not even clear how to define what a free not spline is so these are something we'd really like to be able to do but very very difficult typically what people do is they they fall back to some kind of gritting type


technique and if you think of even what wavelets are they're really just a dyadic grid approximation type of spline so what we're going to focus on today is a particular family of splines that we call we don't call that we're kind maxify splines by Stephen Boyd a number of years ago and these were developed just for the idea of approximating a convex function with a continuous piecewise affine approximation okay so let's do continue with this really simple example to just define a max of fine spline we're interested in approximating


this could convex function over R capital R regions so we assume that we have our affine functions these are parameterised by a set of slopes and a set of biases we're gonna have our set to those here's an example for R equals four we have four separate four of these distinct affine functions and if the key thing about the reason why we call the maxify splines is very conveniently if we want to approximate a convex function by these splines all we have to do is take the maximum the vertically highest in this case I find function okay so if


you think of these four f-find functions that we thrown down here and we think of approximating this blue curve all we're going to be using is simply the top right the piece that actually sits on the top okay and so the really important thing here is that it's just by fixing these four sets of slopes and biases we this automatically generates an implicit partition of the input space right yet you switch from one partition region to the next whenever these affine functions cross and that's gonna be you know really important for laters this makes


sense to everybody very very simple right of course this also gives a continuous approximation so let's just think a little bit about pointing towards deep nets without going to a lot of details just imagine so we're still in one dimension and we take our input X we scale by a add a bias B and then pass it through a ray loop right this operation here well it's pretty easy to show that this is a max affine spline approximation with R equals to a find functions the first is being 0 0 is the flat function 0 function and then the second being


basically that think of this like a like like the Ray Lu but now shifted over and with the slope change by the parameter a two okay so just this should get your yourself thinking about other deep net operations and whether they can be related to the max affine splines we're going to define a max if I'm spline operator simply by concatenating K of these max affine splines so you can think of an input vector now we're no longer in 1d X is in D dimensions and then we have K different splines the and the output of each of those splines will


just be one entry of this output vectors Z we're gonna call that Amazo or a max affine spline operator so what's the key key realization well let's start by just talking about deep nets okay if you think of the lion's share of what the deep nets that are used today basically any architecture you can think of using piecewise linear or you know affine operators fully connected operators convolution operators leaky Ray leaky Ray Lu or Ray Lu absolute value any of these types of pooling z-- these are all built these the state-of-the-art methods


are all built out of these kind of architectures and these kind of operators and it's actually pretty easy to show that all of these operators that comprise the layers of bit essentially all of today's state-of-the-art deep nets are maxify spline operators you can think of then each layer of a deep net is just a max affine spline operator and so that what we're doing is we're doing that we have a convex approximation going on at each of these layers and therefore a deep net is just a composition of maxify spline operations


no longer a convex operator because composition of two convex operators isn't necessarily convex okay so so this is gonna so we're just going to call this in a fine spline operator it remains continuous but it doesn't have this max affine spline property anymore I just as an aside if you wanted the overall net to be convex it's pretty easy to constraint to show that all you need to do is just ensure that the all of the weights in the second layer and onward are positive number right that guarantees that the overall


mapping will be convex any questions about this very simple baseline stuff okay so they'll recall the nice thing about these these particular splines is that as soon as you fix the parameters right the slopes and the offsets wherever those hyper planes are these affine varieties cross that defines a partition and that's really where things get interesting right is to think about the partitioning that goes on in these maxify spline operators because that allows us to think a lot about the geometry of what's going on in a deep


network so again just reiterating what I just said if you think about a set of parameters of a deep network layer they're gonna automatically induce a partition of the input space of that particular layer into convex regions and then if you compose several of these layers we're going to form a non convex partition of of the input space and and this provides really interesting non-trivial links to classical ideas out of signal processing information theory computational geometry namely ideas like vector quantization k-means and and