problem that's something else i i changed my mind on you know the last two years uh this may be outside the scope of this particular interview but um yeah there's there's basically two major architectures right for self-supervised learning particularly in the context of vision and the the main uh characteristic of both of them is how they it's the fact that they can handle multi-modal prediction right so if so if you have a system let's say you want to do video prediction or something like that right so you have a piece of a video clip you want to predict the next video clip or you just want a machine that tells you whether a proposed video clip continuation clip is a good continuation of the previous one you don't want it to predict you just want it to tell you although it's a good one okay so you have two architectures the first one is elite and variable predictive architecture that predicts the next video clip and of course you have to parametrize the prediction with a little variable because there are multiple predictions that are plausible okay so i used to be a big fan of that and about two years ago i changed my mind maybe a year and a half um the the other approach is something i played with in the early NUMBER_TOKENs came up with some of the early models for this and it's called the joint embedding architecture where you have the the first and the second video clip both going through a neural net and then what you're doing is you're training the system so that the representation of the second video clip is easily predictable from the representation of the first one okay so they are you're not predicting pixels you're predicting representations uh but you still have a system that can tell you here is a video clip and a second one tell me if they are compatible if one is a good continuation of the other the the main the main reason why i stayed away from those architectures because i knew that you had to use um in the past you had to use contractive learning right you have to add pairs of things that are compatible as well as pair of things that are incompatible and in high dimension there's just too many ways two things can be incompatible and so that was kind of due to failure and and you know i played with this you know back in you know this used to be called siamese networks right in the paper in NUMBER_TOKEN NUMBER_TOKEN on doing signature verification using those techniques jeff intern had a a slightly different method based on maximizing mutual information with uh his former students to becker um but then in the last two years we've had methods that are non-contrastive that that allows us to train those uh those joint-embedding architectures so i've become a big fan of them now and it's because of algorithms like uh like byol from our friends at dpine like barlow twins who whose idea came up came from uh stephan denis who was doing a postdoc with me at uh at fair he's now uh uh as it's a professor at university of alto in uh in finland and uh and then more recently v craig which is the a um kind of a improvement if you want on valor twins that is also based on the the same idea that jeff
NO_SEGMENT_TOKEN