the idea that the whole object has a compound color, which might be called pale green or


GLOM: How to represent part-whole hierarchies in a neural network (Geoff Hinton's Paper Explained)

move. And at the object level, every location belonging to the object has exactly the same


compound color. So the object is whatever this all over. When deciding which other locations


the object level attend to preference would be given two locations with a similar compound


color. So what he's saying right here is that, you know, you could give preference to two


similar color locations when you decide what you want to attend to. But the color isn't


as easy as simply saying what color is there in the location that you are at. But you could


be so if this is green, and this here is blue, then the bottom layer would say yes, I'm green


and yes, I'm blue. But they could also be saying, well, I am part of a green blue object,


right? And then the the higher layer here, you know, attending or caring about multiple


or bigger region, its color would then be, you know, green, blue, and the consensus could


reach on, well, we are a green blue object, even though the object isn't a pure green


or pure blue all throughout. So he, I think, yeah, it's it's, I think it's a side suggestion.


Maybe he has this as a core motivation between the system. But it's just interesting to see


how he thinks of things and he extends the color here to textures and even shapes. The


individual texture elements have their own shapes and poses in spatial relationships,


but an object with a textured surface has exactly the same texture everywhere at the


object level. glom extends this ideas to shapes, an object may have parts that are very different


from one another. But at the object level, it has exactly the same compound shape in


all of the location that it occupies, basically saying that, okay, every pixel that's part


of a cat head is a is a cat head has the shape of a cat head, even though the individual


locations might not recognize that. And that information could be passed around through


this consensus mechanism over time. So the cluster discovery versus cluster formation,


we've seen that and he makes a lot of, he makes a lot of analogies to face recognition.


But yeah, the clusters are not the islands of similar embedding vectors at a level can


be viewed as clusters, but these clusters are not discovered in immutable data. They


are formed by the interaction between the intra level process that favors islands of


similarity and dynamically changing suggestions coming from the locations embedding at adjacent


levels. So the core here is really this consensus algorithm that creates these clusters. And


yeah, the clustering algorithm doesn't work by simply looking at embeddings and deciding


which ones go together, but the embeddings themselves update themselves in order to form


clusters. And yeah, this is a replicating embedding vectors. This is a response to a


criticism that I guess he got where someone said, Well, why don't Why do you represent


if you have these, you know, these columns at the bottom, it makes sense, you have all


the different vectors, but then as you go up, you know, you have that kind of the same


vector for all locations, because it's the same object, why does it make sense to replicate


that everywhere, and not just have one, because, you know, in a database, we just have one.


And it basically says that, in order to reach the consensus, first of all, it's important


to have different vectors, they might be slightly different. So they might have some nuance


in them, because, you know, they might get pulled into different directions from the


sign of bottom up signal, then from the consensus algorithm on the same layer. So I, you know,


I believe that it is that is important. Here, I think it's just this is a criticism he got.


And then he decided to put this in here, learning islands. So what we haven't discussed about


this yet is how this is trained. And Hinton says this is trained as a denoising auto encoder.


Let us assume that glom is trained to reconstruct at its output, the uncorrupted version of


an image from which some region has been have been removed. So he goes into self supervised


learning with this system. This objective should ensure that information about the input


is preserved during the forward pass. And if the regions are sufficiently large, it


should also ensure that identifying familiar objects will be helpful for filling in the


missing regions. To encourage islands of near identity, we need to add a regularizer. And


experience shows that a regularizer that simply encourages similarity between the embeddings


of nearby locations can cause representations to collapse. All the embedding vectors may


become very small, so that they are all very similar. And the reconstruction will then


use very large weights to deal with the very small scale to prevent collapse. And then


he says contrastive learning is the answer to this. So how do you regularize the model


such that this consensus is formed? He says contrastive learning might be useful, but


you can't simply apply it straight out. So it learns to make representations of two different


crops of the same image agree, and the representations of two crops from different images disagree.


But this is not a sensible thing to do. If our aim is to recognize objects, if crop one


contains objects A and B and crop two from the same image contains objects B and C, it


does not make sense to demand that the representation of the two crops is the same at the object


level. Okay, so he says that contrastive learning is good, but you have to pay very careful


attention at which layer you employ it. Because, you know, if you go down far enough, then


contrastive learning, especially, you know, this this type where you crop the image into


different parts, and you say, well, since it's the same image, the representations should


agree in would say, well, at the top layer, yes, but at the bottom layer, certainly not


because they display different things, right. So you have to be careful where you apply


this contrastive learning. And he gives a bunch of suggestions on how to solve that.


He says things like, well, negative examples, for example, might not might not even be needed.


Well, that's it. Sorry, that's a different thing. So the obvious solution is to regularize


the bottom up and top down neural networks by encouraging each of them to predict the


consensus option. Yeah, this is the way to geometric mean of the predictions coming from


the top down and bottom up networks, the attention weighted average of the embeddings at nearby


locations at the previous time step, the previous state of end, I guess, and there should be


an end, and the previous state of the embedding, training the inter level prediction to agree


with the consensus will clearly make the islands found during feed forward inference be more


coherent. So he says you could regularize the model to, to regress to the consensus


option. So it's sort of like a self a self regression. And he asks whether or not that


will lead to a collapse. Because if you don't have negative examples and contrastive learning,


this could lead to simply a collapse. An important question is whether this type of training


will necessarily cause collapse if it is not accompanied by training the inter level predictions


to be different for negative examples that use the consensus options for unrelated spatial


contexts. So here is that problem, right? If you use the consensus opinion for unrelated


spatial context, that might be a problem. He says using layer or batch norm should reduce


the tendency to collapse, but a more important consideration may be the achievability of


the goal. It goes into why regularization could help. And he says, if however, an embedding


at one location is free to choose which embeddings at other locations, it should resemble, the


goal can be achieved almost perfectly by learning to form islands of identical vectors and attending


almost entirely to other locations that are in the same island. And I don't know, I don't


know if this is what I suggested. So I guess this is kind of a convoluted paragraph. And


I had to also read it multiple times. And I still don't exactly know what he's trying


to say right here. But I think what he's saying is that what we want to do is we want to sort


of regularize the network to produce this consensus, right? So we have a bottom up signal,


a top down signal, we have a current value, and we have the signal from the attention


mechanism. Now, what we want to do is we want to reach a consensus such that these islands


form. However, if you attend to any sort of things here that have nothing to do with you,


you might not be able to reach this consensus, right? That's I think that's the problem I


think he's touching on the problem that I said before. So what he says is, you know,


what you should do is you should simply attend to things that are in the same islands already.


So if an embedding at one location is free to choose which embedding at other locations,


it should resemble the goal can be achieved by learning to form islands of identical vectors


and attending almost entirely to other locations that are in the same island. Now, I think


here, what he's doing, he makes the case for the attention mechanism itself, right? So


he says, if if we simply draw in information from the same layer here, you know, anything,


any old information might come in, and we might collapse and or we might never reach


consensus because any old information might come in. However, if we introduce the attention


mechanism into this whole thing, and only draw in information from the selected neighbors


that already are in the same group in the same island as me, then this consensus algorithm


works. So if the network, the network is now forced kind of to learn to build these islands


of similar things in order to make this consensus work if we regularize this consensus. So I


believe he makes the case for the attention mechanism. I don't think he, in this case,


considers kind of the up the next up layer islands, what I would say is you need to consider


the island membership all the way up the columns in order to decide which things which locations,


right, it's free to choose which embeddings at other locations it should resemble. I think,


yeah, this is the case for the attention mechanism. Okay, I hope you're still half with me. If


not, I'm, I'm bit confused too. But I think what he's doing is he says, contrastive learning


would be good, you can use it, but you have to be careful at which layer you do it. Another


regularizer to form these islands would be this regularize the network to conform to


the consensus option, opinion. However, if you simply aggregate information from the


same layer, then that wouldn't work because, you know, the different things in the same


layer might correspond to completely different parts of the image. Drawing in information


from there would not help you. How do you solve this by introducing the very attention


mechanism that he introduced in order to only draw in information from parts of the same


layer that actually are related to you? Okay, the next thing, the next consideration he


does is representing coordinate transformations. How does this represent coordinate transformations,


there was a capsule net paper where he explicitly represents coordinate transformations in kind


of four dimension quaternion space. And he says, that is probably not needed, because


you don't want to hear says you could represent this by a by four by four matrices. However,


if you simply allocate 16 numbers in each embedding vector, in order to represent the


part whole coordinate transformation, like the transformation that relates the part to


the whole, that does not make it easy to represent uncertainty about the aspects of pose and


certainty about others. So the problem here is that we know that humans, when they watch


something right here, when they watch a scene, like this is a chair, and there is a person,


a very tiny person on the chair, we don't see necessarily the coordinate frame of the


world, what we see is we see the coordinate frame of the chair, like maybe this is the


center, and we see the person in relation to the chair, our brain seems to do this intuitively,


and hinting things that a system like this should also do it intuitively. So somehow,


the coordinate transformations involved going from the eye to the reference through the


frame of the chair, and then from the chair to the person, they should be somehow in encoded


in this network. However, he also says that it's probably not necessary to encode them


explicitly as you know, explicit coordinate transformations, because not only does that


make it harder, probably to learn, but also, you can't represent uncertainty. In fact,


you can represent uncertainty, that's the next thing right here, much better by having


a higher dimensional thing that you're trying to guess, right? If you are trying to guess


a distribution with three components, and you simply have a three dimensional vector,


you have no way of representing uncertainty. However, if you have a nine dimensional vector,


you can have three opinions about the distribution. So this is an opinion, this is an opinion,


and then this is an opinion. And then you can sort of aggregate and you can say, Well,


I'm pretty sure about these two things, because all my opinions are pretty close. But this


one here, I'm not so sure because my individual things say different things, things say things.


All right, I've this video is too long. So that's his argument right here, we don't need


explicit representing of uncertainty, because by simply over parameterizing, we can already


represent uncertainty well. And we also don't need disentangled position information and,


and so on. Sorry, we don't need different position informations, because, again, the


work can take care of that. And he gives a good example, like why would you have disentangled


coordinate frame if you have an image? And in the image, the picture in it is this. How


do you know if that is a rhomboid shape? Or if it is a rec, if it is a rectangular piece


of paper viewed from the side, I should probably draw it way closer, something like something


like this. I suck at this. You get probably get what I mean. Like, if it is a different


object, it has a like the object and the coordinate transformation are dependent upon each other.


And so it makes sense for the neural network to actually entangle the two, because the


two things depend on each other. In essence, he's just saying, don't worry about explicitly


representing all of the different things. We got it like the neural network can do all


of these things, like uncertainty or position, and post transformations. So here he compares


it to different other architectures. comparison to CNN comparison to transformers comparison


to capsule models. And at the end, it goes into video at the very beginning, he says,


the paper is about actually a video system. And you can kind of see that because we go


through this algorithm in multiple time steps, right? You have, it's like you analyze an


image with these columns, which gives you sort of a 3d 3d tensor with the image at the


bottom. And you go in the next time step, you have a new 3d tensor, right, you pass


this whole information around with the image at the bottom. And it says, well, why does


that need to be the same image that could also be different images. So you could use


the system to analyze video. So what he does is he says, at the same time, you do this


time step to find agreement, you could actually swap out the video frame, the X, you can swap


out the video frame, and produce a slightly different video frame. And you could actually


have a kind of an ensemble regularizing effect. So as the whole columns here, the whole system


comes to a consensus over time, you feed in different information at the bottom. And what


he says is that, you know, if this is a slow enough video, then the top layers here would


probably could still reach an agreement, while the bottom layers would change rapidly. But


that could be sort of an ensemble or a regularizer, regularizing effect that it even has. So he


intrinsically connects these two time dimensions, because they would be separate, right, you


could input a video. And then in, you know, in each frame, you could do this consensus


finding algorithm. But he says, No, it's actually cool to consider them together to do the consensus


finding while you sort of watch the video, it's just not clear that you always need the


same amount of consensus finding steps as you need as you have video frames. So maybe,


you want to, maybe you want to take like five consensus steps per video frame, or the other


way around? Not sure. In any case, I think that's a pretty cool idea. And he says things


like, if the changes are rapid, there is no time available to iteratively settle on a


good set of embedding vectors for interpreting a specific frame. This means that the glom


architecture cannot correctly interpret complicated shapes. If the images are changing rapidly,


try taking an irregularly shaped potato and throwing it up in the air such a way that


it rotates at one or two cycles per second. Even if you smoothly track the potato, you


cannot see what shape it is. Now I don't have a potato, but I can give you an avocado. So


if you give me a second, how's that? Could you track the shape? I don't know. Probably


in his correct. All right, he talks about is this biologically plausible? And I don't


want to go too much into this. He discusses some restrictions like yeah, we still use


backprop and is backprop plausible and so on. I love this sentence. In the long run,


however, we are all dead. And then the footnote saying there are alternative facts. But yeah,


he discusses whether it's biological plausible. How could you modify it to make it more plausible?


For example, when you want to do contrastive learning, there is evidence that dreams during


so during sleep, you do contrastive learning, like you produce the negative examples during


sleep, and then during the day, you collect the positive examples and so on. So I think


this is a more speculative part of the paper, but it's pretty cool to it's pretty cool to


read it. And lastly, he goes into discussion. He also says that this paper is too long already.


I'm gonna just briefly talk about this. And he trashes the neuro symbolic people a bit


like he trashes the people that say no, no, you know, neural networks can never do whatever.


And he says pretty clearly look, neural networks can represent trees, I've given you a system


also BERT can output parse trees. So shut up, I guess. And he comes up with this glom


BERT name, which, you know, is is already coined. If you wanted to do glom BERT, that's


already taken Sorry. I also by the way, also coined the I coined the name may glow mania


right now. Okay, if you want to, if you want to use it, it better be a pretty cool machine


learning system and be based on glom. Right? That was the paper. I think it's a cool system.


It has a bunch of parts that are maybe not super friendly to hardware at the time like


this iterative procedure. But honestly, it is not much more than a neural network. Sorry,


a recurrent neural network with very complicated recurrence functions. The video extension


might be a bit tricky. And, but the rest and the regularization might be a bit tricky,


the exact objective. So the denoising auto encoder objective isn't super detailed in


the paper, he simply says, reconstruct a corrupted version of the input. How exactly the input


happens, maybe there's a CNN, maybe the CNN feeds information into actually multiple layers.


None of that is exactly specified. So there's lots to figure out. I do think the ideas are


very cool. And I love idea papers. And therefore I recommend that if you're interested more,


give this thing a read, give this video a like, share it out. And I'll see you next