So let's dive into how they do it. And they say it's deployed in portal, our video calling service and available for use across a range of other Facebook applications,


[News] Facebook's Real-Time TTS system runs on CPUs only!

from reading support for the visually impaired to visual reality experiences.


Okay, so first, they show this graph right here, what are they doing?


So this is their entire system, it is chunked in multiple parts.


If you're a deep learning practitioner, you're very, very keen on taking this text and just like giant neural network and just run it through and generate audio end to end, right?


This doesn't really work in this case, first of all, because it would be too much of a, too many parameters to evaluate this many times.


But also, especially text and audio are such different modalities that you'll have to basically chunk this into individual parts.


And that's what they do. So they have this linguistic front end.


And the linguistic front end generates two different things. It generates what needs to be said, and how does it need to be said.


So what needs to be said, they call this linguistic features, the linguistic features, they're things like phonemes, and so on.


So that's, sorry, they don't even have a title for this one.


So the linguistic front end converts the input text. So this is probably it's a sentence by sentence thing.


So this is one sentence of text into a sequence of linguistic features such as phonemes and sentence type.


OK, so these linguistic features are if it's like share on Facebook, it would be like, OK, is one and then a and then these are the phonemes of share.


It would kind of chunk it into that so that that is much closer to what we think of an audio signal.


This will make one sound. This will make a sound and this will make a sound.


So we sort of chunk it into that and then hear the how is it said.


This would be, for example, the fact that share on Facebook is sort of an instruction.


And this is different from if this would be a question, like if it said, do you want to share this on Facebook?


Then the what would output much the same phonemes right here, except it's different word now. But but the how would output a this is a question.


So as you can see, the information flow right here informs the later stages.


This would then cause at the end of the sentence the voice to go up because it's a question.


So it is this linguistic front end. As you can see, it still deals with text.


It deals with text and it outputs these these how features and these linguistic, these what features.


So the what features now go into an acoustic model. And what does the acoustic model do?


The acoustic model is meant to generate a spectrum, a spectrogram of the sound.


Now we'll skip this for the moment and go to the neural vocoder.


The neural vocoder is a kind of a standard thing in text in any speech producing.


It takes a spectrogram of the sound and turns it into actual audio.


So this here, I think they achieve it with they say something similar, like a wave RNN.


Based on plus like a CNN.


So we'll look into that quickly. So the spectrum, the spectrogram of the sound is going to be a bit of an image.


And the image has time on the this axis and frequency on this axis.


And then there is a a it's usually some somewhat like color coded, but it's just intensity.


Right. So whenever there is whenever a frequency at a given time is expressed strongly, it will light up.


So it could be something like this.


So over time, the mid frequency is always there, but this frequency right here is not there.


And this is here at the beginning. There is a sort of a way to read the spectrum.


But this represents maybe something like this represents the not even sure how much sound this represents.


But this can represent maybe something like 200 milliseconds of audio.


And you have to basically perform a Fourier transform to transform that into 200 milliseconds of actual wave audio.


But since the audio has to be output at what this 24000 samples per second, what's coming in here is not that much.


The acoustic model outputs not 24000 spectrums per second.


So first of all, this has to be up sampled. So this time dimension here has just too few samples.


So we use the CNN because this is basically an image in order to up sample this in order to make a long image out of this.


And basically the CNN will have to impute. It will have to. This is learned. Right.


This is learned how this spectrogram would look if it were sampled much more densely.


And now it has basically the correct number of samples here. And now this wave RNN can step through it.


And look at these slices and look at the last slices or actually also can look at the last spectrograms maybe.


I don't know. They don't really say what the RNN exactly goes over. But this is how I imagine it.


And the wave RNN is based on a WaveNet architecture. And that means WaveNet is sort of like if you look to produce this thing right here,


you can look back all the way to the beginning. But this would be too many connections.


Right. It would be too much memory. So what you do is you can look back at the directions at the things right before you in very great detail.


But then as you go back further and further, you basically lose detail. See, there's only two connections here in this long stretch where there is many,


many connections here at the very beginning. So right before you.


So that means you look in more detail what you've produced recently.


But you also sort of look back in a more blurry way at what you've produced a long time ago.


So this is a WaveNet architecture. And that's they say they use something like this in order to then actually produce the final audio from the spectrograms.


So actually neural vocoder, you can you can train this thing. You can train it by itself.


Right. You simply feed spectrums and you make it go audio. And you know, there's a lot of audio on the Internet.


You can simply produce spectrograms from that and then train the vocoder to produce the audio.


So the good thing about this pipeline here is you can train a lot of these things independently from each other.


I don't actually know whether they do that or not, but you can.


You see this box up here, this prosody model, and that will take in these how features.


So how does something need to be set? How? Oh, come on.


Well, this is an H. How does something need to be said?


And it will transform it into features that the sort of neural network can understand.


These these neural networks, they need features like they need embeddings.


And as you can see here, it also takes into account the speaker embedding, which is not only how what the sentence is,


the fact that the sentence is a question, you would also get the information that the speaker should be, you know, kind of calm.


Sorry, that would be the style. The speaker should be maybe a woman voice.


And then the language should be it should have like an English sound or a German sound and so on.


So this model here will take in all of that and emit features that these neural networks here can understand.


So as you see here, the neural vocoder, not only does it transform spectrograms to audio,


but it takes into account how you want the audio to sound.


And so does this acoustic model. So the acoustic model is sort of along with the prosody model is a bit of the heart of the thing here.


It takes in these linguistic features. So what needs to be said, right?


The acoustic model, it takes into account the again, the speaker embedding, language embedding and so on.


It takes into account the output from the prosody model of the how you would like it to be said.


This includes what type of sentence this is and this it synthesizes it all in order to come up with the spectrum right here in order to come up with these spectrograms.


So this is sort of the the heart of the of the thing.


OK, so about the prosody model, they say we use style embeddings that allow us to create new voice styles, including assist and soft, fast, project and formal,


using only a small amount of additional data with the existing data sets that we don't have to create a separate model for each style.


We need only 30 to 60 minutes of training data for each voice style.


So that happens because you you take into you take in actually an embedding of the feet of the speakers and you train these things independently.


So that means you can sort of generalize to a new style really quickly.


Here the they describe in essence how their acoustic model works.


The fact that they output 13 dimensional MFCC features concatenated with the fundamental frequency and a five dimensional periodicity feature,


which is much easier for the acoustic model to generate. So that's what the acoustic model generates.


OK, and then here are conditional neural vocoders is the final part of the pipeline consists of two components, a convolutional neural network that up samples or expands the input feature vectors from the frame rate 200 predictions per second.


So that's not it's not 200 milliseconds.


It would be 200 spectrograms a second to the sample rate 24000 predictions per second and this similar to wave RNN which synthesizes audio samples autoregressively.


That means one sample at a time at 24000 samples per second. Crazy, right?


So this all seems like it's a lot of computation and now they describe how they get this to run faster than real time.


And they list their their individual contributions here.


So if they just run this on one CPU core, you see it takes 80 seconds to produce one second of audio.


Then they do optimized inference operators, which means they basically use a PyTorch JIT along with some cities PyTorch these PyTorch JIT,


which is kind of where you can sort of compile your deep learning model to an optimized to an optimized form that runs faster.


And they also customize this so they get to 20 seconds per one second.


Then they do parameter reduction by sparsification and they are able to abuse the sparse matrix compute operator.


I think they implemented a custom one to do that.


So this here is very much like if you have a neural network and that's somehow connected, connected, connected.


You want to train it in such a way that it's sparse, meaning that only very few of these connections have non-zero weights.


And because of that, you don't have to store the non-zero weights and you don't also have to compute because something multiplied by zero is going to be zero.


So you don't have to compute that. So they're. They achieve sparsity.


Sixty ninety six percent. And this basically you do by some sort of teacher, teacher, student model or there are many ways to do this.


But you can you can sparsity regularize a neural network and basically force most of the connections to be sparse while still maintaining a good training error or a good generalization error.


So they supercharge this. And with this sparsity, they bring this down to five seconds per second and then they go further and do blockwise sparsification and distillation.


So they distill it to an entirely smaller model that then is also blockwise sparse.


So they also enforce this blockwise sparsification. And that not only can you then have a better operator,


they implement this block sparse matrix compute operator that is specifically designed to multiply block sparse matrices.


They also in the text, they describe that it also optimizes cache access.


So if you know about CPUs, they have these level one, level two, level three caches and you can optimize your computations.


That's what libraries like LaPak and Blas do. You can optimize your computations such that your cache access is optimized and therefore you can speed up a lot your computations because the amount of times that you actually have to go to your RAM and retrieve something is minimized.


And that tends to be the slow part of the process when your cache misses.


So again, they achieve 94 percent block sparsity and that almost gets them to real time.


And it's still one CPU. So now they parallelize the operators that are doing the heavy lifting to four CPU cores.


And that doesn't divide it by four, of course, because there is an overhead in parallelization and synchronization.


But that gets you to this one half a second needed to produce one second of audio.


So there you're at real time. And that is pretty impressive.


And they go on to describe in detail how they did it.


But they also give some examples of what they can do and would like to achieve even better in the future.


So here is an example where they can adapt their model to a given style.


And here they have a British accent.


Recently, we successfully applied our new approach to create a British accented voice.


This is the first of more accents and languages to come.


And also you can adapt it. Their idea is that you have sort of an assistant and this assistant will be able to adapt to, let's say, your mood.


We're also exploring features to make our voice respond intelligently with different styles of speaking based on the context.


For example, when you're rushing at the door in the morning and need to know the time, your assistant would match your hurried pace.


When you're in a quiet place and you are speaking softly, your AI assistant would reply to you in a quiet voice.


And later, when it gets noisy in the kitchen, your assistant would switch to a projected voice so you can hear the call from your mom.


Right. So this ties in very much with sort of conversational AI, so assistants and so on.


But it also ties into wearables, I think. So the fact that you are now smaller than real time means you can run this potentially directly on your smartphone.


You could run this on your fridge or something, on your stove, in your car without having to stream, basically.


So you'll get much more real time on device assistance, maybe even in your watch.


And I'm excited by this technology. So far, it seems you can get it in Facebook products, but I'm sure this will come to places.


All right. If you enjoyed this, please consider subscribing. Thank you for listening and watching.


Big Bird: Transformers for Longer Sequences (Paper Explained)

Hi there, today we'll look at Big Bird transformers for longer sequences by Manil Zahir and Gurugra


Rugganesh et al of Google Research.


So this paper on a high level proposes to replace the quadratic attention mechanism


in transformers by a mix of random attention, windowed attention and selective global attention,


therefore achieving a complexity of linear memory requirement instead of quadratic memory


requirement.


And as a result of that they can process longer sequences than traditional transformers like


BERT and achieve better results in some NLP tasks and they also evaluate on genomics tasks.


So we'll go through this paper a bit, look a bit at the proof because they give a theoretical


kind of guarantee that their random attention mechanism can still be Turing complete and


still achieve the same things as a full attention mechanism but we'll also look at the drawbacks.


I sort of have mixed feelings about this paper and I think I'll voice my concerns as we go


through here but first let's look at the paper, let's look at the architecture and I think


this is actually a pretty cool paper for the empirical progression of the field to process


longer sequences with transformers.


As always if you like content like this feel free to share it around, leave a like and


tell me in the comments what you think about the paper and about what I think, whatever,


you just go nuts.


Alright so the basic premise right here is that the transformers they've been pretty


impactful especially in NLP.


So they say transformer based models such as BERT have been one of the most successful


deep learning models for NLP.


Unfortunately one of their core limitations is the quadratic dependency mainly in terms


of memory on the sequence length due to their full attention mechanism.


So really briefly the full attention mechanism and I've done numerous videos about attention


mechanism BERT, attention is all you need and so on so if you want a detailed explanation


of what that is just go look up the corresponding videos but briefly what you'll have in NLP


is a set of tokens, a sequence of tokens as an input and you want to transform them layer


after layer into sort of a higher order representation of that same sequence and for that you build


these layers out of nodes and you have as many nodes usually as you have tokens in the


sequence and the next set of, so each token is represented by a vector at the beginning


and each layer transforms this sequence as I said into sort of a higher level representation.


So you want the vector of this token right here to be a better representation than the


vector was right here and you do that by incorporating information from all the other tokens into


that particular vector.


Now as I said this is called an attention mechanism and we don't actually have to go


into how it works right here but you can see pretty clearly that if you want to do this


for every token you need to have information routed from every token to every token like


from here to here, from here to here and so on and this is just one token and then you


need to do it for this token and for this token and for this token.


So what you'll ultimately get if n is your sequence length you'll get some n squared


amount of computation and memory requirements for this.


So this is a problem and usually this means that you know this sequence length in BERT


this is limited to something like 512 tokens which is okay for some applications but if


you want to summarize you know entire articles, entire books even or do question answering


with lots of context it's not really enough.


So people have been thinking about how to scale this input, how to scale this and of


course the main culprit is this quadratic attention mechanism because if you you know


double the 512 you need you know four times the amount of compute and memory.


So how does this paper go about reducing that quadratic dependency?


The goal right here is of course to get this to some O of n right because then as we double


the input length we simply need to double the compute requirements and that would be


fantastic and that's what this paper does and it does so without you know sacrificing


the properties of the transformer.


So here's the architecture that BigBERT proposes.


By the way BigBERT another character from Sesame Street I guess will continue the naming


here after Elmo and BERT.


I'm waiting for the model that's the count.


Yeah that's going to be a fun model but so BigBERT basically has three different types


of attention here.


These are adjacency matrices in this attention mechanism.


So here is the input layer and the output layer is right here.


So that basically means that node i right here would be connected, sorry that's not


a straight line, would be connected to this particular node and also to this particular


node.


So we're now trying if we have node i right here we're now trying to not connect it to


all of these nodes but we'll say we'll just select some at random and then connect it


to that.


Okay this is what we call random attention and you can pretty clearly see if you connect


each of the i nodes to r equals 2 to two random nodes then you don't have an n squared anymore


but you'll have an like an O of r times n which you know if r is a constant is an O


of n attention mechanism.


Okay so the main goal between the random attention mechanism is that for each query basically


you select random tokens that you attend to and that random number is a fixed number that's


not dependent on the sequence length.


And the paper is a little bit unclear about whether or not those random ones are the same


for every sequence or are switched up or are the same for every layer or are switched up


but they formulate all of this as sort of in sort of a graph in sort of a random graph.


So they formulate the attention mechanism in form of a graph.


So if we transform all of these nodes into a graph a full attention mechanism would mean


that each graph each node is connected to each of the other nodes right fully connected


graph I don't maybe that's it.


So that would be a full attention mechanism and then they say well if we just have random


connections between these things then there are some theorems from graph theory that say


that each random walk in this graph is going to so this graph is going to mix pretty quickly


so I can get from each node to each other node by a random walk in a logarithmic time


and this random walk which basically means that you go from here to here this would be


one layer of the transformer and then if you want to go from here to here that you would


have to do that in the next layer so this formulation as a random graph leads me to


believe that layer after layer the random attention pattern is going to be the same


but also the formulation of the paper leads me to believe that this random attention differs


from sequence to sequence so I believe what's happening is that they you know get a new


sequence then they decide on this pattern right here once and then they use this layer


after layer the same pattern again so you can see that in the traditional attention


information can basically flow from each of the nodes to each other node in one single


step right because each node is connected to each other node you see this in the graph


right here however if we only select a subset then you know it needs to if I want to go


from as I said from here to here then I need to do it in two steps and therefore I need


two layers and that's going to be the culprit of this method here and you know while it


is mentioned in the paper it's sort of I feel at least that's my my assessment of this paper


it's kind of swept under the rug a little bit I mean they do have a theorem that clearly


says we can construct an example of a task that in the full attention setting can be


solved with a single step so a single layer that in our random attention setting needs


a lot of layers so a lot of steps but you know the rest of the paper is sort of shaky


on on this thing but nevertheless you can see how the random attention can if you have


enough layers do the same information routing as the full attention okay however this is


not a property of the random attention and we'll see this in the next thing right here


so the next ingredient that this paper uses is window attention and you can see over here


that Big Bird is ultimately going to be a combination of the three types of attention


which will which we are looking at here so window attention basically means that each


each eye each token at the eye of position is going to attend to itself of course so


here is I but it is also going to attend to its neighbors so here is I minus one and here


is I plus one and this is a you know this is a window size W that you can that is a


parameter but also it is a constant and therefore you again go from n squared to W times n which


you know is O of n if W is a constant and this might be familiar to you because we've


already seen this in the long former papers I've made a video or I think even two videos


on the long former which used exactly the window attention in combination with the global


attention and if you want to know more about that go watch these videos but the new thing


in Big Bird right here is this edition of the random attention again the the window


here is is has exactly the same properties as the random attention so you have instead


of a fully connected graph you have a sparsely connected graph now if you have random attention


the sparsely connected graph is like like the one on the right but if you have a windowed


attention you can it is kind of not randomly connected but each node is connected to its


neighbors like this and you can also see that if I want to go from this node to this node


right here I can't do it in one step but I can do it in two steps I go here and I go


here so in the terms of the attention layers if I want to go from node one to node three


I have to do it in two steps because each node is only connected to its neighbors so


the connection patterns would sort of look like this so I have to go from one to two


and then in the next layer from two to three so the paper basically makes up for the lack


of full attention by adding layers and you all also might recognize this from a convolution


operation like this basically because it is a convolution operation right in a convolution


each node a only aggregates input from its neighbors for the next layer and then we know


that as we go up the layers the de facto window that each node looks at is going to be like


a cone kind of like this so this is very similar to how a convolutional neural network works


and the reasoning is very similar because the reasoning is well in a sentence the most


important words for any given word are probably going to be its neighbors like the words around


it and as you go up the layers you branch out more and more but ultimately the this


neighborhood principle holds in NLP as well so again we already saw this in the long former


but that's the reason behind the window attention and that's the second ingredient and then


the third ingredient is this global attention now the global attention is selected tokens


that are so important and that's you know fixed by the developers that are so important


that they are they are connected to everything else so for example in these transformers


you often have what's you know this kind of CLS token so this is a special token that


you prepend to some piece of text and the output of this token is going to be your classification


output because you don't want to bind your classification if you need to classify the


entire sequence you don't want to bind that decision to one particular word what you want


to do is you will have an extra token and that's this CLS token that kind of aggregates


information from all of this so layer after layer layer after layer you'll have so if


we go here layer after layer we have this one special node and in each step every single


other node is able to send information right here to this node and receive information


from this node okay so now as a result of this as you as you may be able to see every


single every single path is kind of maximum length of two because if I want to go from


any node to any other node I can simply you know send information to this global node


and then the global node in the next step can send information to whatever other node


and that is a property that they use in their proof that this attention mechanism is as


sort of as powerful as the classic fool attention mechanism and we'll go through that in one


second but first I hope this was clear that this combination of random attention window


attention and global attention is what is called Big Bird okay they have some engineering


tricks that go along with this but in concept you can imagine Big Bird being long former


plus these random attention right here and you know as an engineer as an NLP engineer


that makes kind of total sense I you know I totally believe that a the introduction


the addition of these random attention of these random attention patterns can absolutely


help your classification or whatever your NLP tasks because you know more attention


better and I also am completely willing to believe that you know using the full attention


matrix while it is of course more accurate it won't hurt too much to leave some of that


attention away because essentially all the path lengths are just becoming two or even


with the random attention are really short or logarithmic to route information from a


node to some other node so the loss that you incur is kind of in a logarithmic scale in


terms of performance while the gain that you make is sort of in a in a quadratic or like


a linear scale you go from quadratic to linear and that seems to me like a good empirical


trade-off all right however the the proofs here the proof of of how how these how these


things are constructed are a little bit I don't know so what they do in the proof that


this function can sort of appropriate is a universal approximator people have already


shown that full attention mechanisms are universal approximators so they show here that this


sparse attention mechanism is also a universal approximator they make big use of star graphs


what they say is okay if we have a star graph which is one node connected right here to


every other node this is a star graph if we have a star graph we can achieve the same


thing then with a full graph a full graph is where every node is connected to every


other node but as I already said what they need for this is multiple layers of this star


graph so and that has to do with the fact that if I want to route information I basically


have to go via this middle node right here and there is an additional complication because


this middle node in our case right here is only one node I can't route information at


the same like I can't have this routing right here at the same time that I have this routing


right here like going from here to here because I only have one middle node and I kind of


this is not how that like this is very dumb math but maybe you have to imagine that there


is one memory slot and you can only use that one memory slot at the same time for one of


these things so essentially what you'll have to do is you'll have to do the green thing


first and then in the next step you'll have to do the blue thing second and then so these


are now pairwise routing between nodes but ultimately what an attention mechanism does


is it does everything to everything right in a single layer it routes information from


all the nodes to all the other nodes and to achieve that you need multiple rounds of this


and it turns out that in the worst case you actually need n rounds of this so you know


you trade off your you go from n squared to n memory and compute requirements in a single


layer but in the worst case you need n layers to recover the power of the full of the full


transformer and that is the last one of their theoretical results right here so first they


prove universal approximations and second they prove Turing completeness these two properties


have been proven for full attention mechanisms and third they prove that there are tasks


where we actually do need n layers to solve them with their limited attention so you know


I'm not sure but I feel you can make any sort of polynomial algorithm into a linear algorithm


like this like I have a like a cool sorting algorithm right so if this is my sequence


that I want to sort what I can do is I can simply you know take a random subset of them


like this this and this and then kind of go and sort them and then put them like I sent


them to the to the global memory like this I sort them and then I put them back right


and if I do this for enough if I do this for enough rounds okay you know if I do this for


enough rounds you know at the worst case I need n rounds to sort my or log n rounds


if I do it smartly but you know in you know the single step here is the single step is


just O of n so I have now an O of n sorting algorithm I you know I have my sort of a bit


of wary to express things like that and yeah but you know it is from an empirical standpoint


I absolutely believe that this this is enough now my second coral right here is that if


you look at the proof first of all what it makes use is this star graph and the star


graph corresponds to the global attention so that's not much to do with the random attention


though they use the random attention in their proof but I at least believe that it would


be possible with the global attention only and then the second thing is if you look at


the parameters that they use for the for the experiments and I've already set this in the


long former video so in the long form of video it turned out that if you look at how big


this window attention is it turns out that it you're still well you know the original


BERT attended to 512 tokens and then you look at the window and the window was still 512


tokens it's just that the global attention was even more so ultimately they ended up


using more memory than the original BERT and here if I look at the parameters of their


thing and they have multiple experiments right here and I believe this is the base version


so that this is the base version they also have this large version but here this is the


12 layer version and you can see they have this block length and we'll get into the block


length in one second but then you can see that their window size is three times the


block length the number of random tokens is three times the block length and the number


of global tokens is two times the block length so that results in eight times B so eight


times 64 is you know can I calculate this or am I stupid it's 512 yes I actually calculated


this before so this is 512 tokens so you know you you go from from BERT that has 512 tokens


and attends to 512 tokens to also attending to 512 tokens of course the advantage here


is that they now have 4096 sequence length so they have the freedom to not attend to


as many tokens as they have in the input length but you know to put it in perspective this


here uses more memory and more compute on it on its face than BERT because BERT attends


to as many tokens but has a smaller input sequence and you know I there's sort of a


thing where in order to make these sparse attention things work you have to go pretty


pretty you know high in the number of things you attend to you can leave away some but


it's not like you can scale up orders of magnitude of your input sequence length so that's the


this promise of linear attention is sort of it's kind of fulfilled but not there yet the


second thing I would like to point out is that in a lot of cases the number of random


tokens is actually set to zero so really making use I believe of these of the of the global


of the number of global tokens so it that seems a bit strange in that they continuously


refer to their random attention mechanism but then in a lot of experiments they don't


actually have a random attention mechanism I believe they have to do that because that's


kind of what makes them different from the long former in principle but still yeah so


the last novelty let's say is an engineering novelty in that they now always consider not


single for example they don't consider single random attention they always consider these


in blocks and that's because our current hardware is really bad at sparse stuff really bad at


single indexing gathering single things so if you can do everything in blocks you basically


get you get these blocks almost for free so it takes only marginally longer to retrieve


this full two by two block right here than it would to retrieve the single instance right


here of course that means you have you know four times you still use four times more memory


but it is not four times slower than the original thing so you can use these blocks right here


you can do it for the random attention you can do it for the window attention as you


can see here so you break this window pattern a little bit into blocks and that makes it


a lot faster so that speeds up and get the speed up almost for free and then they make


another approximation in that the way they do this windowing is and let's just go really


briefly so you can see right here that it would be very cumbersome to gather so what


we need we're just going to focus at this this dotted thing right here is a bit confusing


so you want to attend to these things and these you can just get out with a matrix slice


really easy but then you want to attend to this kind of blocky thing right here from


the window attention right like this thing and this is hard to get out because you'd


have to kind of index each row individually and that's very slow so what they do there


is this matrix role operation where you can sort of roll the axis around so what you'll


do is you'll take this thing right here and you put it to the left right here and you'll


take for example this thing right here and you'll put it to the right or no like it's


up and down but in essence that's what you do and you can you can fold all of this blue


stuff into a rectangular matrix if you know if you can see right here so you kind of roll


this back roll this back roll this forward and you replace whatever is missing by these


now this again gives you some inaccuracies because this block right here was never intended


to be attended to and all of a sudden you see you have the k6 in here so it gives you


a bit of inaccuracies at the edges of the sequence but you can take that you know you


can take that hit for the increased performance that you gain by now having a rectangular


matrix tpus are really efficient at this not as efficient at this and then the only thing


that's really slow is gathering these random blocks right here but also by having the same


amount of random blocks per input token what you'll do is you'll end up with just one of


these columns right here or you know are of these columns and that again gives you a rectangular


matrix so this thing right here you can process very very efficiently using a tpu and you


know the mistakes you make are basically this thing right here and this thing right here


because those weren't intended and are at the edges of the sequence so these were the


tricks of big bird to quickly summarize big bird is basically taking a transformer saying


well why do we need all of this attention all of this full attention maybe we only need


some of that and can already do a big job a good job especially now considering the


attention mechanism goes over multiple layers so we don't need a routing from each token


to each token we we can make up for not having a fully connected graph by simply running


multiple layers so their sparsity is first of all you have this random attention which


i believe changes from sequence to sequence but stays within or among the layers of the


same sequence then you have the window attention with the reasoning so the reasoning behind


the random attention is that if you have a randomly connected graph the path lengths


are on average logarithmic so you can route information efficiently the reasoning behind


the window attention is that probably neighbor information is very important and that has


been shown empirically and then the global attention the reasoning behind this is that


some of the tokens that are fixed by the developers are so important that it it's very beneficial


that each other node is connected to them and that they are connected to each other


node the result of that is the big bird attention mechanism which is basically long former which


already had these two plus the random attention this achieves a linear linear complexity in


terms of of memory and compute though linear has to be qualified a bit because it's modified


by the window size by the number of random attention tokens by the number of global tokens


and in practice often ends up being you know fairly large ish and also the the the theoretical


guarantees now come with the fact that you need multiple layers in the worst case you


need sequence length amount of layers which you know in the worst case would result right


back into a quadratic requirement for memory and compute they do some engineering some


engineering tricks right here and their results are pretty good so the results in various


tasks and we'll we'll look at some of the tasks right here so these are def set results


using base size models for example where you can see they do outperform basic roberta models


they outperform long former which may mean that the random attention is useful but you


know in these things it's all also always may just mean that you've thrown more compute


at it at least I'm not really looking that they outperform the models because as you


can see right here if they compare to state-of-the-art and you know granted these are models that


have been trained specifically for these tasks and are you know crafted and engineered and


big bird manages to big bird manages to hold itself against them in a lot of tasks and


even get state-of-the-art on some what I'm more interested in is that it you know it


can reach good numbers it doesn't necessarily have to be state-of-the-art but it can reach


good numbers which tells me that okay probably the the empirical hit that I take by not having


the full attention is you know it's justifiable by the speed up and memory savings I do get


yeah especially when result when you see results mixed like this you know sometimes the other


model is good and sometimes the big bird is good on different variations and so on I would


not you know I would not make a big deal out of the fact that it is state-of-the-art I


get that the authors have to do that I would do so as well but you know you don't don't


think that this is the like the best thing now it's very probable they just thrown also


a lot of compute at it what is cool is they do some genomics experiments so not only do


they have NLP state-of-the-art but also they go into genomics and experiment with data


there don't want to go into that because you know ultimately it's another task and I believe


the paper is about the architecture alright so that was big bird I hope you enjoyed this


video and learned I learned something certainly if you want to check out the proofs they're


R Squared Theory - Practical Machine Learning Tutorial with Python p.10

you


Hello everybody and welcome to part 10 of our machine learning tutorial series in this part what we're going to be talking about is


We've been talking about linear regression, and we've got to the point where we could calculate the best fit line in our python code


But now the question is how good of a fit is our best fit line?


How do we determine the accuracy right so the way that we determine?


accuracy is through R squared or the coefficient of


Determination and the coefficient of determination is calculated using what's known as squared error


So first we need to figure out what the heck squared error is and then we can calculate R squared or the coefficient of determination


So to exemplify this consider. You know you've got two graphs


and


Then you've got some plots on those graphs, and then what you want to do is


Draw the best fit line okay, so there's something like like this, and then I have no idea something like this


And if I asked you which one is a better fit


You would say the the one on the right


And then if I asked why is that the best fit you might think for a moment?


But you would probably come up with well the the dots are


Closer to the line on the one on the right as they are closer than they are the one on the left anyways


Now of course we don't have any ticks on our axes here


And so I might say that the one on the left were actually zoomed in really far


They're actually much closer, so you don't really know but but really it's it's how good of the fit is it?


How good of the fit is the best fit line?


How good of your fit is your model to your data set so it's very relative to your data set?


And you'll see more why in just a moment, so we know it's the distance


So so so what how do we actually calculate this well we use squared error


So we've got a graph here, and then we got some data points some beautiful data points


And we have our best fit line and the way that we calculate


Squared error is we say the error is the distance between the point and the law in the best fit line


And then what we say is it's not just error. We want to square that value, so we want to do squared error is you know?


e squared okay, so you might ask why are we squaring it right well in


This case the distance right in one case the distance might be positive in another case the distance might be negative


So one reason why we square it is so that we're only dealing with positive values


You might then ask why is it e squared and not like absolute value of e?


Well, we want to square it because what if you had a point that was like


way out here


that would be an outlier and


Your linear data set should not have an outlier


Because we only want to do linear regression on


linear data right that only makes sense so we square the error because


we want to


We want to penalize for outliers, so then you might ask well why not


using to the power of four or six or


Eighteen for that matter you could use these other ones if you want to penalize for


Outliers you can use a bigger day a bigger value there if you want to penalize even heavier


For outliers, it's just so happens that the standard is going to be squared error, and if you're not using squared error


And maybe you're you're publishing something publicly either in a paper, or maybe you've got some data


Some data some sort of module in Python or something you'd want to alert people to the fact that you are not


Doing it the way that most people do it


Okay, so that is squared error now. How do we calculate the coefficient of determination?


Or R squared so R squared is calculated in the following so R squared equals


and it is


minus and


It's 1 minus the squared error and generally you're going to see squared error denoted as se


So it's the squared error of the y hat line what the heck is the y hat line remember y hat


best-fit regression line all the same thing


divided by the


squared error of


The mean of the Y and that's the mean of the Y's of your data set


So so what might that actually look like?


The mean of your Y's might be that


So it's just a simple straight line and what we're trying to do is


compare the accuracy of that line to the accuracy of like the best-fit line and obviously the best-fit line is almost certainly going to


be better than the mean of the Y's


Better than the mean of the Y's, but we want it to be like way better


Okay, so looking at R squared and the calculation of R squared


What what's like a what's a good value right like what do we think might be a good value versus?


What do we think might be a bad value?


So let's consider a value like let's say R squared equals


How would we arrive at 0.8 well we would know that in order for R squared to be 0.8 it would have to be


The squared error of the y hat line divided by the squared error of the mean of the Y's would have to be


Like this equation here would have to be equal to 0.2


Right that's that's the only way we could get to to 1 minus. What is 0.8?


So how what would be an example of?


This this equation here being 0.2. Well we would find that we would need the squared error. Let's say of Y


Maybe maybe the squared error of the y hat line is


2 and the squared error of the mean of the Y's is 10 okay equals


So if that was the case


We were saying you know the squared error of the y hat line is actually


Significantly lower than the squared error of the mean of the Y's is that a good thing or a bad thing


Well, that's that's a pretty good thing


We would prefer it to be even lower than that, but you know that's pretty good, so that means this data is


Probably pretty linear right so it's it's so an R squared of 0.8 is is pretty good


What if your R squared was like 0.3 for example, so if R squared for example was


How would we arrive maybe at that equation well we would need the?


squared error


Divided by or the squared error of the y hat divided by the square of the mean of the Y's


we would need that to be 0.7 right and


We could get that by you know 7 over 10


And now the squared error of the y hat is a lot closer to


The squared error of the mean of the Y's so obviously this is this is more negative


So we want the R squared value to be high


How high is kind of determined by you?


but the accuracy in this case of our


Model is let's say we're gonna call it 0.8. That is the R squared value


So it's not percent accuracy is it is the R squared. It's the coefficient of determination


That is the value so now that we know what the calculation for R squared is and we know what squared error is


And we know how to calculate the y hat line. We've already done that


We know how to calculate the mean of the Y's we haven't done that necessarily


Actually we have done that because that was part of our our best-fit line calculation


So we've done that we know how to do everything here, so we can definitely calculate this in Python


So that is what we are going to be doing in the next video if you have questions comments concerns up to this point


Please feel free to leave them below otherwise as always thanks for watching


ML News Live! (Dec 30, 2021) Anonymous user RIPS Tensorflw | AI prosecutors rising | Penny Challenge

you


you


All right cool hello I just measured there's about a five second lag or so


but I can hear chat and everything is all right so yeah welcome to the first


the first edition of ML news livestream this is out of necessity my editor is


on Christmas vacation and I don't have time to edit myself so we're just gonna


have to do this live it might be a bit slower than you're used to and a bit


more a bit less concise but I hope you forgive me yeah hello everyone and and


thanks everyone who is here everyone who's gonna watch this in the future I


will put all the links to stuff that I present into the description once the


broadcast is over and I hope I can leave it up in public not that you know if I


say something stupid I might have to take it down because that's usually the


stuff that gets edited out all right so first of all first of all most


importantly please please check out weights and biases look at this awesome


tea beaker I use it for tea but you know you might use it first I don't actually


know if they if they sell it or if they just give it to people who shill for


them in any case check them out I've been if you followed me on the stream


you know that I've been working on a little bit of an implementation of a


question-answering system and this is my panel for this and it's just so easy


with weights and biases it it takes one line not even like as soon as you're


logged in it automatically logs here and this is my sweep that I'm currently


doing it's just optimizing hyper parameters for me I don't have to do


anything except sit and wait and it's optimizing all my hyper parameters and


you can see right here I can sort of filter by the runs that did best this is


Bayesian optimization I have no clue how Bayesian optimization of hyper


parameters works like in depth but it just does it for you you can filter it


tells you which parameters are important here I can see the learning rates are


probably going to lead to good performance and it kind of just logs


everything and then gives you an idea not yeah here's my training loss but


also it logs all the system like GPU usage memory usage blah blah blah


whatever you want and after that you know you can look at this in sort of


table form you can filter it and you can look at all the logs here I've checked


it before there's a lot of errors I might want to casually check in on that


and then lastly you know I can I just stop it from here or pause it or or


whatever and this runs distributed like this runs in a bunch of servers they


just pull the new experiments they run it and they report it back super awesome


yeah please check out weights and biases they've been an awesome sponsor and I'm


pretty sure it's good for you it's completely free for personal use also


some of you might have noticed merch but it's not done yet as you can see the


logo is kind of way too far out so it sort of hides under the arm but this is


more it's mostly it's mostly for myself just because I think it's funny to have


merch but I will put it up if anyone wants one too but this has to move a


little bit closer to the middle I think and there's a few other fixes I need to


do all right first story first this is too funny someone opened a a an issue on


the tensorflow github repo saying I effing hate tensorflow this is in this


is insane like just the the audacity of this person to open a I mean people


sound off on reddit and and Twitter and whatnot and we're used to that but


people sort of abusing the the github issue system in order to sound off on


libraries it's you might have guessed it's been met with various different


reactions I think most people say this don't do this please although this being


the sort of first of its kind at least in our community it is a bit funny but


you know the joke has been done now so please don't do it again it's not like


the first one's funny and the rest isn't but if you want to see this is what it


said it said I hate tensorflow yes I effing hate tensorflow I created this


github account only to this issue please use Jax or PyTorch or anything


else don't use tensorflow don't report me don't mad don't mad at me don't ask


me why because I don't want to talk about it anymore


I effing hate tensor effing flow and as you as you might imagine people have


reacted and reacted and then this got posted on reddit and Twitter and then


people reacted some more and if if you want to if you want to give this a go


it's I think it's funny don't do it but this was funny and then obviously it's


been locked as spam as it should be there is a bit of a bigger discussion on


reddit so exactly of why people hate tensorflow I have not used tensorflow in


quite a while I have to say at least I haven't used any like I haven't


programmed anything myself I have used code I've modified code but if when I


program myself it's usually in torch so people say that tensorflow has sort of


become a big buggy pile of garbage or equivalent I think software libraries


you can always you can always argue about it and negative opinions you know


as as reddit is negative opinions will always sort of bubble to the top and be


more prominent than people who say yeah it's kind of okay so I don't know what


you think about tensorflow it's it does its job and it's powering a lot of the


world's machine learning and I think it's pretty easy it's pretty good to get


started it has Keras as a high-level abstraction and yeah I they put a lot of


work into it and yes there's some bugs but I just thought I just thought it was


it was quite funny if you want to see all the opinions as I said github is github


is fine enough to record the history of everything that has been said right here


so feel free okay thank you wait wait how do I make this go away the South


China Morning Post writes that Chinese scientists develop an AI prosecutor that


compress its own charges the machine is able to identify eight common crimes


such as fraud gambling dangerous driving and picking quarrels and prosecutors in


China already use an AI tool to evaluate evidence and assess how dangerous the


suspect is to the public so this is a story about sort of advances in


statistical systems in China in the criminal justice system and it is a bit


more it is a bit more nuanced than the title seems to suggest so what this has


also been done in the US there was big stories about systems that assess how


likely criminals are to be repeat offenders and now this system right here


is a system that apparently is able to identify charges so if you look for


quarrels these are the these are the things that it recognizes credit card


fraud running a gambling operation dangerous driving international injury


sorry intentional injury obstructing official duties theft fraud and picking


quarrels and provoking trouble so what it's doing is you input a description of


the case like a verbal description and it will output the charges of the


defendant the charges that it would charge the defendant with it is not as


far as I understand it is not the judge it is not you know simply assigning


those charges and then that's it it is seen it seems to automate sort of the


the most boring tasks that a prosecutor does namely you know there is a


description there's a police description and everything and then the charges to


be filed have to be formalized and I can imagine look I'm not trying to I'm not


trying to downplay the severity of this but I can imagine that for something


like someone you know stealing something from a store this is a very standard


procedure right there is a police report it always looks the same kind of lists


the things they stole how they behaved yada yada yada and the sort of the


mental the mental effort to take that and formally file the charges you know


putting all the crosses in the right place selecting the right things from


the list and so on is maybe you know it's it's straightforward I can totally


see that a statistic like an NLP system will simply be able to do this out of


the box now this is not to say that there are many dangers along with this


and many worries but you know that for example it says that picking quarrels


and provoking trouble is a catch-all charge often used to stifle dissent but


prosecutors do that anyway right the system simply replicates what the


prosecutors have done in the 17,000 cases in the last five years that it's


been trained on so you get all the problems of you know this has been


trained on old cases there are sort of old biases in there and whatnot also the


machines probably will not be able to distinguish slight nuances but you would


hope that in a functioning criminal system this goes before a judge the


judge would take a look at the charges and what has been what what is being


said so I mean you know apart from all of all of what you can think right here


I do think that low-level low-level sort of grudging tasks that people have to


do taking natural language putting them into some sort of formal systems can be


automated by AI and the fact that this year is being done in the criminal


justice system is is worrying but it's it's sort of more by association than by


what this particular thing seems to be not saying that there isn't horrible


stuff going on but you know look into it yourself if you want to find out more


I'll link to it there is a was a popular very popular tweet going around by a max


wolf saying I forced the bot to look at every Pokemon and told it to generate


its own here are the results and the pictures are quite neat it's sort of


pictures of of of new Pokemon and they actually look like something so given


how many Pokemon there are isn't this like the sixth or seventh or eighth


generation of Pokemon we're on now I'm surprised that in the space like in the


latent space of Pokemon there is still a Pokemon that doesn't exist it seems like


every combination of pixels that you know looks like a shape of any kind has


been made into a Pokemon so like like how how how does this thing here not


exist this seems like it exists right I don't know but I guess that's the beauty


of that's the beauty of Gantt's yeah so if you if you're interested check it out


it's pretty cool project next story Gibson Dunn writes New York


City and acts a law restricting the use of artificial intelligence in employment


decisions this is a report and apparently from 2023 on New York City


employers will be restricted from using artificial intelligence machine learning


products in hiring and promotion decisions and specifically here it says


so it defines automated employment and decision tools it defines as any


computational process derived from machine learning statistical modeling


data analytics or artificial intelligence that issues simplified


output including a score classification or recommendation that is used to


substantially assist or replace discretionary decision-making for


making employment decisions that impact natural persons there's a fancy way of


saying if you have some system where you throw in a bunch of CVS and you get out


like a score of how good an employee will be you cannot use these systems


anymore so the law prohibits use of such tools to screen a candidate or employee


for an employment decisions until unless it has been subject of a bias audit okay


a bias audit is defined as an impartial evaluation by an independent author


yada yada yada so this is first of all first of all okay first of all the


definition here could cover nothing and everything any computational process


derived from machine learning statistical modeling data analytics or


artificial intelligence that issues a simplified output including a score


calculation recommendation yada yada yada so what if an what if an HR person


googles a person right like the Google is certainly based on statistical


modeling the outputs are a simplified output it's not you don't get the entire


Internet you only get a simplified output pertaining to the person you


search you get a score in the classification you know the search


results no one looks past page one on Google and yes do HR people do use that


to substantially assist them in decision-making so search engines are


now banned unless they've undergone a bias audit but this applies to search


engines what if I throw my throw the CVS into a machine that simply checks for I


don't know like like some some really exclusion criteria I have for example if


a person is even permitted to work in the country that I'm willing to hire


well is that an automated yada yada yada yada it is so these definitions are made


so they can cover everything potentially and then of course there is


the backdoor you have the bias audits and the bias audits they sound nice but


we all know that they're going to be you know going to be I don't know as flawed


or more flawed than the original systems and all this is in my opinion is a shift


of money and resources you can bet that the bias audit companies are lobbying


hard for this law to be passed you know I'm not like this is not saying anything


against you know these these systems do have bias these systems to have giant


problems okay this is not commenting on that this is commenting on this law it's


it's made such that it covers it can cover everything and it is made to shift


resources to a thing that in my opinion doesn't solve any problem at all with


these automated hiring tools of course they are tricky right but we have to


face the reality a lot of places now they get so many applications it's it's


impossible to handle them humans make mistakes too so now if you don't want to


undergo this bias audit you have to get more humans you have to overwork them


more you have to have them more like hey human check all of these CVS for who's


even eligible right this is a grudging task and now the humans are there being


like all right you don't you think they don't make mistakes you think the


humans are not biased well okay but that's that's my opinion you can decide


for yourself at least the violations are moderate let's say at least for so I


guess medium-sized businesses can afford violate to violate this next story this


this is actually there's a bit more drastic there is a tweet by Kristen


Livdahl saying my 10 year old just asked Alexa on our echo for a challenge and


this is what the what she said I presume that is referring to Alexa so the prompt


is tell me a challenge to do and the response is here is something I found on


the web according to this website the challenge is simple plug in a phone


charger about halfway into a wall outlet and then touch a penny to the


disposed to the exposed prongs and there's even a little timer to go along


with this so this naturally shocked a lot of people well shocked I mean people


deliberately saying well I'm not shocked I always knew but you know it is it is a


rather a rather grave suggestion right if someone were to actually follow this


there is serious danger involved right here especially if you hold on to the


penny as you do it but even if you don't you know things go bang and start


burning so the bigger question is more of you know what should we do about it


of course Amazon responded saying they'd fix it immediately but you know


obviously there's going to be other cases the question is a little bit is


there something you can proactively do against these types of mistakes while


still retaining the service right of course we could say we don't do


question-answering anymore the problem here is not that this challenges exist


because obviously Alexa found this on the internet somewhere this is a


challenge people actually do it's you know akin to eat this eat this cleaner


eat this tide pod it's a little bit like this it's just something stupid that


people challenge each other to do and it probably originated from someone just


trolling however if Alexa find something like this a lot on the internet you know


should it suggest that or what can we do here should we try to detect this should


we display a warning do we have to take down question answering all together so


that people don't get the impression that this is some authoritative


information it is a hard problem right you cannot possibly cover all the


dangerous cases because well language is rich and the internet is good at coming


up with new stupid things so it's impossible to cover all of them however


you know what else should you do should you sort of put on 20 disclaimers well


if you put them everywhere people are just gonna disregard them it's it's the


same as you know you you put with every file you open you put like this could be


a virus after a while people stop paying attention so I'm not entirely sure what


to be done here and I don't know if the people who are very outraged about this


have better ideas but it is a it's a cool challenge if you are looking for


something to work on with like some real-life benefits if you can get to a


general place where things like these are prevented then very very you know


excellent work there is this article by market watch this fund is driven by


artificial intelligence this sorry this fund driven by artificial intelligence


is ditching big tech here's what it's doing instead so this is about a fund


like a hedge fund that is run by artificial intelligence so the story is


about how it used to have a lot of the sort of of the five core FA and G stocks


in it and made some profit but now recently recently it has it has been


dropping those it only has Apple now so it says as recently as September these


were its top four positions making up nearly a quarter of the exchange-traded


fund now only Apple is in the portfolio so this might lead you to believe wait a


minute there is you know something here I you know what what is going on this AI


knows something I don't and ah it's strange and then you read it uses IBM


Watson to pick stocks me who trusted its money to IBM Watson a product that one


jeopardy yes good job you won jeopardy and then failed to solve any other


problem ever allegedly allegedly so you know good job trusting your money to


Watson I mean if it you know if it works out good for you happy you get rich off


of it but this is this is not financial advice this is for entertainment


purposes only okay this year the food recognition benchmark 2022 it's a real


life real life not hot dog okay so recognize food from images extremely


useful especially distinguishing hot dog from not hot dog you you can take part


in it if if you have a class not hot dog not hot and so easy so easy okay not hot


dog now you you look at this is there even hot dog nah nah 100% so far but you


know can you build a machine that does the same like me as a human who knows


take part in the food recognition benchmark there's prizes apparently


there's a drone there's another drone I'm not curious I I already solved the


challenge and there's an oculus yeah you go for it yeah this is the helpful


helpful things section of the broadcast so we're gonna go a bit faster now and


I'll just point some stuff out that might be helpful to you and as I said


all the links are will be in the description so great yo now has native


embeddings so without iFrames there's like these these like two lines of of


code it's actually it's not two lines of JavaScript unless you count the comment


I guess it's one tag and one line of JavaScript so you can see this right


here this is an embedded radio space so this what is it mushroom nah okay let's


try let's try it I only have my mouse right here so this wait no clear ah so


this and then this and then we're gonna I'm did I ever say that I am terrible at


drawing


syringe no I was going for hot dog okay maybe I like the bun and then the thing


in the middle and then the other one hot dog see we could put this on the on the


food challenge okay you can now embed gradient spaces very cool like natively


with with JavaScript active loop is has developed this hub which is supposed to


be a data set format for AI this reminds me of this X KCD comic where they go


there's 16 different standards for data transmission let's develop one that


unifies them all and then it's like six months later there's 17 different


standards for data transmission it is a common interface for data sets what I


think what I find cool is that it has lots of back-end possibilities so you


can choose sort of where your data set is stored can be in an s3 bucket and in


a Google Cloud bucket in memory so on and therefore you can send these around


and you know give data sets easily to other people no matter where they are


located it might be it might be something for you yeah check it out if


you are looking for a solution like this echo is a library by the way these


things they are sometimes not super new but they have new releases and or I've


just recently seen them I call this news but it's it's more like news to me echo


is a library that visualizes NLP models and they also have a new version out now


and this is a pretty cool library so for example here you can see you can ask


which of the tokens have led to a particular decision like here the


decision negative is influenced very much by you know the words like weakness


and expect and below expected so you can you can decide for you can make better


informed decisions about what your model does they have lots of visualizations


like what you know what words are considered in particular instances at


which layer of a transformer did the model you know think or did the model


get most of the information so you can see here the word London which is the


correct token at layer 0 it was ranked 1444 as a token so it wasn't really on


top of the list of an answer but then as it goes through the model it sort of


gets more sure layer 9 seems to be particularly effective right here in


layer 10 it sort of broadens its scope again a little bit but layer 11 actually


gets gets it right problems its scope I guess it puts it at rank 53 maybe a


short hesitation before the finish line it's interesting is this it I wonder is


this a general pattern that sort of the last layers sort of fan out again I'm


not sure because it seems super monotonic until like layer 9 right here


I'd be interested there is some evidence from research that these transformer


models especially BERT or so that have been trained with masked language


modeling they have their more abstract or their most abstract representations


in the middle somewhere because towards the end they need to do do language


modeling again and actually go back to tokens sorry just


double s refreshing yeah so what are the patterns in neural activations in BERT


when it process a piece of text and so on there's lots of examples of


visualizations you can do so if you look to understand your your language models


check out echo then this is continuum also with a new release I believe no


this is September 17 it's just news to me this is a data set library like a


data loader library that you can plug into torch data loaders is if I


understand correctly and what it allows you to do is to do continual learning so


lifelong learning continual learning and so on and what is this on the right


usually data sets there they're sort of bounded even if you can't load them in


memory all they are usually defined by like a set of files or some table in a


database or whatnot and with this library this extends it a bit more and


enables things like continuous learning lifelong learning iterated learning and


so on there is a this reddit post and it's about a repository that makes the


rounds every few weeks on reddit and people are always excited by it and it's


okay I messed it up it must be coming somewhere it's the transformers deployer


repository that makes transformers inference faster and what it does is it


says it somewhere it says it somewhere what it does is it goes it parses your


Python code it builds the AST the abstract syntax tree from it and then


it's able to go down the abstract syntax tree and whenever you have something


like a weight layer where is it where is it behind the scene transformer source


is parsed to AST operators like matrix multiplication or a layer norm are


wrapped by a quantizer linear layers are replaced by quantized versions of them


and some unsupporters unsupported operators are replaced so this is very


automatic you plug it in and it does as much as it can for you this is not a


substitution for you writing good code but it can quite can speed up your


inference a little bit so you can see here from the example code you have to


wrap your code in a context of this QA T calibrate object and then you can do


various operations on the model such as yeah this setup model QA T will will


auto auto enhance your model let's say okay this was this Facebook has a model


called slip which is a derivative it's a mix between clip which is we train


images and language together and self supervised learning if I understand


correctly yes and self supervised learning is when you take images and you


do any of the any of the five million self supervised learning tasks usually


it's either some kind of contrastive objective where you try to bring


representations close together and far apart or it doesn't even have negatives


in any case this model beats clip in various benchmarks various zero shot


regimes so on and shows that you can in fact combine the the image language


supervision like clip and self supervised learning which I guess they


do like they do in the sim CLR so this is specifically for for image data that


has text associated with it all right there is a paper and their models are


available code is available cool this I found this Joel Jang has made another


music video to my be my weasel song I just found that to be fantastic so it


moves it sort of the the images they stay they stay quite like there is a


continuity to them which tells me that maybe Joel optimized the latent


representation always kind of started from the last one when starting to


optimize I initially always started from a random one then optimized and then


calculated the trajectory in between them I don't know if if that is


different here but it looks very cool I didn't I found this by like accident so


thanks Joel very cool and there is a github repo with the code where you can


I suppose create your own this is obviously upgrading from my original


design with that had clip and regular GAN big again this is upgraded to now


VQGAN which is obviously much much more pretty if you want to check it out this


is available and my original report remains available and I believe the song


itself I linked somewhere as an mp3 file so yeah there is this library RL Hive


which is another library to do reinforcement learning the


reinforcement learning library space is like so there is more RL libraries than


RL researchers or problems to solve however it also tells you a little bit


that there appear to be a lot of stuff not solved yet like in library design


it's not like the whole world has converged on a reinforcement learning


library and this might stay this way because the field is just so large or it


might be because no one has sort of cracked the code yet and therefore if


you are unsatisfied with all the RL libraries so far then give RL Hive a


try as a neat documentation with some quick start experiments and it is easy


to run single and multi-agent reinforcement learning tasks compute is


a library a general-purpose GPU compute library and the special thing is that


it's cross-vendor so you can see it supports various graphics cards so this


is it is a bit lower level than you know the frameworks like tensorflow and and


whatnot even if they they sometimes have as far as I understand they sometimes


have a little bit of a support for different other than Nvidia GPUs for


example I mean TensorFlow has has a TPU support but here we talk about graphic


cards so this is for anyone who wants to write a little bit of a lower level


code like GPU code that maybe does a little bit more complicated things than


just plugging neural network layers together but doesn't want to go super


far down close to the metal and having to contend with all the different


graphic card vendors which inevitably you end up writing it for the one you


have and then trying to port it to something else so I have not much clue


about this stuff if you're interested in lower level but yet still abstract GPU


code then maybe give compute a try if you have something need to implement and


you think you could benefit from acceleration but the frameworks like


TensorFlow Pytorch are just too a bit a bit too restrictive this might be for


you okay the CPPE data set the it's a it's a data set that is essentially an


image recognition localization segmentation or or multi sorry multi


localization data set and is specifically for around personal


protective equipment so the difference here between this and other data sets is


that this is specifically for the niche of the medical field and they've paid


special attention as far as I can understand that the pictures are real


life pictures well on the top left I'm not entirely sure how real life that is


that this person is posing but these are not kind of staged let's say these are


at least that's what they say in the paper they they really paid attention


that their complex scenes their kind of real life scenes and therefore the task


is quite hard so if you're interested give this a try this was it yeah I knew


I had it open somewhere at the transformers deploy it's twice okay


the there's a call out to the first international symposium on the settling


machine I hope that's one how one pronounces it settling machine this is a


symposium about well yes did settling machines what is it settling machine I


like if you don't know what it is it's I this is not going to be enough time to


explain it but essentially it is a neural like you can imagine it as a


neural network ish thing but that doesn't necessarily go with numbers and


vectors but it's fundamental units are propositional logic so it will try to


make sense of the world by applying propositional logic rules and sort of


coming up with almost a bit like program building internal program building


making sense of the world by building up symbolic rules and then trying to


compute some output out of it that's not to say that there are no like numbers in


it not to say that there isn't there isn't vectors in any of this however it


it is built on fundamentally different principles if this is interesting to you


check out Wikipedia article if this is still interesting to you maybe check out


the first symposium Jojo Gan is a one-shot face stylization system and all


like the entire reason I wanted to show this there is a repo to go along with it


so you can try it out there is a hugging face space there is you know a nice gift


that explains it to you with famous people however the reason I'm showing


this is to show that if you put Elon Musk on something right now you'll just


get like exposure massive and I'm gonna start to do it doing this from now on


just put Elon Musk on stuff it used to be it used to be like used to be Lena in


the early days and then that just kind of kind of got frowned upon and then it


used to be Obama you put Obama on anything and then it used to be I think


used to be like drum and now it's Elon so you know what there there you go if


you want fame well either post Elon or post this picture with the airplane and


the dots the survivor bias picture and this is the last story the Guardian


writes worried about super intelligent machines they are already here now this


I almost dismissed this article but then I saw who wrote it John Norton and then


I remembered we have already talked about John Norton before namely in this


article right here the truth about artificial intelligence it isn't that


honest also by John Norton in which you know I rant a lot about this truthful QA


benchmark but I have never ever seen anyone butcher it's so hard like John


Norton of the Guardian it's it's just like if you think that the benchmark


itself and the tweets by the authors are no kind of frame to to give a false


impression the article here is just outright lie about what the results of


this research are it's just wrong and so I was interested in this article and it


just it's like it has three paragraphs of who people have been worried about AI


machines and it finally comes to this dystopian fears Nick Bostrom yada yada


yada and then it says but for anyone who thinks that living in a world dominated


by super intelligent machines is a not in my lifetime prospect here is a


salutary thought we already live in such a world the AI's in questions are


called corporations they're definitely super intelligent the collectify cube


yabba-dabba-dabba immense Ralph and resources that you said machines John


machines you just don't you don't you don't get to switch the topic okay like


like this is it's not a lie like the other but it is clickbait as hell like


what you just you don't get to do this machines corporations are not machines


you can make an argument that corporations are organisms their hive


minds they have the life of their own they have power and possess goods can


trade by themselves and have their own dynamics which no single human is sort


of the master of but you don't get the call the machines and you don't get to


make an article about how people are afraid of a eyes rising up and then and


then saying it's all right here they are called corporations I mean calm on and


then this doesn't even and then it's like one such super intelligent machine


is called Facebook oh no oh no ah what's the deal with dumping on Facebook I don't


get like I have asked myself this so much like it's Facebook seems to be the


like whatever that the worst even though I like I don't see a difference between


Facebook and Twitter in terms of how bad the environment is it's both like a cesspool


of stuff but Facebook YouTube also doesn't have with these people YouTube


is also like me but Facebook is is like super super duper bad I don't know why


I've not figured out yet in any case sorry no the super intelligent machines


are not already here and it is also not a funny thought experiment you yeah to


think about it all right this this was it what WTF is a Facebook well yeah we


should almost call it the Facebook right why would white people just call it


Facebook that that's kind of dumb yes this was it for the first edition of ML


News live thank you so much for tuning in and let me know how you how you like


live streams because for sure like we will edit the news again because this is


just too slow and too long but in general I enjoyed live streaming quite a


bit for you know doing just doing research and chatting to people but I


might do it on the second channel by the way there is a second channel it's


called Yannick out of distribution but yeah I might live stream there some more


check out weights and biases please it's definitely a very cool platform the


[Rant] REVIEWER #2: How Peer Review is FAILING in Machine Learning

It's review time, review time.


So NeurIPS has recently released the reviews for submitted papers and pretty much everyone is not


happy. And I think the reason is that even though you have the reasonable reviewers of these


conferences, there is always always reviewer number two. And reviewer number two leaves very


short review says that either there are not enough experiments, or the theory is too weak,


or the assumptions aren't warranted, or they just don't like your face. And that's why they give you


a weak reject. Actually, some of them think your paper is fantastic, and give you a weak reject.


So a lot of people are angry, upset, dissatisfied with the quality of the reviews in machine


learning conferences. And today, I want to go look into bit into how this works, why this is the way


that it is, and what we could potentially do about it. So what's happening with publishing in ML, the


system seems to be overloaded. There's so much attention in machine learning right now that


there hasn't been a few years ago, that there's a huge influx of new people wanting to publish


in this field. That creates a lot of submissions and not enough reviewers to peer review these


submissions. So a lot of reviewers are recruited that probably shouldn't be reviewers. I hear


stories of undergrads being recruited as reviewers, people from way outside the fields, people that


don't have time. So too many submissions, too few inexperienced and not really expert reviewers


creates pretty much a random process. And this was also shown in a few years ago in the then


NIPS experiment, where it showed that for most papers being accepted is pretty much a coin flip


with a weighted coin. The natural response as an author is going to be you're going to submit even


more papers. If it's a coin flip, you can just submit whatever and there's a chance it might get


in, which of course only makes the problem worse. So this entire process of science where you


submit your manuscript and then you get the reviews and then you try to improve it,


it's completely broken. Because not only do you not care about the reviews, the next set of


reviewers at the next conference are going to be different. So no matter what you improve right now,


the next set of people will have completely different criticism. It just doesn't work


like it is intended to work. The review process is basically just some kind of a random nuisance


to people that they have to get through. And at the same time, people who are reviewers have every


incentive to make it as hard as possible for the people that are submitting. So in order to analyze


this, I want to look at the incentives of the different groups in this process and kind of show


how the incentive structure upholds this system that benefits pretty much everyone participating


in it, but creates a worse outcome for all of us. So first of all, let's look at paper authors,


what are your incentive if you're an author of a paper? First of all, authors, they want to get as


many papers as possible as fast as possible. Now in the current conference system, the fastness


isn't really up for debate. It's as fast as it is. However, authors can simply upload their paper to


archive and be as fast as they want there. Another incentive for authors is to have as little comments


on your paper as possible, because comments usually mean criticism. And you don't want comments and


especially you don't want public permanent comments. The good thing for authors right now is


on archive comments aren't possible and conference reviews even if they're made public,


no one goes to look at them, everyone just goes to archive. So authors right now are getting a


pretty good deal with respect to not getting their work criticized. Authors are also incentivized to


give as little credit to people as possible. And again, the current system is totally in favor of


that the no commenting on archive basically means that you can claim whatever you want. And if


someone wants to refute you, they have to make a big deal out of it and basically write their own


paper. And again, people will probably not find that on the other hand in the conferences, reviewers


are supposed to detect when you're not giving proper credit to other people. However, most


reviewers don't do that. Going out and really looking if everything is credited properly is one


of the most time consuming tasks when you review a paper. And most reviewers simply aren't going


through that trouble. The only downside for most authors is even though all of this is pretty much


in their favor, a lot of them still require that stamp of approval that peer review accepted at a


good conference. So their incentive is to keep submitting to conferences as many papers as


possible basically count on that random process to get them accepted. And after that, they're just


fine. They have the stamp of approval, there's absolutely no requirement to revise it. There's


absolutely no requirement to have other people comment further on the work. So I guess the


complaining here right now is just about the noisy process. And everyone complains that


their particular paper, which is at the behest of the noisy process as everyone else's paper,


got an unfair treatment in that random process, which half the papers do probably more. The


incentives in the systems are actually even bigger for what I call the big names. Okay,


these are the big research institutions of companies, or big name professors, anyone that


has some sort of reputation, people argue that anonymous reviewing is actually good for small


authors good for unknown authors because it hides their identity and the big names pretty basically


aren't able to play their big name credit to a paper. However, there's an easy way to know


that this isn't the case. The big names are doing just fine. Here's the issue. If you want your name


to be attached to something, you're gonna find a way to do it. People are suggesting archive blackout


periods and whatnot anonymous submissions to archive, you have to realize that if someone


wants to give some information to the public, they are going to in fact, right now the big names are


finding every possible way to have their names attached to things and massively increase their


chances of getting through the anonymous peer review process. You got to realize if you're


well connected, not only do you have an advertising platform, but you can also pretty easily find out


who your area chairs are, who's reviewing in which track your paper gets and so on. So allow me to


be a little bit skeptical about the claim that we need more anonymity in this process. I think we


need less. Second, what are the incentives of the conferences itself? So the conference organizers,


they want to have a good reputation, which basically means they want to be like a cool


nightclub. Lots of people want to get in, but they have to reject a lot of those people in order to


make the club exclusive and have a high reputation. So conferences have every reason to invite


everyone to submit as much as possible, but then to reject as much as possible to make it seem


like it's super hard to get in. This only makes the problem worse. And I think the current explosion


isn't really desired by the conferences as the process is super noisy, they're slowly losing


their reputation that way. But still, the incentives aren't to lower the amount of submissions and


increase the overall quality because that means a higher percentage of submissions will have to get


accepted, which means that the conference appears to be less exclusive. And lastly, let's look at


the reviewers themselves. This is the most screwed up part in the system. I have every incentive to


be a reviewer for one of these conferences because I can write that on my CV, hey, I was a reviewer


at big name conference. And then once I am accepted as a reviewer, I have every incentive to do


absolutely nothing. In fact, the less time I waste with this, the better because I'm not getting any


public credit. I'm anonymous, right? anonymous peer review. I'm not getting any reputation out of this.


And in fact, I can only lose from accepting papers. And I can only lose from writing detailed


reviews. If I'm short and vague, and I reject a paper, not only can I not really be criticized,


because I'm not saying much, it's actually in my overwhelming interest, if the paper has some sort


of big mistake, and I overlook it, and I accept the paper and the other reviewers see that mistake,


this looks really, really bad for me, even though I'm anonymous in the broader context, it also


looks bad for the area chair supervising me if they don't see it, it looks bad for the conference


if their area chairs don't see it. So there's a massive push to not make mistakes. However,


if I reject a paper that was actually good, I can just say, well, they can resubmit to the next


conference. So already have a giant prior to reject a paper. Add to that that usually the


papers that are review might be my competition. And by the conference incentive of being pretty


exclusive, the more of my competition gets accepted, the less I might get accepted. Because


not only are there limited amount of space, not formally, but informally, other work might overlap


substantially with my own and therefore make it less likely that I get published. Also other work


might actually criticize my work. And I don't like that. And this is a bit cynical. But I'm not saying


everyone does this. But there is an incentive for you as a reviewer, especially if the work is close


to what you're doing to reject it now, implement the same or a very similar idea, and then submit


to the next conference where these other authors also will submit and hope for the random process


to just for your paper to get more lucky than their paper. Blackplanting on archive counters


this a little bit. But I'm afraid that with proposed solutions like more archive blackout


periods, more anonymity, these problems will only get worse. Maybe some people don't realize this,


but as a reviewer, it's really easy for me to reject a paper, I can almost always find reasons


to reject a paper. If it's a theory paper, I can ask for experiments. If it's an experimental paper,


I can ask for more experiments. Why didn't you test that data set? Why didn't you compare against


this method? Why are your assumptions so strong? They're never guaranteed in practice is the problem


even relevant? Your theory is too weak? Have you looked at this other special case? And if I really


want to, I can just ask many, many, many questions, not even criticisms, just many questions. And I


know the authors just have a one page rebuttal, they can never answer all my questions if I do


that. And then I can simply argue the authors failed to address all my questions properly. So


you might be asking, why do some reviewers actually do a good job? And that is, I believe,


really a lot to do with goodwill. Most people are actually well intended. Most people actually


want to do a good job in reviewing have the ethos of science and do take the time to do the reviews,


even though they're incentivized to do them badly, even though they're incentivized to reject papers,


a lot of people still do a good job. However, reviewer number two usually doesn't. And it only


needs a very few reviewer number twos to make the field a whole lot worse. Now there's a question to


be said, aren't we all a bit reviewer number two? Have you ever written a review that the authors


might think is completely unreasonable? And while there is some truth to that argument, I definitely


know that there are differences in reviews. In fact, I've heard people brag about writing two


line reviews where the second line is you didn't cite and compare to my own work and then laugh


about that. So goodwill won't carry us all the way if the incentive structure is bad. And I believe


most of this is because we've taken out the reputation game out of the review process.


In smaller fields of science, it used to be that the journal editors knew the reviewers and their


reputation at least towards the journal editor was on the line for all of future. If they did a bad


job. Right now everything's so big, so anonymous, people hardly remember the names of their co


reviewers, no reputation is being damaged by bad reviews. And that's how we get here. So of course,


I'm not the first one to observe these problems. So many people have proposed solutions. And most


of these solutions fall into the basis of what I would call a C based methods, which is basically


where someone evaluates the reviews while the reviews remain anonymous, and that someone is


usually the area chair. So right now the area chair can already decide that a reviewer is really


bad. And then the reviewer will not be invited to review the next time around. I just want to point


out the irony of the situation. conferences nowadays have so little reviewers that they


require every author to be a reviewer. But then your punishment for writing bad reviews


is that you won't be invited to be a reviewer the next time around. I mean, can you make a better


point that the system is failing? Of course, the problem with all AC based methods is that you're


basically moving a problem that has everything to do with people being unaccountable, noisy, not


expertly, having no time and every single incentive to do as little as possible. You transfer that


problem to even less people that have even less time that have even more stress that have an even


broader view and topic area and are single people instead of three or four people. So it's even more


noisy. If anything like this is implemented, you'll just instead of seeing complaints about bad


reviews. In addition, you will also see complaints about bad ACs that will certainly not make the


problem any less. In fact, I would argue any AC based solution will make the problem worse. Other