Machine Learning Holiday Live Stream

squad let's just go with squad


Machine Learning Holiday Live Stream

data sets is not defined


Machine Learning Holiday Live Stream

right


Machine Learning Holiday Live Stream

good let's just remind ourselves and mostly me how this works


Machine Learning Holiday Live Stream

okay yeah so this is all


Machine Learning Holiday Live Stream

this is all good


Machine Learning Holiday Live Stream

I used to we used to all do the all of this by hand right like load the data and process pre process stuff and yada yada yada so, and this has been, it's gotten so much easier recently, it is, it is crazy.


Machine Learning Holiday Live Stream

So we'll get the training data set, and this is an index of all right this is not just an iterable yeah.


Machine Learning Holiday Live Stream

So, it's technically what we can do is, how does this map work right here.


Machine Learning Holiday Live Stream

map.


Machine Learning Holiday Live Stream

Right, we can we can do this.


Machine Learning Holiday Live Stream

This map function.


Machine Learning Holiday Live Stream

No. Yep. Do this map function. And what it does is you can give it a like takes a callable accepting a dict as an argument and iterates over the data set by calling the function with each example.


Machine Learning Holiday Live Stream

So that's what we do.


Machine Learning Holiday Live Stream

We'll map it. Or how do we, how do we then train.


Machine Learning Holiday Live Stream

How do we train.


Machine Learning Holiday Live Stream

Trainer dot train so we got the model, we have a data set, and how does the data set need to be structured for training, that's my, my only other question.


Machine Learning Holiday Live Stream

So, like how do we how do we know what the, the loss is.


Machine Learning Holiday Live Stream

IMDB.


Machine Learning Holiday Live Stream

Tokenize function.


Machine Learning Holiday Live Stream

So we tokenize it. But then what's the.


Machine Learning Holiday Live Stream

How do we tell the model.


Machine Learning Holiday Live Stream

What the label is.


Machine Learning Holiday Live Stream

We just give it the data set, but is that just like inherent in this data set.


Machine Learning Holiday Live Stream

Because here you see map.


Machine Learning Holiday Live Stream

So.


Machine Learning Holiday Live Stream

We return, we map the tokenize, tokenize the data sets.


Machine Learning Holiday Live Stream

Small train data set.


Machine Learning Holiday Live Stream

Right, this is what I don't get. So we simply extract the text.


Machine Learning Holiday Live Stream

We tokenize it, which is cool, which is fine. Right.


Machine Learning Holiday Live Stream

But then we only have the text.


Machine Learning Holiday Live Stream

How does it know what the labels are.


Machine Learning Holiday Live Stream

Labels.


Machine Learning Holiday Live Stream

Now this is for evaluation. Report metrics.


Machine Learning Holiday Live Stream

Report metrics.


Machine Learning Holiday Live Stream

This must be inherent to the data set somehow.


Machine Learning Holiday Live Stream

Auto return by tokenizer.


Machine Learning Holiday Live Stream

OK. I see.


Machine Learning Holiday Live Stream

Cool.


Machine Learning Holiday Live Stream

So if, if we transform this right now. So let's make a function.


Machine Learning Holiday Live Stream

Dev process data because why not because.


Machine Learning Holiday Live Stream

I feel like implementing a Turing machine. Every function could be called process data.


Machine Learning Holiday Live Stream

So this is going to be an example right here.


Machine Learning Holiday Live Stream

We.


Machine Learning Holiday Live Stream

We pass and at the end we'll just.


Machine Learning Holiday Live Stream

Pass in the zero if data point right here.


Machine Learning Holiday Live Stream

All right. So.


Machine Learning Holiday Live Stream

If we do this, we'll just get our normal.


Machine Learning Holiday Live Stream

Our normal thing.


Machine Learning Holiday Live Stream

Now what we need to do is we need to get the context.


Machine Learning Holiday Live Stream

The answer and the question.


Machine Learning Holiday Live Stream

And for the context and the answer.


Machine Learning Holiday Live Stream

Let's just concatenate them.


Machine Learning Holiday Live Stream

Yeah.


Machine Learning Holiday Live Stream

We'll do it. We'll do the cheapest. We'll do like the GPT three cheap trick right here. So.


Machine Learning Holiday Live Stream

Process data of data. Yeah. That's what we're going to do. OK. So we'll do the input.


Machine Learning Holiday Live Stream

Text is. Let's put the.


Machine Learning Holiday Live Stream

Context right here.


Machine Learning Holiday Live Stream

And.


Machine Learning Holiday Live Stream

We'll do. We'll do GPT three style. We'll do.


Machine Learning Holiday Live Stream

Context colon.


Machine Learning Holiday Live Stream

Of this.


Machine Learning Holiday Live Stream

We'll do.


Machine Learning Holiday Live Stream

Answer colon.


Machine Learning Holiday Live Stream

I know T5 does things differently a little bit.


Machine Learning Holiday Live Stream

But we don't care.


Machine Learning Holiday Live Stream

Example.


Machine Learning Holiday Live Stream

Answers.


Machine Learning Holiday Live Stream

We have multiple and we could have multiple answers, but we'll just take whatever the first one is.


Machine Learning Holiday Live Stream

So.


Machine Learning Holiday Live Stream

The answer.


Machine Learning Holiday Live Stream

Text.


Machine Learning Holiday Live Stream

Zero. Like that.


Machine Learning Holiday Live Stream

So this is going to be our input.


Machine Learning Holiday Live Stream

And then.


Machine Learning Holiday Live Stream

This one right here is the prompt.


Machine Learning Holiday Live Stream

And output text is going to be simply the question.


Machine Learning Holiday Live Stream

All right.


Machine Learning Holiday Live Stream

Now we need the tokenizer.


Machine Learning Holiday Live Stream

The tokenizer.


Machine Learning Holiday Live Stream

We have it somewhere up. Right. Good.


Machine Learning Holiday Live Stream

Oops. Oops. Oops.


Machine Learning Holiday Live Stream

See what did I do?


Machine Learning Holiday Live Stream

Encode.


Machine Learning Holiday Live Stream

The input text. So these are going to be the input IDs.


Machine Learning Holiday Live Stream

Do we need a mask?


Machine Learning Holiday Live Stream

We can give it two things.


Machine Learning Holiday Live Stream

Can we also give the T5 thing two things?


Machine Learning Holiday Live Stream

Maybe not.


Machine Learning Holiday Live Stream

Tokenizer.


Machine Learning Holiday Live Stream

Probably not. It's going to be fine. Right.


Machine Learning Holiday Live Stream

This right here. Encode.


Machine Learning Holiday Live Stream

Return tensors.


Machine Learning Holiday Live Stream

PT will do that.


Machine Learning Holiday Live Stream

Let's just do this once. OK.


Machine Learning Holiday Live Stream

So the input IDs is going to be.


Machine Learning Holiday Live Stream

Good.


Machine Learning Holiday Live Stream

Could we put a second one here? Like what happens if we do this?


Machine Learning Holiday Live Stream

Then.


Machine Learning Holiday Live Stream

Are these two lists now?


Machine Learning Holiday Live Stream

Let's see.


Machine Learning Holiday Live Stream

This is the tensor bracket.


Machine Learning Holiday Live Stream

Anyone see whether these are two lists?


Machine Learning Holiday Live Stream

Probably not. Right.


Machine Learning Holiday Live Stream

So if we do this, the last one is a 10 1.


Machine Learning Holiday Live Stream

And then if we do this.


Machine Learning Holiday Live Stream

The last one is not. Here is 10 1.


Machine Learning Holiday Live Stream

And then there is 304 something something. OK.


Machine Learning Holiday Live Stream

I mean, we could just do language modeling as such. Right.


Machine Learning Holiday Live Stream

And we could then even let it figure out the answers.


Machine Learning Holiday Live Stream

But that's kind of crappy because we would also train it to do the context.


Machine Learning Holiday Live Stream

T5 is encoder. Yeah. T5 is encoder decoder.


Machine Learning Holiday Live Stream

You must specify the decoder input plus pad plus plus label.


Machine Learning Holiday Live Stream

Yeah. So the query, the queue here.


Machine Learning Holiday Live Stream

Would probably be the part of the decoder.


Machine Learning Holiday Live Stream

Like here, we're going from a T5.


Machine Learning Holiday Live Stream

This is Pandas.


Machine Learning Holiday Live Stream

This is validation down here.


Machine Learning Holiday Live Stream

We have target IDs and source IDs for this one. Right.


Machine Learning Holiday Live Stream

So input IDs.


Machine Learning Holiday Live Stream

But we don't have...


Machine Learning Holiday Live Stream

If input IDs, decoder input IDs is this.


Machine Learning Holiday Live Stream

And LM labels is what language model labels.


Machine Learning Holiday Live Stream

So here we have input IDs.


Machine Learning Holiday Live Stream

The attention mask.


Machine Learning Holiday Live Stream

Encoder outputs.


Machine Learning Holiday Live Stream

See our decoder input IDs. That's, I guess, also what we want for the decoder.


Machine Learning Holiday Live Stream

Provide for sequence to sequence training.


Machine Learning Holiday Live Stream

Uses the pad token ID as the starting token for decoder input IDs generation.


Machine Learning Holiday Live Stream

If...


Machine Learning Holiday Live Stream

Take a look at T5 training. There we go.


Machine Learning Holiday Live Stream

We could use those extra IDs, right?


Machine Learning Holiday Live Stream

Unsupervised denoising training. Supervised training.


Machine Learning Holiday Live Stream

Translate English to German. The house is wonderful.


Machine Learning Holiday Live Stream

Labels. Return tensors.


Machine Learning Holiday Live Stream

Easy enough. Easy enough.


Machine Learning Holiday Live Stream

Thank you for letting me know.


Machine Learning Holiday Live Stream

So we put our end of sentence here.


Machine Learning Holiday Live Stream

Then we put... Let's do this.


Machine Learning Holiday Live Stream

Yep. And our end of sentence here.


Machine Learning Holiday Live Stream

Input IDs, labels.


Machine Learning Holiday Live Stream

Output text.


Machine Learning Holiday Live Stream

PyTorch tensors. Let's remove this one.


Machine Learning Holiday Live Stream

And we should be good to go.


Machine Learning Holiday Live Stream

So input IDs and labels.


Machine Learning Holiday Live Stream

Excellent. Excellent.


Machine Learning Holiday Live Stream

Where was it? Where was T5 training?


Machine Learning Holiday Live Stream

Too many tabs open.


Machine Learning Holiday Live Stream

Supervised training.


Machine Learning Holiday Live Stream

OK, if we put this into the model now, do we get a loss? That's the question.


Machine Learning Holiday Live Stream

If we get a loss, everything's fine. Everything's good.


Machine Learning Holiday Live Stream

So where is our model?


Machine Learning Holiday Live Stream

Model.


Machine Learning Holiday Live Stream

Let's call this X because why not?


Machine Learning Holiday Live Stream

My coding voice is very different. Yeah, because I can't keep up hours of shouting.


Machine Learning Holiday Live Stream

So what if we put X into the model?


Machine Learning Holiday Live Stream

That is not cool. OK. What if we put...


Machine Learning Holiday Live Stream

...if we unpack X into the model? That is better. Better. Better. Better.


Machine Learning Holiday Live Stream

What is this?


Machine Learning Holiday Live Stream

OK, let's call this Y.


Machine Learning Holiday Live Stream

And let's look at type of Y.


Machine Learning Holiday Live Stream

Investigate. Sequence to sequence language model output. Thank you. That is very helpful.


Machine Learning Holiday Live Stream

What can I... Is it a dict?


Machine Learning Holiday Live Stream

Yes, it is. OK, so we have a loss. We have a loss.


Machine Learning Holiday Live Stream

I guess that is...


Machine Learning Holiday Live Stream

...why do you always wear shades? So people ask the question.


Machine Learning Holiday Live Stream

Is branding. Is just branding.


Machine Learning Holiday Live Stream

OK, so we have this. Very cool.


Machine Learning Holiday Live Stream

Now...


Machine Learning Holiday Live Stream

So if we do this, we can even see it.


Machine Learning Holiday Live Stream

Now we can use our data set map.


Machine Learning Holiday Live Stream

Process data. What do they do here? Map.


Machine Learning Holiday Live Stream

I guess we can use...


Machine Learning Holiday Live Stream

...fatched and we can remove all the columns. No?


Machine Learning Holiday Live Stream

We can probably remove all the columns.


Machine Learning Holiday Live Stream

Module object is not subscript. Remove columns.


Machine Learning Holiday Live Stream

Data sets. Ah, this is it.


Machine Learning Holiday Live Stream

Mm-hmm. Mm-hmm.


Machine Learning Holiday Live Stream

It's not happy. It's not very happy.


Machine Learning Holiday Live Stream

List indices must be integers or slices.


Machine Learning Holiday Live Stream

OK, we should do batched false.


Machine Learning Holiday Live Stream

All right. I guess we should also drop...


Machine Learning Holiday Live Stream

We should probably also drop some of the too long stuff. No?


Machine Learning Holiday Live Stream

I mean, that's what they do here. They put a max length.


Machine Learning Holiday Live Stream

Return overflowing tokens. Ah, we don't want that.


Machine Learning Holiday Live Stream

We simply want the max length, right?


Machine Learning Holiday Live Stream

So, no. Stop it. Stop it.


Machine Learning Holiday Live Stream

OK, so tokenizer.


Machine Learning Holiday Live Stream

See, now is the trouble, right? The max length is what?


Machine Learning Holiday Live Stream

The max length is what for t5 small? Does anyone know?


Machine Learning Holiday Live Stream

Let's see, t5. Or can I figure this out somehow?


Machine Learning Holiday Live Stream

There should be this auto... There's this auto config, no? By hugging face?


Machine Learning Holiday Live Stream

Can I load this somehow? Somehow. Here, like...


Machine Learning Holiday Live Stream

There is... auto config from pre-trained t5 small.


Machine Learning Holiday Live Stream

No? This should exist.


Machine Learning Holiday Live Stream

Yeah, see?


Machine Learning Holiday Live Stream

So here we have task-specific parameters.


Machine Learning Holiday Live Stream

Model type, number of positions, number of layers, task-specific parameters,


Machine Learning Holiday Live Stream

vocab size, translation, summarization, max length.


Machine Learning Holiday Live Stream

Is this, though... Is this the max length of the encoder or the decoder?


Machine Learning Holiday Live Stream

I mean, here it's 512.


Machine Learning Holiday Live Stream

We can do the following. We can do...


Machine Learning Holiday Live Stream

We can do max length here to something...


Machine Learning Holiday Live Stream

I mean, what is... What is 512? Let's say minus 128. OK, and then...


Machine Learning Holiday Live Stream

And then here we can do 128.


Machine Learning Holiday Live Stream

Padding equals...


Machine Learning Holiday Live Stream

How about that?


Machine Learning Holiday Live Stream

So we pad everything max length, min length.


Machine Learning Holiday Live Stream

230. Yeah, thanks. Thank you. So...


Machine Learning Holiday Live Stream

We'll see. We'll make everything such that it is at max 512 long.


Machine Learning Holiday Live Stream

We can still change these things. We should probably make some variables somewhere, but...


Machine Learning Holiday Live Stream

Yeah.


Machine Learning Holiday Live Stream

So, good. Now that the dataset is mapping, what can we do?


Machine Learning Holiday Live Stream

I guess all we have to do is give it to the trainer. No?


Machine Learning Holiday Live Stream

Hugging face trainer.


Machine Learning Holiday Live Stream

Mm-hmm.


Machine Learning Holiday Live Stream

Compute loss, yeah, and then...


Machine Learning Holiday Live Stream

Do we have examples right here?


Machine Learning Holiday Live Stream


Machine Learning Holiday Live Stream

Can we just get a trainer or do we need the specific one?


Machine Learning Holiday Live Stream

Evaluate? No. Predict? No.


Machine Learning Holiday Live Stream

Training arguments. Ooh, that is a...


Machine Learning Holiday Live Stream

Model.


Machine Learning Holiday Live Stream

We put in model.


Machine Learning Holiday Live Stream

Yep.


Machine Learning Holiday Live Stream

Train dataset, evaluate dataset.


Machine Learning Holiday Live Stream

Yeah, this should be doable, no?


Machine Learning Holiday Live Stream

Here, trainer, we just give it...


Machine Learning Holiday Live Stream

This, and we're good.


Machine Learning Holiday Live Stream

Really? Hmm.


Machine Learning Holiday Live Stream

We could make it even smaller, given that it's going to take a while.


Machine Learning Holiday Live Stream

Also...


Machine Learning Holiday Live Stream

We could just do this. Let's just cancel for now. How long does it take?


Machine Learning Holiday Live Stream

What?


Machine Learning Holiday Live Stream

But they did it. They did it up here.


Machine Learning Holiday Live Stream

They did it here, no? They made this smaller dataset.


Machine Learning Holiday Live Stream

Um...


Machine Learning Holiday Live Stream

What do I need to do? No?


Machine Learning Holiday Live Stream

Here. Here is a dataset.


Machine Learning Holiday Live Stream

Okay, I guess they just took the training one.


Machine Learning Holiday Live Stream

Like this.


Machine Learning Holiday Live Stream

Okay.


Machine Learning Holiday Live Stream

Excellent. So now we have a dataset and we should just be able to give this to a trainer.


Machine Learning Holiday Live Stream

Yes, yes. Okay, so let's get some of them. Training arguments.


Machine Learning Holiday Live Stream

Good. And let's also get some of them. Trainer itself. Good.


Machine Learning Holiday Live Stream

Yeah, okay. That's where the 384 comes from. Thank you.


Machine Learning Holiday Live Stream

Okay, model is model. This is that.


Machine Learning Holiday Live Stream

And I guess... Okay, so...


Machine Learning Holiday Live Stream

And...


Machine Learning Holiday Live Stream

This and that. Good.


Machine Learning Holiday Live Stream

So here we have...


Machine Learning Holiday Live Stream

Train dataset and eval dataset.


Machine Learning Holiday Live Stream

This seems to work so far. This is more like magic.


Machine Learning Holiday Live Stream

How... Where is all of this...


Machine Learning Holiday Live Stream

Like... Okay.


Machine Learning Holiday Live Stream

There seems to be so much magic going on and this is...


Machine Learning Holiday Live Stream

Okay, I see. Finally, finally we get something.


Machine Learning Holiday Live Stream

Something that crashes.


Machine Learning Holiday Live Stream

So...


Machine Learning Holiday Live Stream

What did we do?


Machine Learning Holiday Live Stream

Input shape. Too many values to unpack in the input shape.


Machine Learning Holiday Live Stream

So where are we going wrong?


Machine Learning Holiday Live Stream

Too big context? Nah. We have too many...


Machine Learning Holiday Live Stream

We have too many things. So here we might be...


Machine Learning Holiday Live Stream

We might want to inspect this X a little more. So these are our input IDs, right? So...


Machine Learning Holiday Live Stream

Input IDs are right here.


Machine Learning Holiday Live Stream

And if we do the shape of them, we get two numbers.


Machine Learning Holiday Live Stream

Now this is complaining that if it looks at the input shape, right?


Machine Learning Holiday Live Stream

So if we go to T5 modeling...


Machine Learning Holiday Live Stream

Wow! This opens in a Colab directly.


Machine Learning Holiday Live Stream

This is crazy.


Machine Learning Holiday Live Stream

Okay, if we go to the input shape...


Machine Learning Holiday Live Stream

This is the input embeddings.


Machine Learning Holiday Live Stream

No, that's not true. See the input IDs... Oopsie.


Machine Learning Holiday Live Stream

Dot size.


Machine Learning Holiday Live Stream

Well, what is it? Aren't you going to tell me? Too many values to unpack? Well, what is it?


Machine Learning Holiday Live Stream

So...


Machine Learning Holiday Live Stream

Maybe we need to batch this ourselves somehow, right?


Machine Learning Holiday Live Stream

Because...


Machine Learning Holiday Live Stream

Here, as you can see, the shape... Where was it? Is 1 and 384.


Machine Learning Holiday Live Stream

So this already assumes that it's kind of batched.


Machine Learning Holiday Live Stream

And...


Machine Learning Holiday Live Stream

I might be completely wrong right here.


Machine Learning Holiday Live Stream

Unpack is about the input shape being only one object. Try printing the shape. Yeah.


Machine Learning Holiday Live Stream

I can print. Can I print in here?


Machine Learning Holiday Live Stream

But it seems the problem is the batching, no?


Machine Learning Holiday Live Stream

Does this work?


Machine Learning Holiday Live Stream

This seems weird. Like if this works, this is...


Machine Learning Holiday Live Stream

This is magic.


Machine Learning Holiday Live Stream

I've not developed in a Colab in so long.


Machine Learning Holiday Live Stream

No, right? This does not work.


Machine Learning Holiday Live Stream

So if...


Machine Learning Holiday Live Stream

All changes saved, yeah?


Machine Learning Holiday Live Stream

Ah, okay.


Machine Learning Holiday Live Stream

Now it says too many values to... Yeah, it did not load, reload this thing. Can I... Sorry.


Machine Learning Holiday Live Stream

Click the six frames.


Machine Learning Holiday Live Stream

Yep. Well...


Machine Learning Holiday Live Stream

It just tells me where it's happening.


Machine Learning Holiday Live Stream

Which I know, right? I just...


Machine Learning Holiday Live Stream

Compute loss.


Machine Learning Holiday Live Stream

Models. Unpack the inputs. That's fine.


Machine Learning Holiday Live Stream

I think it's the...


Machine Learning Holiday Live Stream

The collate function is wrong. It assumes that what comes out of the data set is a single example.


Machine Learning Holiday Live Stream

So it probably batches it together.


Machine Learning Holiday Live Stream

Need to use auto reload too.


Machine Learning Holiday Live Stream

Thank you.


Machine Learning Holiday Live Stream

Let's try.


Machine Learning Holiday Live Stream

Auto reload.


Machine Learning Holiday Live Stream

Magic function not found.


Machine Learning Holiday Live Stream

Too bad.


Machine Learning Holiday Live Stream

It probably assumes that. So what we could do is we could...


Machine Learning Holiday Live Stream

Simply tell it that this is not a batched thing.


Machine Learning Holiday Live Stream

Right.


Machine Learning Holiday Live Stream

I found at least two devices.


Machine Learning Holiday Live Stream

Great. Now we're...


Machine Learning Holiday Live Stream

Okay.


Machine Learning Holiday Live Stream

Process data.


Machine Learning Holiday Live Stream

Ah...


Machine Learning Holiday Live Stream

How about that?


Machine Learning Holiday Live Stream

No attribute detach.


Machine Learning Holiday Live Stream

Yeah. Okay.


Machine Learning Holiday Live Stream

Sounds good.


Machine Learning Holiday Live Stream

We'll try again.


Machine Learning Holiday Live Stream

Too many values to unpack.


Machine Learning Holiday Live Stream

Still no.


Machine Learning Holiday Live Stream

Yeah, it still thinks it's there.


Machine Learning Holiday Live Stream

But it doesn't matter. We have bigger problems.


Machine Learning Holiday Live Stream

We have bigger problems. Look at this thing.


Machine Learning Holiday Live Stream

Now where was it? Here.


Machine Learning Holiday Live Stream

Right. So we need to...


Machine Learning Holiday Live Stream

Do we really need to get back... Which ones are on?


Machine Learning Holiday Live Stream

You need to run auto reload to afterload.


Machine Learning Holiday Live Stream

Okay. Okay. Okay. Okay.


Machine Learning Holiday Live Stream

Okay. So see, I'm so unfamiliar with Colabs because I've just been using my Vim in my terminal.


Machine Learning Holiday Live Stream

That I have no clue about the magic functions.


Machine Learning Holiday Live Stream

Okay, so...


Machine Learning Holiday Live Stream

There. Yeah, there you go. Exactly. So this is what I meant.


Machine Learning Holiday Live Stream

It batches it to eight. It assumes that what comes out of the data set is already a batch.


Machine Learning Holiday Live Stream

And therefore,


Machine Learning Holiday Live Stream

it doesn't get it. So either we somehow tell the tokenizer or something that... Or we tell the collate function.


Machine Learning Holiday Live Stream

Collater default.


Machine Learning Holiday Live Stream

We'll batch our processed examples together.


Machine Learning Holiday Live Stream

Let's see. There's the default one.


Machine Learning Holiday Live Stream

Label IDs handles a list of values per object.


Machine Learning Holiday Live Stream

Maybe this one, data collater for SICK to tokenizer.


Machine Learning Holiday Live Stream

Dynamically pad the input as well as the labels. Maybe this one's good for us.


Machine Learning Holiday Live Stream

So let's try this one.


Machine Learning Holiday Live Stream

Yes, I'll make a video about my Vim config.


Machine Learning Holiday Live Stream

Okay, all tensors need to be on the same device. Found at least two devices.


Machine Learning Holiday Live Stream

How can I... Where is my data set? So how about...


Machine Learning Holiday Live Stream

I look at... What's X?


Machine Learning Holiday Live Stream

What is X.inputIDs? What's the device of that?


Machine Learning Holiday Live Stream

CPU. Okay.


Machine Learning Holiday Live Stream

So how do I put that on? And X is what? X is just...


Machine Learning Holiday Live Stream

X is a data point.


Machine Learning Holiday Live Stream

Now...


Machine Learning Holiday Live Stream

Can I just tell HoggingFace to... Or do I have to actually say...


Machine Learning Holiday Live Stream

Did the model have a device?


Machine Learning Holiday Live Stream

Okay.


Machine Learning Holiday Live Stream

So now what we'll have to do is we'll have to input IDs equals X input IDs to model device and labels equals X labels to model device.


Machine Learning Holiday Live Stream

Yeah, okay. So now we got sort of the same...


Machine Learning Holiday Live Stream

We got the same problem here.


Machine Learning Holiday Live Stream

So we don't want to necessarily do this.


Machine Learning Holiday Live Stream

We want to output this because after a while we could also overflow and then output several... Could we put several different ones?


Machine Learning Holiday Live Stream

No, I guess we can continue doing this. And then here we'll just have to...


Machine Learning Holiday Live Stream

Do something like that or expand. We'll just go none.


Machine Learning Holiday Live Stream

Do a little unsqueeze. Nice. Okay.


Machine Learning Holiday Live Stream

So now we still... Oh no.


Machine Learning Holiday Live Stream

Now we have... We need to remove the print statement again, otherwise we'll just be bombarded by print statements, right?


Machine Learning Holiday Live Stream

Okay, let's do this. Let's do that.


Machine Learning Holiday Live Stream

Please save. Thank you.


Machine Learning Holiday Live Stream

Good.


Machine Learning Holiday Live Stream

Problem solved.


Machine Learning Holiday Live Stream

Happy holiday. Thank you. Happy holiday to you too.


Machine Learning Holiday Live Stream

Good. We're at the next error.


Machine Learning Holiday Live Stream

Expected a sequence length of 384 at dimension 1. Got 517. What? What?


Machine Learning Holiday Live Stream

We even limited it to...


Machine Learning Holiday Live Stream

We limited this.


Machine Learning Holiday Live Stream

Got 512 data collater.


Machine Learning Holiday Live Stream

We're here.


Machine Learning Holiday Live Stream

Label. Do we have label or label IDs? We should have label IDs, no?


Machine Learning Holiday Live Stream

Because... Yeah. We should have label IDs.


Machine Learning Holiday Live Stream

That seems reasonable.


Machine Learning Holiday Live Stream

Still the same. Okay. So the problem is that here... What's f in features?


Machine Learning Holiday Live Stream

What's features?


Machine Learning Holiday Live Stream

What is it?


Machine Learning Holiday Live Stream

List of dict f of k. So what's k?


Machine Learning Holiday Live Stream

If k is not in label and v is not none...


Machine Learning Holiday Live Stream

And here we say expected a sequence of length. Got... Okay.


Machine Learning Holiday Live Stream

We can do this and we can print the length of that and let's see what happens.


Machine Learning Holiday Live Stream

384, 384, 384.


Machine Learning Holiday Live Stream

And then we get like a long one.


Machine Learning Holiday Live Stream

Why is it not respecting the max length?


Machine Learning Holiday Live Stream

So this one... Let's see.


Machine Learning Holiday Live Stream


Machine Learning Holiday Live Stream

Hmm...


Machine Learning Holiday Live Stream

Yes, max length. Pad to max length. Max length.


Machine Learning Holiday Live Stream

I'm just searching for this now. In all the things I have open.


Machine Learning Holiday Live Stream

Yeah, I should be able to truncation true. Is that a thing? Okay.


Machine Learning Holiday Live Stream

Let's see. Auto models. Tokenizer.


Machine Learning Holiday Live Stream

Pre-trained tokenizer. Can I click on it? Please.


Machine Learning Holiday Live Stream

Yes. No.


Machine Learning Holiday Live Stream

This is version 2.


Machine Learning Holiday Live Stream

Sure.


Machine Learning Holiday Live Stream

There we go. Max length.


Machine Learning Holiday Live Stream

It's called model max length, really.


Machine Learning Holiday Live Stream

Truncation.


Machine Learning Holiday Live Stream

Defaults to false. Well, that explains something.


Machine Learning Holiday Live Stream

Let's in this case do true.


Machine Learning Holiday Live Stream

How about now?


Machine Learning Holiday Live Stream

384, 384, 384, 384.


Machine Learning Holiday Live Stream

This looks better. Okay.


Machine Learning Holiday Live Stream

Now we just need to remove the print statements.


Machine Learning Holiday Live Stream

Otherwise the file will be gone.


Machine Learning Holiday Live Stream

Cool.


Machine Learning Holiday Live Stream

Okay, so this isn't too long. This isn't five hours like before.


Machine Learning Holiday Live Stream

So hopefully, hopefully something cool will happen.


Machine Learning Holiday Live Stream

Truncation. You throw away everything after the first 384 tokens. Is that on purpose? Yes.


Machine Learning Holiday Live Stream

We can still refine like it.


Machine Learning Holiday Live Stream

Keep these tokens that it's handled in the... Yeah, the tokenizer can sort of split and, and...


Machine Learning Holiday Live Stream

Tokenizer can stride over the document. But in first instance, I'm just happy with getting any sort of result.


Machine Learning Holiday Live Stream

Can still be refined later, but we just want to train anything at all.


Machine Learning Holiday Live Stream

Well, this is the fun part. So I think people sometimes ask, you know, about streaming and machine learning.


Machine Learning Holiday Live Stream

And, you know, streaming video games is sort of the most like super entertaining, right?


Machine Learning Holiday Live Stream

Because it's like stuff is happening. And then streaming coding is still okay if you can talk and code and read chat and so on at the same time.


Machine Learning Holiday Live Stream

But then streaming machine learning is just sort of...


Machine Learning Holiday Live Stream

Even if we make the data sets deliberately small like we did here, it is just...


Machine Learning Holiday Live Stream

It is, it is... We just have to wait.


Machine Learning Holiday Live Stream

And then what do we do during waiting?


Machine Learning Holiday Live Stream

Could you show line data set equals? Data set equals...


Machine Learning Holiday Live Stream

Well, we have data set is simply loading it here from... We simply load the squat data set.


Machine Learning Holiday Live Stream

So that's kind of trivial. But then we have this mapping function here that tokenizes the text and puts it into sort of this framework context and answer.


Machine Learning Holiday Live Stream

And we want to output the question.


Machine Learning Holiday Live Stream

What do you do during wait normally?


Machine Learning Holiday Live Stream

I watch... I love watching loading bars and screens.


Machine Learning Holiday Live Stream

But usually I... I don't know. I code something else.


Machine Learning Holiday Live Stream

I guess there is something else to code, which is... I'm making like a super... Sure, doing an AMA.


Machine Learning Holiday Live Stream

Am I not doing an AMA during all of the live stream Swordfight? Yes, the XKCD meme.


Machine Learning Holiday Live Stream

So I see if I have enough GPU left. That's a tight rope, no?


Machine Learning Holiday Live Stream

Try to stream some reinforcement learning. Watch an agent not really working.


Machine Learning Holiday Live Stream

Yeah, if you get the visualization set up correctly, that could be something. Also like smaller problems, right? Like way smaller problems is also something.


Machine Learning Holiday Live Stream

This here I sort of wanted to try. And oh, we can look at our... We can look at the...


Machine Learning Holiday Live Stream

This here, it should be... Yeah, yeah. Look at this. At least the CPU utilization is going on. That is something that's happening.


Machine Learning Holiday Live Stream

We don't have metrics yet. We might want to log more intermittently. Training completed.


Machine Learning Holiday Live Stream

Training completed. Excellent. We have zero logging. We did log, right? It said it logs to...


Machine Learning Holiday Live Stream

It logs to report to...


Machine Learning Holiday Live Stream

Didn't it say it would log to weights and biases? Did we even log in?


Machine Learning Holiday Live Stream

WondB. We didn't log in. We didn't log in.


Machine Learning Holiday Live Stream

Oh, we did. Somewhere. At some point, maybe.


Machine Learning Holiday Live Stream

Am I planning to do the ML year recap video? Did I announce an ML year recap? I don't know. Alpha fold happened or something. Was it this year or last year?


Machine Learning Holiday Live Stream

I have no idea. The paper of the year. No idea. It's really hard to predict what paper is going to have a big impact.


Machine Learning Holiday Live Stream

For example, the foundation models paper. It can either have a really big impact or it can just be forgotten by everyone.


Machine Learning Holiday Live Stream

There's no way to tell right now.


Machine Learning Holiday Live Stream

What do you think about the recent hype in graph neural nets? I've not looked super much into graph neural nets.


Machine Learning Holiday Live Stream

It seems they're pretty cool, but they also seem to be quite limited right now because they're essentially RNNs on a dynamic graph.


Machine Learning Holiday Live Stream

Check the report. Yeah, so WondB has a trainer. So there should be it should be possible.


Machine Learning Holiday Live Stream

Here, report to WondB. That's fine. With the training arguments. Good stuff. We might want to restart and run all.


Machine Learning Holiday Live Stream

OK, now once the model has trained, we can then let it generate some stuff. I really don't think that it will produce something interesting right now.


Machine Learning Holiday Live Stream

OK, good stuff. Reusing the data set. That's what we want. OK, here. Yes.


Machine Learning Holiday Live Stream

This one. OK, as this is going to come in, we can already think about what we want to do.


Machine Learning Holiday Live Stream

So what we want to do is we want to simply take the model that we have and let it generate something again.


Machine Learning Holiday Live Stream

So let's just try. Let's just try this again. I think. No. No, like.


Machine Learning Holiday Live Stream

Try this. And where was the generate? Here. And then we do batch decode.


Machine Learning Holiday Live Stream

There we go.


Machine Learning Holiday Live Stream

Aren't graph neural networks just a year behind NLP or attention? I'm not sure what a year behind.


Machine Learning Holiday Live Stream

You mean like in a year they're going to be as popular? It could be.


Machine Learning Holiday Live Stream

I've also seen like there's also a big history of such things just flopping like there being a lot of hype and then not much.


Machine Learning Holiday Live Stream

It depends. It depends if sort of large breakthroughs, I guess, are being made in relevant domains.


Machine Learning Holiday Live Stream

That's usually what fuels these things like transformers. I think BERT has transformers have existed.


Machine Learning Holiday Live Stream

And I remember I remember people sort of the sentiment was, yeah, transformers for language, they kind of work, but they use so much memory and they're harder to train and so on.


Machine Learning Holiday Live Stream

And people would still sort of be on the be on the fence and say we'd rather use LSTMs with attention.


Machine Learning Holiday Live Stream

And I think only the advent of BERT really say it wasn't even the attention is all you need paper.


Machine Learning Holiday Live Stream

It was the BERT paper that really fueled the transformer hype.


Machine Learning Holiday Live Stream

So if there is something similar for graph neural networks in a relevant domain, so either in a domain that exists like but I don't I don't really see graph neural networks being that useful in vision or or NLP or something.


Machine Learning Holiday Live Stream

So I'm not entirely sure. They are probably super useful in biology.


Machine Learning Holiday Live Stream

But it's not like biology is that it's just not that accessible to most people.


Machine Learning Holiday Live Stream

And they don't want to work in biology because if I'm good at like if I do something in NLP, like I can build a little app and so on.


Machine Learning Holiday Live Stream

It's very tangible and very useful for most people. And same with vision.


Machine Learning Holiday Live Stream

But with something like biological data, I mean, yeah, sure, I can download some bio data from from online.


Machine Learning Holiday Live Stream

But then, you know what? What am I going to do with it?


Machine Learning Holiday Live Stream

Are you familiar with PINNS? No, I'm not actually.


Machine Learning Holiday Live Stream

Socks, 1 dB socks giveaway. Well, if you if I'm I don't know, I should wear them probably.


Machine Learning Holiday Live Stream

Yeah. Sorry, I'm trying to keep up with chat. Most interesting meta learning paper.


Machine Learning Holiday Live Stream

I'm not I'm not super I don't have a maybe not anyone, but I don't have a good overview over meta learning.


Machine Learning Holiday Live Stream

According to a Luther graph, neural networks are largely NLP research from 2019 and 2020.


Machine Learning Holiday Live Stream

If you want to achieve Soda, you'd have to use 2020.


Machine Learning Holiday Live Stream

So you're saying that current NLP methods beat the graph neural network methods.


Machine Learning Holiday Live Stream

So when people in the past said, hey, these graph neural networks are actually pretty good.


Machine Learning Holiday Live Stream

Then all you had to do essentially is wait a year and then the field would have sort of caught up.


Machine Learning Holiday Live Stream

I guess I hope that's what you're saying. OK, so now we're sort of kind of a bit ready.


Machine Learning Holiday Live Stream

No. Yes. So here, let's say. Yes, we can generate a context. Hi.


Machine Learning Holiday Live Stream

The dog was loud and.


Machine Learning Holiday Live Stream

The answer is dog. Right. And then end of sentence.


Machine Learning Holiday Live Stream

OK, and then we have. Yes. Does that does that work?


Machine Learning Holiday Live Stream

OK, so input IDs and then we'll put them to. Device.


Machine Learning Holiday Live Stream

Like this, like this. How about this?


Machine Learning Holiday Live Stream

Ha ha. You have to. Either decoder input IDs. OK, so.


Machine Learning Holiday Live Stream

And code, let's encode just the. I see.


Machine Learning Holiday Live Stream

Ha. What am I going to put here?


Machine Learning Holiday Live Stream

How do I how do I have T5 answer a question then?


Machine Learning Holiday Live Stream

No. How?


Machine Learning Holiday Live Stream

Will there be Yannick merge sometime soon? Yes. Yes. There is. There's merge.


Machine Learning Holiday Live Stream

It's I mean, I'm still I'm still fleshing out the merge, right? I'm I'm still this is a changing thing.


Machine Learning Holiday Live Stream

If you I guess you can have it today. But I'm just telling you this is it's going to change.


Machine Learning Holiday Live Stream

So if you're into super rare stuff, maybe it's going to be deleted soon.


Machine Learning Holiday Live Stream

It's also going to be augmented. But let me.


Machine Learning Holiday Live Stream

Let me show you.


Machine Learning Holiday Live Stream

Yeah, OK. It's up. So but as I said, this is subject to change.


Machine Learning Holiday Live Stream

And and so on. But if you so you can either go here or you can go the URL is.


Machine Learning Holiday Live Stream

Store.YKilcher.com. That's it. That should be a redirect to the it's a Teespring store. So and it's going to be integrated with super rare, super rare in the sense that I might decide something is just too dumb of an item and just remove it again because I need to thin it out a little bit.


Machine Learning Holiday Live Stream

But, you know, no one stops you. I'm just taking like if you if you really let's say I remove this T-shirt, right? No one stops you from just taking the channel logo and slapping it on a T-shirt yourself.


Machine Learning Holiday Live Stream

It's going to be literally exactly the same thing. So, you know, the rarity is the rarity is is only because people are usually too lazy to do that.


Machine Learning Holiday Live Stream

So, yeah, I'm going to probably thin this out a little bit. But I've ordered some samples and I think I'll announce it once I have the samples ready.


Machine Learning Holiday Live Stream

There's also one. So it's usually it's the same like it's the channel logo on various items.


Machine Learning Holiday Live Stream

It's sort of the the channel model, which I don't usually highlight, but skill greater than destiny is kind of the I don't know. It's been sort of a battle cry since I was young.


Machine Learning Holiday Live Stream

It yeah.


Machine Learning Holiday Live Stream

It sort of it sort of says that you can if you if you dedicate yourself, you can overcome sort of the maybe the hurdles of life, whatever.


Machine Learning Holiday Live Stream

It's a bit cheesy, but actually doesn't mean like if you take it literally, it doesn't even make sense.


Machine Learning Holiday Live Stream

And because both are predetermined, like your skill is maybe predetermined. Sure, you can learn something, but and then your destiny is like ultimately predetermined.


Machine Learning Holiday Live Stream

So if there is destiny, you can't change it. But maybe the paradoxical nature of it is what makes it good and appealing.


Machine Learning Holiday Live Stream

And obviously there is the attention for you if you're sewing, if you enjoy that. And yeah, I think that's I'm thinking of a technology good technology, bad technology biased motif as well.


Machine Learning Holiday Live Stream

But that's that if you want to want to do this.


Machine Learning Holiday Live Stream

But as I said, it's not announced yet. It's going to change. And there's also there's there's more stuff coming out.


Machine Learning Holiday Live Stream

In any case, let's like how do we how do we actually generate stuff from this?


Machine Learning Holiday Live Stream

Like, okay, we have a language modeling head here. But what do we have here? We have an auto model also with with a language modeling head.


Machine Learning Holiday Live Stream

Huh? So we should we should provide some inputs. So let's say, okay, how about we make our data set such that it always starts with a Q, right? Like this.


Machine Learning Holiday Live Stream

And then we could put here we could put like Q.


Machine Learning Holiday Live Stream

I'm not making NFTs. No, no, no. Unless they actually become more useful. I'm not I'm not going to make NFTs. Don't worry about that.


Machine Learning Holiday Live Stream

So we got decoder input IDs.


Machine Learning Holiday Live Stream

Yes, outputs. Wait, that's what that's what we that's what we did.


Machine Learning Holiday Live Stream

Oh, OK.


Machine Learning Holiday Live Stream

Oh, no, wait, is it really? Could we do that?


Machine Learning Holiday Live Stream

Aha.


Machine Learning Holiday Live Stream

OK. Well, good. Yes.


Machine Learning Holiday Live Stream

Most definitely. Totally. It's a good model so far.


Machine Learning Holiday Live Stream

OK. How do I the hugging face trainer? You know, how do I control its its logging? How do I control it? Log level, log level replica.


Machine Learning Holiday Live Stream

Like how often how often it logs.


Machine Learning Holiday Live Stream

Yeah, yeah.


Machine Learning Holiday Live Stream

The logging frequency. Yes, that's what I want.


Machine Learning Holiday Live Stream

Frequency.


Machine Learning Holiday Live Stream

Hey, where is it?


Machine Learning Holiday Live Stream

Can't find it.


Machine Learning Holiday Live Stream

Where is it?


Machine Learning Holiday Live Stream

OK, no.


Machine Learning Holiday Live Stream

Logging steps.


Machine Learning Holiday Live Stream

Does this exist?


Machine Learning Holiday Live Stream

There we go. Logging steps.


Machine Learning Holiday Live Stream

Number of update steps between two logs if logging strategy equals steps. OK.


Machine Learning Holiday Live Stream

So this is steps.


Machine Learning Holiday Live Stream

And we'll just set it to.


Machine Learning Holiday Live Stream

Is this a trainer or is this the training arguments?


Machine Learning Holiday Live Stream

Training arguments. OK.


Machine Learning Holiday Live Stream

So in our training arguments.


Machine Learning Holiday Live Stream

Wait, we might want to reprocess the data set, right?


Machine Learning Holiday Live Stream

And here now let's do this because.


Machine Learning Holiday Live Stream

Oh no, it's being tokenized like that. Sorry, sorry.


Machine Learning Holiday Live Stream

Uh huh. OK.


Machine Learning Holiday Live Stream

Training arguments, logging steps 10.


Machine Learning Holiday Live Stream

Train. Yes, please.


Machine Learning Holiday Live Stream

And now we should finally see something.


Machine Learning Holiday Live Stream

Here. Hopefully, please.


Machine Learning Holiday Live Stream

Runs.


Machine Learning Holiday Live Stream

This is green, so this means it's going.


Machine Learning Holiday Live Stream

Yes.


Machine Learning Holiday Live Stream

Yay.


Machine Learning Holiday Live Stream

And we are decreasing in loss. Excellent.


Machine Learning Holiday Live Stream

Oh wait, I need to catch up in chat.


Machine Learning Holiday Live Stream

For how long will you stream? Not for too much longer.


Machine Learning Holiday Live Stream

So maybe last year during I remember during Christmas.


Machine Learning Holiday Live Stream

You know, everything's kinda.


Machine Learning Holiday Live Stream

And still during Christmas, right? And I remember PewDiePie streaming and I thought, oh, that's neat because, you know, nothing else is going on.


Machine Learning Holiday Live Stream

So and.


Machine Learning Holiday Live Stream

Yeah, so I thought I'd do the same thing and maybe someone.


Machine Learning Holiday Live Stream

Well, appreciate it.


Machine Learning Holiday Live Stream

Or just be sort of mildly entertained while they search for other content.


Machine Learning Holiday Live Stream

Will it be delivered all over the planet? The merch? I think so.


Machine Learning Holiday Live Stream

So I've deliberately I first wanted to go with a different like with what was it?


Machine Learning Holiday Live Stream

Don't remember the other one, but there was another one that could be integrated with YouTube, but they are not shipping to India right now, which is kind of I thought kind of sucked.


Machine Learning Holiday Live Stream

So I think the Teespring should be able to ship anywhere.


Machine Learning Holiday Live Stream

Don't feed in the decoder input IDs manually.


Machine Learning Holiday Live Stream

It makes the outputs go weird. Well, it won't let me do anything else.


Machine Learning Holiday Live Stream

Yeah, so I have not done Hugging Face training in a long time from scratch.


Machine Learning Holiday Live Stream

But this is just the first the first instance.


Machine Learning Holiday Live Stream

So we should be. Yay. Yeah, we're we're nicely decreasing.


Machine Learning Holiday Live Stream

No, not towards the end here.


Machine Learning Holiday Live Stream

Is this one epoch?


Machine Learning Holiday Live Stream

Three epochs.


Machine Learning Holiday Live Stream

OK.


Machine Learning Holiday Live Stream

Batch size of eight. Yeah, that works out. 125 per epoch.


Machine Learning Holiday Live Stream

And then we'll see what happens.


Machine Learning Holiday Live Stream

In 35 seconds.


Machine Learning Holiday Live Stream

Cool.


Machine Learning Holiday Live Stream

Dun dun dun dun dun.


Machine Learning Holiday Live Stream

Do not forget to share your model. I don't think this model is necessarily the best to be shared.


Machine Learning Holiday Live Stream

OK, so we still.


Machine Learning Holiday Live Stream

We're still kind of.


Machine Learning Holiday Live Stream

Yep.


Machine Learning Holiday Live Stream

So the model isn't the best right now.


Machine Learning Holiday Live Stream

What data set are using its squad. So we're using squad.


Machine Learning Holiday Live Stream

But I think this would be sort of the process.


Machine Learning Holiday Live Stream

A sort of the process to get this started. Right.


Machine Learning Holiday Live Stream

I mean, what if I just don't do anything here?


Machine Learning Holiday Live Stream

What does it do? It just gives me like an end of end of sentence immediately.


Machine Learning Holiday Live Stream

Is this to be expected to it? Does the.


Machine Learning Holiday Live Stream

Does this even need an end of sentence?


Machine Learning Holiday Live Stream

Not sure.


Machine Learning Holiday Live Stream

So the question does does the model is the model initializing.


Machine Learning Holiday Live Stream

Like if I reset the model like like this.


Machine Learning Holiday Live Stream

Then. OK, this is a lot of output.


Machine Learning Holiday Live Stream

If I do that and then train again, what happens?


Machine Learning Holiday Live Stream

Yeah, OK.


Machine Learning Holiday Live Stream

No, I don't want to learn about the newest weights and biases updates because.


Machine Learning Holiday Live Stream

I make ads for them.


Machine Learning Holiday Live Stream

I should know about the newest features and updates.


Machine Learning Holiday Live Stream

So this is better right now because before we just kept retraining the same model, right?


Machine Learning Holiday Live Stream

And not reinitialize it. And now at least we're at least we're reinitializing it.


Machine Learning Holiday Live Stream

See, this was like our global step was.


Machine Learning Holiday Live Stream

We just kept resetting the same model. But now now.


Machine Learning Holiday Live Stream

You see. Do we have is this the is the one that we currently have?


Machine Learning Holiday Live Stream

No.


Machine Learning Holiday Live Stream

We'll see in a bit once it sinks.


Machine Learning Holiday Live Stream

But even if not, you can see here loss is going down quite nicely.


Machine Learning Holiday Live Stream

Anyone know if one to be self-hosting is fairly seamless training on a local GPU server to watch progress.


Machine Learning Holiday Live Stream

So the I think that like I might be wrong, but the self-hosting for one to be is is mostly a thing you do as a as a company.


Machine Learning Holiday Live Stream

It's not it's not the same as like.


Machine Learning Holiday Live Stream

It's not like, you know, here is our open source tool. And, you know, we also host it for you if you want.


Machine Learning Holiday Live Stream

They sort of take the different approach. It's it's cloud first.


Machine Learning Holiday Live Stream

And if you are if you're a business and you say, well, I really need the data to stay in my in my in my data centers in my own environment, then they'll give they'll they'll give you sort of a self-hosted environment.


Machine Learning Holiday Live Stream

I don't know how easy it is, though, to get to one. If you just say, well, I'm an individual and I'm just.


Machine Learning Holiday Live Stream

Maybe. Yeah.


Machine Learning Holiday Live Stream

I don't know, but.


Machine Learning Holiday Live Stream

I guess you can ask them.


Machine Learning Holiday Live Stream

So soon we should have this done.


Machine Learning Holiday Live Stream

And then we can see then we'll have a model that's kind of trained from scratch.


Machine Learning Holiday Live Stream

That shouldn't be too bad. No.


Machine Learning Holiday Live Stream

This is for research you're working on just trying out something. This is just trying out something I haven't done. I haven't actually done NLP coding in a while.


Machine Learning Holiday Live Stream

So and.


Machine Learning Holiday Live Stream

I just randomly wanted to.


Machine Learning Holiday Live Stream

Or I just randomly thought that I might might as well live stream it.


Machine Learning Holiday Live Stream

You know, maybe we're not we're not doing a whole lot here.


Machine Learning Holiday Live Stream

We'll see.


Machine Learning Holiday Live Stream

So we got three hundred and seventy five steps. We're almost done.


Machine Learning Holiday Live Stream

Do we get some metrics here in the meantime?


Machine Learning Holiday Live Stream

Not yet. Maybe I need to give it a different name, though. Right.


Machine Learning Holiday Live Stream

I think.


Machine Learning Holiday Live Stream

Like because I'm always logging to the same name. That can't be too good.


Machine Learning Holiday Live Stream

What am I working on right now?


Machine Learning Holiday Live Stream

I've co-founded a startup, an NLP startup, ironically. Right.


Machine Learning Holiday Live Stream

Which means that I work with a lot of smart people that all also know NLP. Therefore, I don't get to code NLP anymore.


Machine Learning Holiday Live Stream

I get to code. I get to code mostly plumbing.


Machine Learning Holiday Live Stream

OK, so. So as you can see, let me.


Machine Learning Holiday Live Stream

Let me train.


Machine Learning Holiday Live Stream

Yeah, let me get a data point from here and we'll take this right here.


Machine Learning Holiday Live Stream

And we'll we'll make our.


Machine Learning Holiday Live Stream

Prepare data.


Machine Learning Holiday Live Stream

Of this one.


Machine Learning Holiday Live Stream

How do we call it? Process data, of course.


Machine Learning Holiday Live Stream

Is there any model for transformers that don't need N squared paper?


Machine Learning Holiday Live Stream

No, but they do have pseudo code, I believe, in the paper itself.


Machine Learning Holiday Live Stream

How does your day look like? It's a wild mess of whatever burns the most.


Machine Learning Holiday Live Stream

It's I do not advise.


Machine Learning Holiday Live Stream

I do not advise following my daily routine.


Machine Learning Holiday Live Stream

OK, so let's call this X1.


Machine Learning Holiday Live Stream

And let's input that into the model.


Machine Learning Holiday Live Stream

So X1 input IDs.


Machine Learning Holiday Live Stream

Not enough values to unpack. Where? How? Why? No. What? Come on.


Machine Learning Holiday Live Stream

Come on, man.


Machine Learning Holiday Live Stream

We'll do this.


Machine Learning Holiday Live Stream

List. It needs to be a tensor. So.


Machine Learning Holiday Live Stream

That.


Machine Learning Holiday Live Stream

Beautiful. Beautiful.


Machine Learning Holiday Live Stream

Uh. Uh. Uh.


Machine Learning Holiday Live Stream

Our model is terrible.


Machine Learning Holiday Live Stream

How and why? Right.


Machine Learning Holiday Live Stream

It just.


Machine Learning Holiday Live Stream

Baffles me. What if we don't train it at all? Like, what if what if we just load the model?


Machine Learning Holiday Live Stream

And we'll just input this.


Machine Learning Holiday Live Stream

How many steps do you usually train for an image classification task or an image reconstruction? Do you smaller or large batch sizes?


Machine Learning Holiday Live Stream

I don't I don't have a general recipe. You just go with whatever is is appropriate.


Machine Learning Holiday Live Stream

And it also depends really on the task. So for unsupervised learning, people do hundreds of epochs for ImageNet.


Machine Learning Holiday Live Stream

I guess you can get away with a bit less. I think like 50 epochs or something like this.


Machine Learning Holiday Live Stream

But also for. Yeah, it's it's I know C410 or so people use 50 to 250 epochs, which is like a lot.


Machine Learning Holiday Live Stream

But then heavy augmentation. It's really depends fully on the task.


Machine Learning Holiday Live Stream

Tell us about your startup. Sure. It's called Deep Judge. It's very cliche name.


Machine Learning Holiday Live Stream

And we do NLP for the legal domain. And if you're allowed to work in the European Union, no.


Machine Learning Holiday Live Stream

Well, if you if you are allowed to work in Switzerland and you want to work in Switzerland and your skills in engineering with an affinity for machine learning, you may you can happily apply.


Machine Learning Holiday Live Stream

That is so. Yes. Let's see if Google finds us. Yes, of course, Google finds us. OK. That is us.


Machine Learning Holiday Live Stream

Me wondering what I missed. You didn't miss anything. It's just me. It's just me dicking around in this.


Machine Learning Holiday Live Stream

OK, so it seems like we haven't trained anything. If or I'm just I'm just doing this completely wrong.


Machine Learning Holiday Live Stream

I might want to look up some more tutorials. I've just jumped into this completely.


Machine Learning Holiday Live Stream

But what you can what we can do, what we can do is you can see that the this X1 that we have right here.


Machine Learning Holiday Live Stream

So the first data point and this is usually a good thing to do if you wonder if your model works at all.


Machine Learning Holiday Live Stream

So we take the training data set right. We look at this data point and the context is whatever architecturally the school has a Catholic character.


Machine Learning Holiday Live Stream

The question is to whom did the Virgin Mary allegedly appear in da da da da da. And the answer is St. Bernadette.


Machine Learning Holiday Live Stream

Whatever. Now, what we can do is we can completely overfit on that one data point.


Machine Learning Holiday Live Stream

That that is that is what we're going to do. And for that, let's restart the runtime first and foremost.


Machine Learning Holiday Live Stream

Then we'll run this these things. Run this. And when it comes to the data set, we are going to.


Machine Learning Holiday Live Stream

So process data right here. And then when it comes to the data set, we're just going to select one data point.


Machine Learning Holiday Live Stream

Right. We'll call that overfit data set. I'm going to take the training data point.


Machine Learning Holiday Live Stream

I want to just get one single data point out of it. We won't even we won't even touch those right here.


Machine Learning Holiday Live Stream

Data set is not defined. Of course, why would it be? Because I need to run this. OK, so da da. OK.


Machine Learning Holiday Live Stream

So we'll get our overfit data set and then we'll just we'll just train on that overfit data set.


Machine Learning Holiday Live Stream

And how where is the number of epochs? Oh, num train epochs is three.


Machine Learning Holiday Live Stream

We'll just put this to. Um, num train epochs is, I don't know, like a hundred or so.


Machine Learning Holiday Live Stream

We'll just overfit on this one data point heavily.


Machine Learning Holiday Live Stream

And instead of the training data set being the training data set, we'll make train data set is overfit data set and eval data set.


Machine Learning Holiday Live Stream

Is also the overfit data set. Overfit data set is not defined. That set. I see.


Machine Learning Holiday Live Stream

So if this works out.


Machine Learning Holiday Live Stream

Image net is 100 to 600 epochs. 50 isn't really possible. OK. Yeah.


Machine Learning Holiday Live Stream

If your judge model doesn't work, claim it is a company secret.


Machine Learning Holiday Live Stream

Training, wait, already completed? This is not, this is, that is shady. OK, let's try.


Machine Learning Holiday Live Stream

See, OK, we're doing, I'm doing something terribly wrong.


Machine Learning Holiday Live Stream

Or we'll just put the number of epochs to 10,000.


Machine Learning Holiday Live Stream

Do we have output? Please. Do I need to rename? I should probably rename, right?


Machine Learning Holiday Live Stream

Ah, see, OK, this is the output. Now we're continuing.


Machine Learning Holiday Live Stream

Yeah, we're overfitting nicely.


Machine Learning Holiday Live Stream

We tested out the extreme overfitting approach leading to super convergence. You talked a while back. You mean the grokking?


Machine Learning Holiday Live Stream

Yeah, I think this will be a different, different domain right here, but I haven't, I haven't actually done or investigated grokking any time myself.


Machine Learning Holiday Live Stream

It seems super interesting, though. Yeah. Change the name of 1db thing. Yeah, I should.


Machine Learning Holiday Live Stream

I see no problem with training equals zero. Are you going to do a more classic paper review?


Machine Learning Holiday Live Stream

Yes. Yeah, this is, this is also planned. The classics.


Machine Learning Holiday Live Stream

Why do you look like Bezos?


Machine Learning Holiday Live Stream

It's a, whenever the hair gets long, longer, it just gets on my nerves.


Machine Learning Holiday Live Stream

Are you loading all the data? I am, but right now I'm just loading a single data point. So.


Machine Learning Holiday Live Stream

Remove the end of sentence token. True, that could be. But this isn't this part of the.


Machine Learning Holiday Live Stream

I've just seen this on like one example. I might be wrong.


Machine Learning Holiday Live Stream

A great masculine jaw.


Machine Learning Holiday Live Stream

What's the secret of the glasses? There's no, there's no secret. It's just that it was a, it was a maybe bad branding decision.


Machine Learning Holiday Live Stream

And now it's I'm stuck with them. So. So see, but now you can you can see that we are, we are converging in.


Machine Learning Holiday Live Stream

Yeah, look at that. I mean.


Machine Learning Holiday Live Stream

What is what is this? Do we? We don't have validation stuff. OK, but training total.


Machine Learning Holiday Live Stream

Training runtime.


Machine Learning Holiday Live Stream

This one right here. I mean, look at that. That is just very low.


Machine Learning Holiday Live Stream

Very, very small loss.


Machine Learning Holiday Live Stream

So we got a thousand, a thousand steps. We might as well sort of interrupt. What is this clear output? No, we might as well.


Machine Learning Holiday Live Stream

Interrupt a little bit. Hey, would you?


Machine Learning Holiday Live Stream

Would you mind stopping? Just interrupting a bit. Stop.


Machine Learning Holiday Live Stream

OK. And we'll just check the output. Maybe it's something better now.


Machine Learning Holiday Live Stream

Ahh, still no, still no.


Machine Learning Holiday Live Stream

Hmm.


Machine Learning Holiday Live Stream

Are we doing something wrong?


Machine Learning Holiday Live Stream

Unk, unk, unk, unk, unk, unk, unk.


Machine Learning Holiday Live Stream

Am I doing something wrong? I'm probably doing something wrong.


Machine Learning Holiday Live Stream

Are you hiring? Yes, we're hiring.


Machine Learning Holiday Live Stream

But you need to be able to work in Switzerland, which is, it sucks and it's not available to most of the world because of legal reasons.


Machine Learning Holiday Live Stream

But we have no choice really. So you need to be able to get a working permission.


Machine Learning Holiday Live Stream

And yes, this still sucks, even though we've completely overfit, which means that we're, I'm doing something quite wrong.


Machine Learning Holiday Live Stream

The question is what? What's so wrong?


Machine Learning Holiday Live Stream

I may not use this in the exact correct way right here.


Machine Learning Holiday Live Stream

So I could sample, I guess.


Machine Learning Holiday Live Stream

So do sample is.


Machine Learning Holiday Live Stream

But the argmax, it should be good if we overfit.


Machine Learning Holiday Live Stream

The problem is that we might just remove these end of sentence tokens, right?


Machine Learning Holiday Live Stream

Because like why?


Machine Learning Holiday Live Stream

Let's restart.


Machine Learning Holiday Live Stream

And this is the last try. If this doesn't work, we'll just take the loss.


Machine Learning Holiday Live Stream

So we'll remove the end of sentence tokens like this.


Machine Learning Holiday Live Stream

And let's hope the tokenizer puts them in itself.


Machine Learning Holiday Live Stream

Like that. Good.


Machine Learning Holiday Live Stream

And we'll do the 1000 epochs again.


Machine Learning Holiday Live Stream

And there we go. OK, last run for today.


Machine Learning Holiday Live Stream

Wrong input, yeah.


Machine Learning Holiday Live Stream

Yeah, we can inspect train data set entries after process data play. That's a good idea. We should do that as well.


Machine Learning Holiday Live Stream

I don't understand the meaning of the output. Can anyone explain? No, no one can explain.


Machine Learning Holiday Live Stream

That's it's it's it's not the desired output. We're trying to predict text.


Machine Learning Holiday Live Stream

And right now we're just it's garbage.


Machine Learning Holiday Live Stream

But what we're trying to do is overfit. Let's see. I could change this.


Machine Learning Holiday Live Stream

Also.


Machine Learning Holiday Live Stream

Overfitting on one data point.


Machine Learning Holiday Live Stream

We're just trying. We're just a bunch of researchers giving our best.


Machine Learning Holiday Live Stream

Your input is from the full data set.


Machine Learning Holiday Live Stream

But I mean, but I take I just select the first entry like I do select.


Machine Learning Holiday Live Stream

I selected this entry.


Machine Learning Holiday Live Stream

You're predicting the text from your full data set, but not from overfitting data set.


Machine Learning Holiday Live Stream

Yeah, but the overfitting data set is taking.


Machine Learning Holiday Live Stream

It's taking the.


Machine Learning Holiday Live Stream

It's taking the first data point, right? The overfit data set selects exactly the first data set.


Machine Learning Holiday Live Stream

The first data point. So it selects the zero of the train data set.


Machine Learning Holiday Live Stream

So this should be the same. I can I can try to just input.


Machine Learning Holiday Live Stream

I could try to input. Yeah, I see.


Machine Learning Holiday Live Stream

Is there a reason you're overfitting? Yes, it's because it's a good way to see if you don't have like it's a good way to catch bugs.


Machine Learning Holiday Live Stream

So what you want to do is you want to take a single data point and then overfit to it.


Machine Learning Holiday Live Stream

And if you can do that, it means that your model is OK or well, it means your model can at least process data.


Machine Learning Holiday Live Stream

That's essentially what it means. But for us right now. So let's let's call that X2.


Machine Learning Holiday Live Stream

Overfit data set zero. Right. So X2 is now.


Machine Learning Holiday Live Stream

It's a whole lot lot of zeros.


Machine Learning Holiday Live Stream

Zero zero zero zero. No, it's actually it's actually OK. It's more than zeros.


Machine Learning Holiday Live Stream

We can decode it.


Machine Learning Holiday Live Stream

Tokenizer decode X2 input IDs.


Machine Learning Holiday Live Stream

Let's see if that.


Machine Learning Holiday Live Stream

Yeah, architecturally, the school has a Catholic character.


Machine Learning Holiday Live Stream

Right. This is fine. This is and the answer, St. Bernadette.


Machine Learning Holiday Live Stream

We might. Oh, this is here. Good. So this is done automatically. We don't have to do this.


Machine Learning Holiday Live Stream

And then the pad. Maybe we'll leave away the pad because right now we're padding. Right.


Machine Learning Holiday Live Stream

And we technically don't need to pad or we also grab the mask.


Machine Learning Holiday Live Stream

Yep. Maybe the masking is is the problem and label IDs.


Machine Learning Holiday Live Stream

Yeah, this is fine. And if we input now this into.


Machine Learning Holiday Live Stream

Let's input this.


Machine Learning Holiday Live Stream

So we'll take X2 right here.


Machine Learning Holiday Live Stream

And the decoder IDs will input nothing.


Machine Learning Holiday Live Stream

This has no attribute to.


Machine Learning Holiday Live Stream

What was X2 again?


Machine Learning Holiday Live Stream

This is.


Machine Learning Holiday Live Stream

Just a list. We need to.


Machine Learning Holiday Live Stream

OK.


Machine Learning Holiday Live Stream

Can we do this?


Machine Learning Holiday Live Stream

I'm just I'm just hoping here. Nope.


Machine Learning Holiday Live Stream

No attribute shape. Yeah, we should.


Machine Learning Holiday Live Stream

We should put it into a tensor.


Machine Learning Holiday Live Stream

How to make a tensor from this, do I need to import Pytorch?


Machine Learning Holiday Live Stream

Yeah, probably.


Machine Learning Holiday Live Stream

Yep.


Machine Learning Holiday Live Stream

So.


Machine Learning Holiday Live Stream

How?


Machine Learning Holiday Live Stream

OK, does anyone know how to make an int tensor?


Machine Learning Holiday Live Stream

Like this?


Machine Learning Holiday Live Stream

Long.


Machine Learning Holiday Live Stream

OK. Yeah, so the output seems to be correct, but we can't overfit.


Machine Learning Holiday Live Stream

So I have. OK, let's say we have a last trick up our sleeves and that is we remove the padding.


Machine Learning Holiday Live Stream

So the model will just have to.


Machine Learning Holiday Live Stream

Take whatever we have it. So the padding.


Machine Learning Holiday Live Stream

We don't pad. We just truncate.


Machine Learning Holiday Live Stream

Dun dun dun.


Machine Learning Holiday Live Stream

Alright.


Machine Learning Holiday Live Stream

OK.


Machine Learning Holiday Live Stream

How's it coming in?


Machine Learning Holiday Live Stream

Excellent. Loss is decreasing. That is a good sign.


Machine Learning Holiday Live Stream

And it's going to take another.


Machine Learning Holiday Live Stream

90 seconds.


Machine Learning Holiday Live Stream

Does anyone know how Yannick deletes a chunk of text with one button?


Machine Learning Holiday Live Stream

It's probably you mean like the entire line like this?


Machine Learning Holiday Live Stream

It's a. Oh, no. It's a Vim command. It's capital S.


Machine Learning Holiday Live Stream

Tim is calling me. I should.


Machine Learning Holiday Live Stream

I should leave soon.


Machine Learning Holiday Live Stream

We're making we're making the the final argument, the final things for the next machine learning street talk.


Machine Learning Holiday Live Stream

Done soon.


Machine Learning Holiday Live Stream

You mentioned me as an inspiration for your master's application to ETH. Nice. But I don't I don't think human resources is too happy with me.


Machine Learning Holiday Live Stream

Given that they've received calls in the past about my behavior.


Machine Learning Holiday Live Stream

So maybe maybe don't don't mention that too much.


Machine Learning Holiday Live Stream

OK. OK. I mean, you know, no to sample through.


Machine Learning Holiday Live Stream

Some things, some things wrong.


Machine Learning Holiday Live Stream

Like something's clearly wrong. I don't know what, but I'm doing something wrong with the with the inputs right here.


Machine Learning Holiday Live Stream

Because it's. Yeah.


Machine Learning Holiday Live Stream

So I'm not sure what exactly what exactly they are.


Machine Learning Holiday Live Stream

What exactly they need. But here fine tuning T5.


Machine Learning Holiday Live Stream

A mixture of supervised. So here in supervised training.


Machine Learning Holiday Live Stream

Right. That's what they do. They have input IDs. They have labels.


Machine Learning Holiday Live Stream

I have labels and not label IDs. The function automatically creates the correct decoder input IDs.


Machine Learning Holiday Live Stream

But you see, you see this.


Machine Learning Holiday Live Stream

It seems weird because I'm not putting it and they.


Machine Learning Holiday Live Stream

Yes, something is something is definitely wrong. So labels. Let's call it labels. Right.


Machine Learning Holiday Live Stream

Oh, no. I was calling it labels all along.


Machine Learning Holiday Live Stream

Label IDs, labels, label IDs, labels.


Machine Learning Holiday Live Stream

Labels. Labels.


Machine Learning Holiday Live Stream

Well.


Machine Learning Holiday Live Stream

Decoder IDs.


Machine Learning Holiday Live Stream

It's worth a try.


Machine Learning Holiday Live Stream

I mean.


Machine Learning Holiday Live Stream

We might just try it until here.


Machine Learning Holiday Live Stream

Might be the parameter where you can auto add the end token. Yes. Yeah. True.


Machine Learning Holiday Live Stream

Yeah. Here you generate.


Machine Learning Holiday Live Stream

So.


Machine Learning Holiday Live Stream

Let's try.


Machine Learning Holiday Live Stream

And instead of.


Machine Learning Holiday Live Stream

Yeah. Here I have labels too.


Machine Learning Holiday Live Stream

Oh, that's what I just changed a second ago. I see.


Machine Learning Holiday Live Stream

Sorry for being dumb.


Machine Learning Holiday Live Stream

No, but here labels. Right.


Machine Learning Holiday Live Stream

It's already running. OK. OK.


Machine Learning Holiday Live Stream

What's your opinion on Jax?


Machine Learning Holiday Live Stream

Well, given that it has a built in.


Machine Learning Holiday Live Stream

If I understand, I have not looked into Jax, but if I understand correctly as a built in XLA compiler, which is really good if you want to do super custom stuff in PyTorch.


Machine Learning Holiday Live Stream

Sorry on TPUs, which I don't.


Machine Learning Holiday Live Stream

But I as far as I've seen so far, I do like sort of the.


Machine Learning Holiday Live Stream

It's much more clean.


Machine Learning Holiday Live Stream

So, you know, maybe if they put a bit more attention also on common usability, not just with TPUs.


Machine Learning Holiday Live Stream

I guess it might be a cool language.


Machine Learning Holiday Live Stream

Any good suggestion for a master's in NLP?


Machine Learning Holiday Live Stream

I have no idea. Sorry.


Machine Learning Holiday Live Stream

I've chosen my university because it was the only one in the country that was like it was that was that was a had the things that I wanted to or that I was interested in vaguely.


Machine Learning Holiday Live Stream

See, got an unexpected labels. Why? OK. How? No. Decoder IDs.


Machine Learning Holiday Live Stream

OK.


Machine Learning Holiday Live Stream

All right. Maybe now.


Machine Learning Holiday Live Stream

Labels. Yeah, no. What was it before? Decoder IDs, right?


Machine Learning Holiday Live Stream

Decoder input IDs.


Machine Learning Holiday Live Stream

That's better.


Machine Learning Holiday Live Stream

That is better.


Machine Learning Holiday Live Stream

No, you don't think so. Catholic Catholic.


Machine Learning Holiday Live Stream

And what did we want to whom did the Virgin Mary allegedly appear? It's better than before.


Machine Learning Holiday Live Stream

No. A two. Yeah.


Machine Learning Holiday Live Stream

We say do sample. True.


Machine Learning Holiday Live Stream

Now it just evolves again.


Machine Learning Holiday Live Stream

I might be imagining that this is better than before, but it has the end of in the main main of the main building.


Machine Learning Holiday Live Stream

I might be imagining that this is better. Maybe also because we trained it for not as long.


Machine Learning Holiday Live Stream

Oh, look at that. Look at that.


Machine Learning Holiday Live Stream

Look at that. Yeah.


Machine Learning Holiday Live Stream

OK. Decoder IDs, whatever that means.


Machine Learning Holiday Live Stream

But if I don't do decoder input IDs, but decoder IDs, I get the correct question.


Machine Learning Holiday Live Stream

So we can, in fact, overfit on a single data point.


Machine Learning Holiday Live Stream

See, this is the output.


Machine Learning Holiday Live Stream

It is the output. Yes. It's padded to the left for whatever reason.


Machine Learning Holiday Live Stream

But it is the output. And what if I put like a cue here?


Machine Learning Holiday Live Stream

Yeah. Excellent.


Machine Learning Holiday Live Stream

So it turns out that the most challenging part about doing machine learning is using the correct keyword arguments in the hugging face library.


Machine Learning Holiday Live Stream

Coincidentally, this would be something where like Copilot would be a killer at.


Machine Learning Holiday Live Stream

I've been I've been using this for a while now. And it's this kind of stuff. It's really good.


Machine Learning Holiday Live Stream

So, yes. So that was essentially this was where I wanted to get to today.


Machine Learning Holiday Live Stream

We have a model that can take in context answer pairs and give us questions.


Machine Learning Holiday Live Stream

The next step is to take this and make this train it on SQuAD, then generate a data set of more questions and answers.


Machine Learning Holiday Live Stream

And then we can use that to train another question answering model.


Machine Learning Holiday Live Stream

And at the end of it, that one should be better than our original one that we just trained on the data set alone,


Machine Learning Holiday Live Stream

which is a little bit magic because we don't introduce new data. Right. We don't.


Machine Learning Holiday Live Stream

But what we do, what we can do is, you know, we always need we need new question answer pairs.


Machine Learning Holiday Live Stream

So the input to this model is going to be a context and an answer.


Machine Learning Holiday Live Stream

And we let it produce a question. Now, the context we just take from the same data set.


Machine Learning Holiday Live Stream

What is the answer? That is that is the question.


Machine Learning Holiday Live Stream

Well, ironically, that is the question. So what we can do is we can use something like a maybe like a sequence tagger,


Machine Learning Holiday Live Stream

maybe from from space here. So that recognizes nouns or noun phrases in the context.


Machine Learning Holiday Live Stream

And then we'll just take these noun phrases as possible answers.


Machine Learning Holiday Live Stream

So what these papers this is not my idea, right? These are papers that that are in question answering.


Machine Learning Holiday Live Stream

What they do is they would find noun phrases. They would let this model that they've trained to produce questions,


Machine Learning Holiday Live Stream

produce a question given the context and this possible answer.


Machine Learning Holiday Live Stream

Then they use the original question answering model to check whether it actually gets it right.


Machine Learning Holiday Live Stream

And that's how they generate new data. That's essentially what we're going to do.


Machine Learning Holiday Live Stream

But yeah, this was it for for today's live stream. It took it took a bit longer, but it's good to refresh myself as well,


Machine Learning Holiday Live Stream

because as I said, I've been doing mostly plumbing coding and not machine learning coding.


Machine Learning Holiday Live Stream

So this is fun. And I hope I hope at least someone someone learned about the trick of overfitting on a single data point.


Machine Learning Holiday Live Stream

Because I remember when I first when first someone told this to me, I was like, oh, wow, this is this is really helpful.


Machine Learning Holiday Live Stream

And I'm sure most people know it by now, but it's still super helpful.


Machine Learning Holiday Live Stream

All right. So in this case, thank you very much to everyone who's who's been here.


Machine Learning Holiday Live Stream

I need to hop on a call with with Tim and Keith for the street talk. And yeah, I'll see you around.


Machine Learning Holiday Live Stream


OpenAI DALL·E: Creating Images from Text (Blog Post Explained)

A sphere made of Swiss cheese, a sphere with a texture of Swiss cheese.


And there you have it.


Beautiful, very appetizing Swiss cheese balls.


My Swiss heart had just just skipped a beat out of this monstrosity.


What's even cooler than a sphere made of Swiss cheese is a Taurus made of denim.


These images are so cool, a Taurus made of denim.


And the point here is that these images aren't photoshopped or sort of human created, they


are AI generated.


And they are generated by this new model that open AI released a blog post about it's called


Dali and it can what it can do is it can take a piece of text such as the one on top here.


The fact that I can select is simply the fact that they don't give you access to the model,


they just give you access of a bunch of things that they've tried, but the model can take


any piece of text and it can output a picture that matches that text.


So here you got a Taurus made of toothpaste, and the quality of these images is super astounding.


And what's even more astounding is sort of the range of capabilities that this model


has.


So the model can do various things such as so in here the input is an illustration of


a baby daikon radish in a tutu walking a dog.


And you see an illustration of a baby daikon radish in a tutu walking a dog.


The outputs are just adorable.


These are generated by the AI.


The same for an armchair in the shape of an avocado, a storefront that has the word open


AI written on it.


I've tried reverse image searching some of these images and I could not find them on


the internet so it's definitely not just a model sort of outputting an image it found


somewhere.


These are actually generated images.


And the astounding thing is that is the same model that outputs all of these different


images.


It's not one model here trained on illustrations and one model trained on chairs.


It's a single model that can take in a piece of text and optionally part of an image or


none of an image and it will output the image either it continues the image you already


give part of or it just generates the image by itself.


So the model is called Dali.


And this is just a blog post for now by open AI.


They say they'll follow this up with a paper.


And if the paper you know brings substantially new things I think I'll make a video on it.


But today we're just going to look at what this model can do, how it works, how it probably


works and we can take some guesses of what we can read in the paper once it's out.


In fact, open AI has brought out two new models along with this Dali model.


They've also released a blog post and a paper about a model called clip, which is more of


a sort of a classifier, not exactly a classifier, it's sort of a it connects text and images


in a different way.


It's not a generative model.


And we're going to look at that in a in a different video.


But you can see the clear trend right here is that open AI is looking into connecting


text and images.


So they say, Dali, which is an, this is a an, I think, a an homage to Salvador Dali


and mixed with the character Wally.


So they say it's a 12 billion parameter version of GPT-3.


So you know, it's it's more like, it's more like not GPT-3, that was more than 10 times


larger.


But it's a 12 billion parameter version of GPT-3, trained to generate images from text


descriptions using a data set of text image pairs.


We found that it has diverse set of capabilities, including creating anthropomorphized versions


of animals and objects, combining unrelated concepts in plausible ways, rendering text


and applying transformations to existing images.


So a lot of the things they don't tell us here, especially the data set, like how did


they get the data set?


Nobody knows.


They don't say this, they simply say it's a data set of text image pairs.


And they sort of allude to the fact that they have large pieces of data, especially in the


clip, then they allude to the fact that you can just find data that connects text and


images on the internet.


And it's true, if you if you search, if you scrape the correct websites, and do it in


sort of a smart fashion, you can find a lot of data where there is an image and there's


a piece of text describing that image.


And we have to assume that they sort of scrape the internet for something like this.


I don't think they have a lot of human explicitly human labeled data for this type of thing.


So we'll just assume that they have like a huge data set.


And of course, they train a huge model on it, a 12 billion parameter version of GPT-3.


GPT-3 is the famous model, the famous text generation model by open AI.


And you can sort of see the same things right here.


So GPT-3, my hypothesis was that it sort of smartly mixes the training data rather than


remember the training data, it sort of remembers it and then smartly interpolates between it.


And I think you can sort of see the same kind of things right here in that these are all


definitely pictures that you could imagine in the real world.


But they have, you know, they have, for example, their change to open AI in here, there are


surely chairs that sort of look like this.


So it just kind of mixes a chair with an avocado in a plausible way.


I'm not saying this to denigrate the model.


I'm saying that, I mean, this is seriously cool, the fact that it can do that.


So they say, like GPT-3, Dully is a transformer language model.


This is very, very interesting, the fact that it's a transformer language model.


It receives both the text and the image as a single stream of data containing up to 1000


and 1280 tokens, and it's trained using maximum likelihood to generate all of the tokens one


after another.


Okay, this training procedure allows Dully not only to generate images from scratch,


but also to regenerate any rectangular region of an existing image that extends to the bottom


right corner in a way that is consistent with the text prompt.


And they say a little bit more here on the right.


And they also say a little bit more down on the bottom.


So I'm going to try to take a stab of explaining how this model works with the full knowledge


that I might be wrong once the paper comes out.


And for that, we have to go back a little bit and look at the models it draws from,


namely the VQVAE.


So the vector quantized VAE literature.


So VQVAE will consider this to be sort of the inspiration of or one of the necessary


ingredients of this model.


So if we combine VQVAE with something like GPT-3, we get Dully.


That's my that's my hypothesis for today.


Why combining these two models?


So GPT-3 is extremely good at modeling language, right?


So if I have a piece of text, let's go down here for a minute.


And let's say I have a cat sat on the mat.


A transformer will be very good at understanding the sentence and being able to complete it.


So if I cross out this and ask a transformer to continue the sentence, it will be able


to continue the sentence just fine if it is if it is trained well.


And that's exactly how GPT-3 works.


Now imagine that I don't have a piece of text, but I have some sort of a description of an


image, right?


Let's say I have I have a box.


Here is a box and the box which is going to be a VQVAE can take in a description of an


image in words, but not exactly words that human on humans understand.


But let's say there is an image language sort of like a programming language.


Okay.


And you input symbols into the image.


Let's say it's a bit like Egyptian hieroglyphs maybe.


So here is the here is the this, this hieroglyph thing.


And then there is the sun, the sun thing.


And then there is the tree, the word for tree like the hieroglyph for tree.


And I input that here.


And the output will be an image where I don't know what there the sun is shining.


Yes, I draw some like a child.


It has a little smile.


Okay, deal with it.


And there is a tree, maybe not exactly the tree from the hieroglyphs, but like some sort


of some sort of tree that fits.


And then there is some human in the scene, maybe the human sits here, the human sits


at the tree, you know, relaxing, chilling.


Okay, so this, now the image on the right is consistent of pixels, right?


And modeling pixels with a transformer is very, very hard.


Because in the case of our model right here, it's something like 256 by 256 pixels.


That would mean the transformer would have to generate 256 times 256, which is like two


to the two to the 16.


This is just too much for a transformer to model the pixels individually.


So there are multiple ways around this, for example, modeling little regions right here,


which are not really satisfactory.


So what this model does is it sort of it doesn't try to model the picture as such it tries


to predict to predict these hieroglyphs right here, it tries to predict sort of a language


that this box can understand and produce a picture from, okay.


So its task is going to be given some sort of a given some sort of a text prefix.


So a human in a sunny field, sunny day or on a sunny day, chilling under a tree.


So this piece of text followed.


So the model is trained to take this piece of text and output this sequence of hieroglyphs.


Okay, so this sequence of hieroglyphs outputting from this piece of text, and that's something


a transformer can do if you have a vocabulary right here.


So if you have a fixed list of hieroglyphs that you could use, right, so in there, there


is the the human is in there.


That's a worse Egyptian.


And then the pyramid is in here as well, some that you need some that you don't need.


So if there is a vocabulary, the transformer is going to be pretty, pretty good at generating


this thing.


So you need you need two parts.


The first part right here is a transformer, language model, a GPT three thing that can


input a sequence of text, and it can output a sequence of text, which is just in a different


vocabulary, namely this picture vocabulary.


And then in the step two, you need a box that takes in this picture vocabulary and actually


produces an images an image right here.


So as I already said, this part is taken over by GPT GPT three, like the custom GPT model


they built for this, and this part is taken over by something like a VQ VAE, the generator


part of it.


So what is a VQ VAE?


A VQ VAE is and you will be able to see that.


So the box that we're going to need is this box right here from from here, up to where


the images and this thing right here is going to be that vocabulary.


So what does a VQ VAE do?


It takes the image here on the left, you can see that here's the encoder, it takes the


image, it encodes it into a latent space.


Now what a what a VAE would do, or what an auto encoder would do is it would encode the


image into a latent space.


And then it would decode it again into and try to reproduce the same image.


And then you assume that whatever is in the middle right here is a sensible representation,


a latent representation of that image, right?


If you can train this model, you're going to get some sort of a representation in the


middle that describes the image, otherwise, you couldn't reproduce the image.


And there have been many models built on this concept.


Now this model right here, it turns out that the classic auto encoder doesn't work too


well.


But this model works quite formidably.


So what you're going to have is you're going to have this vocabulary right here.


It's also called a codebook, let's call it a codebook.


So the codebook is also the vocabulary.


So what you're saying is that you can't just output any or any latent encoding.


So the encoder outputs a continuous vector.


But what you're saying is it has to be one of those.


Like there are a number of vectors that you have at your disposal, Mr. or Miss encoder


or Mrs. encoder.


There is a number of vectors that you have at your disposal.


You can only choose those, you can't choose any vector that you want, right.


So in your latent space, you can't just choose any latent space, there's this, there's this,


there's this, there's this, there's this, you have to choose one of them.


And if you choose something in between, which you'll inevitably will, because this all of


our neural networks output continuous values, we're just going to clamp you, we're just


going to find the nearest one in our codebook, and we'll just say, well, we, we just make


it such that you as if you had output that one.


So the encoder can only hit one of those codebook vectors, and then you feed these codebook


vectors to the decoder.


And the decoder just decodes from these codebook vectors.


And that turns out to be much, much, much better than simply doing the auto encoder


thing continuously.


So imagine that this codebook vocabulary is sort of like a vocabulary of image descriptions.


What you do with an image, you take this dog image, I'm gonna have to draw this myself,


you take the image here of the dog, I can't draw dogs, I'm very good at cats, though.


This is a cat, and you don't just encode this into one of these words you, what you do is


you split the image up into a grid, it's not as fine as pixels, it's fairly, it's okay


large.


So in their experiments, they're going to use something like 32 by 32 grids, which is


also what Dolly uses.


Every image is described by 1024 tokens, that's 32 by 32 tokens.


And then you're going to encode, you're going to make an encoder such that when this grid


is through the encoder, this thing here corresponds to one of the code vectors.


And this thing here corresponds to another one.


So you have your big vocabulary right here.


Right?


And this is the red vector, this is the blue vector, this is the green vector, and you're


going to just describe the image regions with these codebook vectors, like such.


Okay, now the fact that is you have you have a lot of these vectors, right?


You have in fact, you have 8092 vectors in Dolly.


And the image only consists of 1024 tokens.


So you know, it's conceivable, like, it's not like here where you have to reuse the


same token over and over again.


But one of these tokens could, for example, be sky.


So maybe this is the thing that sort of describes sky.


So what you you'll have is like this thing and this thing and this thing and this thing


should be approximately sky, right?


And then maybe the red one is, is, I don't know, animal, and the blue one is vegetation.


And the green one is some something else.


So you can see, if you feed this to a model, that has to make a picture from it, it can


just look at this.


And it's sort of like a description, a low resolution description of an image is not


exactly a downsampled image.


It's a it's a description because these things here contain a lot of information by themselves.


Okay.


It's just that you can't choose any vector in latent space, you have to choose one of


those vectors in the codebook.


So that's a vector quantized VAE.


And they train everything at the same time.