Gaussians over these things, then we should get an idea of where most of the attention went and


∞-former: Infinite Memory Transformer (aka Infty-Former / Infinity-Former, Research Paper Explained)

where no attention went. And the idea of sticky memories is simply, let's oversample the regions


where a lot of attention went. So maybe a lot of attention went to this bump right here. So we


oversample that, and maybe not much attention went to this region right here. So we don't sample


anything like this. Then once we have sampled, we spread these things out, I guess, equally,


we could, and then we interpolate again. And that's how we keep the more important things in


memory more accurately. Now again, this is all heuristics. And this is a bit what my criticism


here is as well. All of these things, you know, in an LSTM, it's at least learned like how to


compress the past, and how to read it, how to use the past, which memories to keep and so on. All


of all of this is learned, right, the LSTM, all the gates are learned, and so on the waiting


functions. Now, that's also the culprit in an LSTM, because you have to back propagate through


time. And that's just not possible for very long sequences. So that's a bit of the LSTM's downfall


as well. Whereas here, we don't have to back prop through time, because everything is a heuristic.


However, everything being a heuristic, it's, you know, like, how do we know? Okay, maybe it works,


but, you know, I'd rather, I'd rather not use just heuristics for doing that kind of stuff.


Yeah. But I guess there's room for improvement. So here they detail that, yeah, they smooth the,


they smooth the signal with a CNN before they do the multivariate ridge regression and so on. There


is a regularization where they regularize the variance of the Gaussian that they predict.


Yeah, these are details. So the ultimate loss has the training loss plus the KL divergence. Maybe


they did that after they just saw the model simply wants to attend to everything all the time. I don't


know. But then they evaluate the model on various tasks, such as this sorting task. And I have to


say, they construct the tasks fairly quickly. And they also evaluate the training loss,


cleverly by making sure the model can't like use simple strategies to solve it. And what they see


is that things like the transformer XL, which tries to have some sort of a long term memory,


but not doesn't do it really, like doesn't. I've made a paper on transformer XL, sorry, a video. So


if you're interested in that, you can read it. And also this, this compressive transformer seems to be


a little bit what the inf deformer is, but without going via this continuous signal, though the


compressive transformer seems to be a transformer that always tries to sort of compress the past


into fixed size memory, if I understand it correctly. And generally, they find that their


model is relatively on par with the compressive transformer outperforming it a little bit. Now,


this being machine learning and so on, I would not, I would not be confident that there is a


difference between the two model or which one is actually better, just from these results. In their


results, they are better. And when they add the sticky memories, they are even better, which I


guess makes sense. But again, take that with a grain of salt, they do analyses on what which


parts of the long term memory this continuous attention goes to. And in general, this seems


pretty reasonable. If you look at kind of, you know, these, where in these long texts where the


attention goes to, like apparently here, the ground truth is you too, as I guess the answer of


a question or oh no here, I guess this is masked out, maybe. And the attention. I'm not exactly


sure where it's trying to predict you to maybe it's masked language modeling or some sort of


question answering. However, it seems to be reasonable. There is a helicopter. It seems to


be reasonable. At least in this one example they show. So they do. Sorry, not mask language modeling


actual language modeling against something like GPT-2. And they outperform that. And they do some


more analysis. So again, I don't want to go too deep into the experimental results right here,


because again, with lots of engineering choices, it seems to be, it seems to be, you know, like


it's tricky to make sense of small differences between models. What I would go for is the general


trends and the general trends are are okay. You know, I don't know if the codes out I haven't seen


any code. If it is out, give it a try, I guess. Otherwise, you know, wait for about 30 minutes


[YTalks] Siraj Raval - Stories about YouTube, Plagiarism, and the Dangers of Fame (Interview)

The following is a conversation with Siraj Raval. Siraj has one of the largest channels in the machine learning YouTube space.


Over 700,000 people are subscribed to him as of this date.


Siraj pumped out lots and lots of videos on topics such as coding tutorials, explaining beginners concepts in machine learning,


and in other topics like blockchain or other computer science things.


Now his rise came to an abrupt stop when a series of scandals hit him at the end of 2019.


And there were a lot of articles written back then, Twitter posts made, and even Siraj himself made an apology video.


But I was wondering how did he feel like during all of this? What did he think back then? How did he come to this?


How did he feel during the highs and the lows of his career? And how does he look back on things now?


I was struck by how straightforward Siraj was in this conversation. I was sure there was going to be wisdom in there for the rest of us,


be that YouTubers or machine learners, and I was not disappointed.


He was definitely honest, looking back with a different view. And we touched on many things in this conversation.


I hope you enjoy it. I hope you find something in there that helps you. And yeah, let us know what you think.


Well, hello, everyone. Today is a special day.


In many ways, Siraj, who is my guest today, is one of the pioneers of the field of ML YouTube.


Now, I'm pretty sure pretty much every single person in the field has heard of Siraj, has seen him, watched one of his videos or something like this.


And if I can maybe frame it a little bit, is that you were one of the first machine learning YouTubers.


You became really popular quickly. Things went uphill, more views and so on.


And then I think it's fair to say it kind of all came crashing down in like a very short period of time.


And then it just sort of crumbled. I can't really frame it any differently.


There seemed to be like things one on top of another that just all came in like a month or so, the same month.


It seemed crazy this time at the end of 2019. So, yeah, I'm happy to host Siraj today.


Thanks so much for being here and talking and you agreed to talk a little bit about your side of things of what happened and what you're doing now.


So, yeah, welcome. Thanks. It's great to be here. I love your videos.


They're definitely got a personality and character to them that I definitely admire and I'd like to see more of.


Thank you. And yeah, so I think you, well, since you're the OG YouTuber of this, you know, that I guess character is a little bit of what it takes.


I want to go back a little bit to the beginning, though. If I recall correctly, you started studying economics. Is that correct?


Correct. At Columbia, that was my freshman year. I was an economics major.


Yeah. And for some reason you switched over to computer science because what took you there?


Well, I took a semester to travel around Europe using Couchsurfing. I was Couchsurfing for three and a half months.


And the first person that I couchsurfed with in London, his name was Alex McCall.


He showed me his terminal window. He had a hackintosh that he made and he really inspired me to get into computer science.


It turned out several years later that Alex wrote the book, the O'Reilly book on JavaScript.


And he has this really cool startup called Clearbit that he already sold by now.


But I got to meet him before all that happened. And once I saw Alex terminal and all the cool things he was doing, I knew that once I got back to Columbia,


I needed to switch over to computer science because that was how you really made an impact in the world.


Yeah. So I guess you saw pretty early that the impact was to be made.


I think a lot of people go into economics and they think like, maybe think a little bit of money.


They go into economics because it's kind of close to it.


But I guess computer science, especially nowadays, is really the impactful field or one of the impactful fields.


Little known fact, I also didn't. I started out in medicine and then switched over to computer science.


So much of the same journey there. And then did you finish computer science?


No, I dropped out my senior year of all times to drop out.


Wow. Yeah. And that was because of YouTube or?


No, no, no. So I dropped out because I had a robotic startup at the time.


We were making a six degree of freedom robot that would pick things up off the floor for older people with something called ALS because they can't bend over.


And we built a prototype, raised money, but it turns out like nobody would buy it.


And also there were some software problems at the time. This was like 2012.


So, yeah, I just moved to San Francisco from there, from New York.


And then that's when I really started to feel like I was around my people.


Yeah. Yeah. You're you're American originally, but from smaller town or big city or?


I'm from Houston, Texas. So I was born here. My parents are from India.


Definitely have a deep connection with India. I still dream about India.


Cool. And then you were you were in San Francisco. And how did you get into YouTube?


So I worked at a several contract jobs in San Francisco for companies like CBS Interactive doing mobile development.


I worked at Meetup for a year just as a general software engineer.


I was I started off as an intern and then eventually the last job I had W2 job was at Twilio, the API company.


And I worked there as a developer educator for about eight months. And then I was fired because I think it was just a performance thing.


That's what they said. So I don't know. But I remember wanting I learned a lot at Twilio about developer education and how innovative it could be.


To give you an example, we were learning about different ways of getting developers to use the Twilio API.


And, you know, as I was writing documentation across nine different programming languages like Ruby and PHP and Python, one thing that I was told by my mentor was that we don't want to use too many exclamation points inside of our documentation.


Because if you have more than three, what developers do is that they subconsciously think of not equals from code.


And that gives them a negative impression of the text. And I was like, that level of detail, I never thought about that.


But it really is an art. And so I started wanting to make videos on the side. And actually, my first three YouTube videos I made while I was at Twilio at the conference room at midnight when nobody was there.


And I showed it to my colleagues there and they were like, my boss was like, you know, that's great. That's cool.


We don't think developers are going to use videos as a learning tool. They want something static like documentation.


And so that's when I thought, well, maybe there's something here. And so once I got fired, I got a severance and I had enough to live in San Francisco for about six to eight months.


And that really gave me the impetus. I remember I had all my stuff in a box that they gave to me from my desk.


And I literally the day I was let go, I walked across the street to a hair salon and then I got my hair dyed. And I was like, all right, I'm all in on this YouTube thing now.


Like, I have to figure out how to make this work. Did you did you just the hair? Did you consciously do that? Did you think I need some sort of a thing?


Yeah, I mean, I was always inspired by a guy named Bill Nye, the science guy and how he was a very unique character for general science.


And I thought, what is my thing? I didn't know what exactly I wanted, but I remember a roommate of mine at the time who was a matchmaker.


She was like, you know, you'd look really cool with like a silver streak in your hair. I just tried it out.


I mean, you chose better than me the sunglasses. Now I have to code with sunglasses, which is annoying.


Do you get you get you get recognized with the sunglasses in person?


I get I get recognized with and without. I think the hairline is gives gives it away.


Yeah. Yeah. That's how that's how branding works, I guess.


So but yeah, so then you just you started creating videos. Was it always machine learning or did you also get into that somehow?


No. So it started out my first few videos were all on Bitcoin. In fact, my first video was called What is Bitcoin?


And that's really I think a Bitcoin is the soul of the hacker community.


Everything comes from Bitcoin and emerges outwards from there.


If I I'm not religious, but like the closest thing to a religion would be Bitcoin.


But I started making machine learning videos just because it seemed really interesting and I was really interested.


AlphaGo really. Was the catalyst for me, like, oh, there's something here.


Let me let me start making videos on this with no credentials, no PhD or anything like that.


Yeah. Also. Also, I felt like this is kind of weird to say out loud, but like I'd spent six months in India traveling across the entire subcontinent before I started working at Twilio.


And one thing that I saw was like, you know, I was living in such a box my whole life in the United States and India is such a beautiful country.


However, there's a lot of issues there. It is a developing country, an ascending country, I like to say.


But, you know, we can't just solve all of these problems in our lifetime. And some of them are just they're going to take many generations to solve.


Perhaps if we created some sort of super intelligence, digital organism, God, it could solve everything for us.


And the thing that I personally could do was use my specific knowledge to help make that happen in the form of funny, interesting videos that would raise awareness around these technologies to as many people as possible.


And that would somehow increase the amount of research happening in the field. And all of this together would accelerate development of a super intelligence.


Yeah. I mean, that's I have one socialist, like borderline communist friend.


And whenever I make fun of communism has never worked, he always says like, but we haven't tried with an A.I. super mind planner.


Right. And then I'm like, yeah, OK, that's got it's got a point. But, yeah. So when did you when did you.


So you had this plan of doing videos. When did you really see that this could be something like was there a moment where you saw like, wait, you know, views go up.


And was there like a particular moment or did it come slowly or when did you really feel like, yeah, I could make this work?


Well, I think it was three months into making videos once a week, because back then I could only do once a week.


It took about 40 to 50 hours for a single video. Eventually, I got up to three a week at my peak.


But after three months of one video a week, I got someone emailed me from this company called Big ML, which was a machine learning platform.


It was my first person who ever reached out to me and they wanted to pay me for a series of videos.


And I was elated because ad revenue was like, you know, nothing really.


I did have Patreon that definitely helped for sure. But that that was my first I think they paid me two K USD for six videos, which was huge.


And that was really like, oh, this is something. And then, of course, Udacity reached out to me and that was the biggest catalyst to help make their deep learning course, Nanodegree.


Yeah. So, yeah, Udacity. But that that also fell through, if I recall correctly.


And and this is so maybe for people who don't know and you have made you have made an extensive like apology videos about this.


But some of your videos or, you know, to that degree were plagiarized, not exactly the videos, but you would sort of write or show some code.


And then you would say like, either like, oh, look at this code or watch me build a trading bot or something like this. And and, you know, just be very vague about the origins of the code.


And then you would you put attribution, maybe really small at the bottom of the code.


But essentially, it'd be other people's code that you you presented. Is that about a fair framing of of things?


So a lot of times you took other people's codes, didn't fork it on GitHub, but just kind of downloaded it, reuploaded it, and then changed the like the readme or maybe some wrapper and things.


So when, when was that? Was this always your your mode of operating? Or did you use like, did you at some point start?


Did it increase? Because that's what I'm, I'm wondering, like, I, right, you started out saying, you know, I could do I could do raise awareness and so on. And you ended by or ended you at some point, you found yourself in a mode where you would, a new video would just be like, I take someone else's code, I make a video claiming essentially inferring that I I made it right.


How how did you get from A to B? So if it was a process, it didn't happen all at once. I mean, if you look at my first few videos, they were like, I really did write the code for the first few videos.


They were like 10 to 20 lines using the skills that I learned at Twilio of like, making something really basic, a skeleton app that a developer could just download and hit compile and it runs, make it as simple as possible.


I would look at these very complex repositories for the initial versions of TensorFlow and, you know, a neuro conversational model by Oriol Vinyals, who's my favorite researcher still to this day, and just try to condense it into, you know, 1020 lines as a rapper.


But over time, I just, it was like a gradual process of, you know, instead of just raising awareness, it became more like chasing clout, rate making the number go up, number go up for views and likes.


And there was also like, almost no of accountability. I was a lone actor. I wasn't working with anybody. So that definitely made it easier to do something like that.


And eventually, like once I moved from San Francisco to Los Angeles, and that was the last year and a half that I worked on YouTube.


So from 2018 to 2019, that's when I think that was a bad move. Like, I'm not really an LA person, but that's when I really started to really chase the clout and pursue fame for the sake of it.


Because I'd already gotten these opportunities. And it seemed like I just needed to get to a million subscribers no matter what.


Yeah. A million is, was that your personal goal? Or, I mean, for me, a million was always the point a little bit where you could live off of ad revenue. Was it like this? Or was it just a number you liked?


No, it was just a number. It was just like a fun little goal in my head. Yeah.


Yeah. So, and did you, did you, did you at any point feel like, maybe I shouldn't do this maybe at the beginning? And did it become easier for you? Or how did you think about yourself? Or did you just think, you know, everyone else is doing it? Or?


Yeah, I mean, I, I guess I, you know, everybody is the protagonist of their own story, right? I felt like I was doing, you know, just having the little name in the very bottom of the GitHub, not forking the code, but just putting it down there. That made me, you know, feel guilt free at the time. But obviously that wasn't how I should have done it.


I mean, obviously, what you did was very public. And therefore, the backlash I felt was also very public. I mean, a lot of, a lot of people got angry. And, you know, once once it all, let's say, came crashing down, a lot of people came forward and said, Oh, yeah, me too. I was also my code was plagiarized, and so on.


I, I feel like I have seen exactly stuff like this in research, like, tons of times people, essentially copying papers, mildly attributing like once, but essentially the entire page would be would be like, taken from, usually it's their earlier papers.


So what authors will do is they will have like one new equation. And then they'll write an eight page paper where seven and a half pages are essentially their old paper. Right. And so, so I mean, but that is never, it's never as public, right? It's never as, as, as, as big, I guess the more public one is the, the worse it gets when something like this really, really happens.


Did you? So, I've read your Udacity course that you you said that became an issue there, right? People try to tell you, you can't plagiarize stuff. Is that is that correct? Or?


So I've seen like a tweet from someone at Udacity saying, you know, the, the course fell through essentially, because they try to tell you that that's not how they do things. Or what is or maybe you can tell a little bit what the Udacity course you said that was a big thing for you? Why did it fall through?


Yeah, so, you know, the, what happened with Udacity was we had a 16 week course that I essentially designed, and then Udacity helped me build a team around that to help me. One issue that one of the people at Udacity had that I was working with, he was also in the initial trailer video, Matt Liener, was that I was not writing the code from scratch.


I was using existing examples. And he didn't like that. We also didn't have that good a working relationship during the course. But I think in terms of falling through, that happened, like, you know, everybody made money from that course, including Udacity.


And there were several cohorts of students that didn't just run once, I think it ran like three or four times. Udacity actually approached me two years after that course was over to do another version of it. And I did help get back to. In terms of falling through, yeah, when all of this happened, then, you know, people came out and said this stuff.


Yeah, I don't know what happened with the course, honestly, I haven't.


Okay, maybe I maybe I got I got this one this one wrong.


Yes. And so I've seen like, I've looked at your social blade and so on. You're at about 700k subscribers. And I've seen also an interview with Lex Fridman and you, where essentially, you also told him like, you know, what matters to me is views.


I'm attuned to views to more subscribers, and so on. Is it fair to say a little bit that you might have lost sight of, you know, the bigger picture or other things just in pursuit of this goal?


It is it is I was definitely disillusioned with AGI and the initial goals that I had at the start. I definitely also had a, you know, an issue with I had like a drug problem near the end.


I was doing too much of a certain drug that makes you really up and have a lot of energy. And there was a point where I pretty much almost overdosed on it. And that's when I knew like, I even like, you know, call the cops on myself to because I thought I was going to die.


I never really said this out loud before, but that was near the end. This is basically like a month or two before, you know, that scandal happened. And I was just, you know, I just felt like I was unfallible, like I was untouchable, like I could do no wrong.


And yeah, I'd never had that level of fame before as well. That was pretty that was that was quite a drug of its own as well on top of that. But yeah, it was a gradual process, I think, of going from uplifting developers and like that being the primary concern to also then chasing cloud, chasing fame, wanting more opportunity, more views, more recognition.


And just making stupid decisions.


Yeah, I can. I mean, I'm, you know, as as a as another YouTuber, I get the draw of this, like I understand I can, I get this feeling of being sucked into these into these metrics. And it's not only the metrics, right? The metrics are correlated with money, correlated with fame, and so on.


I like, yeah, I see the, and so many YouTubers fall into this, right? And your, your mistake was also a little bit that you, your, your setting was in an maybe like an academic or a professional setting where people actually care about, you know, not stealing stuff and things like this.


So maybe, you know, you, unluckily for you chose the wrong field to do something like this. And because in many other fields, I think this would have just, you know, been been completely fine.


So, in addition to, let's say making videos, and you were making insane number of videos, like two a week or three a week, as you said, and that certainly also you had a schedule that certainly must have also pressured you. But then you also, there is this, there's the issue with your paper, right?


And that that to me, that to me, was really something where I thought, this is someone who is almost like blinded by either the speed or, or the fame or, or, as you said, you felt infallible or something like this.


So for people who don't know, you had written a number of research papers, but this particular one, you even made a video about it, I think, like, I wrote a paper in a week or something, like, and it was about it was about a neural, the neural qubit.


And one of your viewers then went public and claims and could show that this was copied from largely from two other papers copied together that the diagrams copied, and the text copied, and you you changed some of the wording, which was the most puzzling thing to me.


So instead of a quantum gate, which is equivalent to a logic gate, you changed it to a quantum door, which makes no sense. I like this is a meme until today, right?


And instead of complex numbers or complex Hilbert spaces, I think it was complicated Hilbert spaces, which also is kind of if you so maybe if you just if you look back now, what is what is your reaction now to to past you in with respect to that that paper?


Yeah. Um, yeah, that was hilarious. That's eternally a meme now. What I Yeah, I mean, I used AI to generate some words and like make things different. I would so this was automated the replacement. Yeah. Yeah. Okay. Yeah. Yeah.


Yeah, I think there's a tool called like, um, I think it's called like, it's a web tool. I forgot it's like AI writer, or something like that you like paste in a paragraph, and then it like, rewrites it. Um, yeah, like what a stupid decision that was I


but there I mean, at this point, it's really, it's not it's not it's not this. It's not quite it's a step up from copying code and attributing someone at the bottom, right? Because there, you can still say, you know, I attributed them, I'm, you know, I can sleep at night, this is really, I go, I take paper, I put it deliberately into a tool that rewords it. And then I say, here's my here's my paper, right? This is what what made you or


how did you? How did you find yourself making that, that step that, you know, like the really from, I can justify this to myself to, I guess, I don't know what maybe you explain better than me.


Yeah, I, you know, it's just like ego, it's like, I'm untouchable, and I can just do anything. And I, um, I guess I didn't really understand what it's like, before I plagiarize that paper, I talked to an actual quantum researcher who works at in Santa Barbara, for Google. And, you know, he's like, we should write this, you know, I was like, we should write this paper together. He's like, yeah, I'm gonna write this paper.


together. He's like, yeah, let's do it. It's gonna take a year. And I remember thinking, like, that's way too long for me, like, I'm not doing that in a year. I'm gonna do this in three days. And just thinking, like, no, I guess I didn't respect the scientific process enough to. Yeah, it was just down to me, I just thought of it has like a another link in the video description. Just adding it, I should have just linked to the seven papers, I just, instead, I put my name on it and just made it into one and


like, oh, people are gonna like me more because of this. I'll have more credibility because of this instead of the opposite. And I don't know, I was just making in general, just, you know, really, um, drugged out, honestly, like that. I don't know why I made a lot of decisions that I did. I'm sober now, by the way.


Yeah, no, at no point it did it did it ever? Because that's, that's the baffling thing, to me a little bit. And that that that shows me or at least seems a little bit like someone who was really lost touch a bit is that when someone is like a, an experienced researcher tells me it's going to take a year to write a paper. And sure, if I think I'm fast, I can, I think I can do it in three months, right? But three days is a


like, is a different thing. So so clearly, your idea was already, you know, I'm going to take a shortcut. It's not like I'm going to write the same paper in three days. It's just how can I make a video out of this in the shortest possible time?


Yeah, I was like, what's my next video? I wrote a research paper and just thinking about that. That's really the handle like I want to make a video that shows or tells people that I wrote a research paper. Yeah.


Yeah. So a lot of I've seen a lot of commentary saying things like, you know, it's a it's a shame you have a you have a good platform, you're charismatic, and you could have, they say, something along the lines of you, you might have just as well credited all these people, and just had the same effect. Like implying, you know, there would be another way of doing this. You could just say, you know, here is a bunch of people who you know, not doing this.


bunch of code by some cool people. I'm going to show you how it works. And and their implication


is you would be just as famous, you would be just as liked and so on. Did you? First


of all, do you think that's true? And second of all, did you think that's true? Like, or


was it really your conviction? No, if I did that, I would be way less popular.


I do think that that's true now. I did not think that was true then. I thought that I


would have to be the guy with who is behind all of this in order for my brand and channel


to grow. Because. Yeah, because it's just hard, like in the YouTube game to like differentiate


yourself and I felt like this was a way I could do that.


Yeah, I mean, it's, it's, it is true, right? I'm not sure that these people are correct.


Like it's for sure good advice to credit the people whose work you present. But I myself,


I'm not sure if they are correct when they say you would have been just as popular and


just as as, you know, well respected by the people who think you really did do these things,


right? I'm not sure as you say how YouTube works is it's a it's tough game. And you at


some some point, this this all came and together also with your with your course, which we


can talk about in a second, but specifically with respect to the code and, and to the paper,


you made an apology video, which was fairly lengthy. It was not your usual style. It was


just kind of you standing there. You you essentially said straightforwardly, you know, here's what


I did. I credit I didn't credit these people enough just took their code, and so on. And


then people notice that only like a few days later in your next videos, essentially, you


did the same thing like there were slides where where you you took from somewhere and


so on. Is it? I don't know. Is it fair to say and so you made these videos, you made


the apology videos, then you immediately started uploading videos and before you really quit


and you quit for a long time after that, what was what were sort of the last videos like


for you? Or, you know, like, after let's say, the apology video and so on. But before you


quit, what was that like?


You're asking about the time between when I quit to the apology video what that was


like?


No, from the apology video, to the point where you didn't upload for four months after that


or uploaded very infrequently? Was how did you feel at the point like of the apology


video and a little after that?


Yeah, well, I mean, I felt pretty bad. Generally, I'm a pretty happy guy, as you can surmise.


But I can say that's the only time in my life where I've ever felt somewhat suicidal. Like,


just for a little bit. And yeah, I didn't know how to deal with that level of sadness.


So I tried about a bunch of different things. Like I moved from LA, I got a dog, I just,


I don't know, did some soul searching, some meditation, just tried out a bunch of I tried


virtual reality, like escapism as well. It was a pretty tough time, as you can imagine.


But in terms of doing the same thing again, I guess I did, but I didn't think that I was.


Maybe there's something wrong with me. I don't know. I need some kind of mentor to be like,


here is how you credit people in a YouTube video about machine learning. And here is


what people are going to find acceptable.


Yeah. Did you think at some point, maybe I can turn this around? You know, maybe I can,


because you were at the beginning when people brought these things up, you were, I saw just


a bunch of Twitter posts and so on, sort of discrediting them, denying them like, no,


I never, never did anything like this. Was there a point where you thought, you know,


people are getting iffy, maybe I can turn it around?


Yeah, yeah, there was. I mean, I tried everything. I was like, maybe I don't need to apologize.


Maybe I do. That would make it better or worse. Maybe I should just deny, deny, deny, like


politicians do. Maybe I should, you know, make fun of, you know, make like, reply videos


to other YouTubers who made videos about me. There's a lot of things that I thought


I could do. Eventually, I decided and I don't even know if that was the best thing for my


brand. I know it was the right thing to do to make an apology video morally. But I don't


know if that actually helped me or hurt me. I still don't know to this day. Yeah.


Was it? So I think if I hear this a little bit out of you that there was a time where


you were still mainly thinking brand mainly thinking, you know, which actions are gonna


let me still reach like the million subscribers or continue on? And then, was there a particular


point where you thought, no, actually, you know, let's, let's do an apology. Let's, let's


tone it down. Was there? Was there a time when you thought when you consciously let


go maybe of the million subscriber goal?


There was, there was. I think it just came from introspection and seeing how like the


the amount of I don't even know what you want to call it. Feedback, negative feedback or


criticism. It just wouldn't go away. It was just there. And it didn't really die down.


And I thought, I mean, there's really nothing else I can do here. I need to just accept


defeat to wave the white flag. Part of my brand is just like, you know, super confidence


and always being okay with being like haters or whatever. So not even, you know what I


mean? And like, there was a point where I was like, ah, you know, I'll just apologize.


And then I also felt, you know, near the end, I did feel, I started to feel like guilty


because some people said that it wasn't just that I plagiarized, but that I was actually


doing the opposite of like accelerating research in the space. Like this sets a bad example


for people and this actually gets in the way of research and it's going to slow it down.


And that's what I was like, okay, that's if that's true, that's really bad. And honestly,


like I was reading too many comments as well. But yeah, I mean, I still don't know to this


day like whether or not the apology video helped or hurt my brand. In fact, if I had


to bet, I would say probably hurt my brand. But you know, at least I felt better afterwards.


And I guess that's what mattered in the end.


Yeah, I mean, I think few people really understand what it's like to get YouTube comments on


a bit of a scale and people, there will always be people criticizing and hating, especially


I guess you with very little credentials in the field. I guess you have always had people


saying, you know, this is a maybe this is a clown has no credentials, whatnot. And it


didn't help that you copied code because then you not authoring the code also meant you


knew less about the code, which might also be sometimes shine through a bit in your videos.


But I think you with time you sort of learn to tune out the haters because you're going


to get them anyway. But then sometimes they're right. Right. And I think it's I think, you


know, I don't think and I don't think many people in the like public sphere get like


have a good good understanding of when should I listen to the to the bad comments and when


not because usually it's no right. So right. Yeah. So then then this this was this was


very shortly people really complaining about plagiarized code. And this this paper, which


was one of the sort of big points raised. And then in a very short like within a month


or so there was also the issue of a course you offered, right? So you you maybe can you


tell a bit how this course even came to be? You you made videos at an insane rate? How


did you how did you think you could also offer a course and why?


Yeah, I think it comes down to two things. One, I felt like I could do more than what


I actually was capable of doing because I my ego was so inflated at the time. So I that's


one. The other is just looking at the metrics. Generally the videos that were about making


money were the ones that did the best. And so I started to follow that trend and tailor


my content in that direction as opposed to what I would have done years ago, which is


like how do we solve them? You know, millennium problems like poverty reduction and water


cleanliness and environmental sustainability, things that actually matter. The course was


around that. Like, well, people want to make money. Let me make a course around making


money with machine. That was what is called, right? It was called make money with machine


learning. That is a hell of a clickbait. Yeah. The most clickbaity, exactly what's going


to get the views title.


And it was supposed to be a paid course. It was I think about $200 per student. And the


issue the first issue was that you claimed it was like a limited entry course with personal


supervision. Now both of these things didn't really turn out to be accurate as as you promised.


So there was an issue of you said, I only let in 500 people. But then you let in twice


500 people. So you had two different slack work spaces with twice the five, I think one


even had 700. But there's a few extra ones, I guess. And then also there was apparently


not really like you can't you can't personally supervise 1200. Like it's impossible. Did


you plan on these things already? Or did they just sort of how did they happen?


I didn't plan on them. I did think that I would have 500. When I put the course out,


there were so many signups so fast. And I got greedy. I was like, I'm just gonna I'm


just gonna let this keep on going. Let's see how many people can sign up for this. And


I thought, yeah, I can just have two different cohorts. And you know, I had people volunteer


to help at the time. You help me like as I guess you call them teaching assistants.


Yeah, but they they how many roughly how many TAs did you have? Do you remember?


There was at least one there might have been written that there was at least one. Yeah.


But they they sort of did they quit after a while? Or did they stick with you? No, they


actually they were amazing. They stuck the whole yeah. Yeah. Okay. But they were they


were volunteers? Yeah. Yeah. Okay. So it was 200 bucks and like 123 maybe volunteer TAs


for 1200 students. And you did you plan on ramp? Did you realize at some point, I can't


provide personal feedback to all of these students? Or did you just think, you know,


whatever, I'll, I'll just I can do this or I did, I did realize I was in over my head.


I think it was like week two or week three, that it really started to dawn on me. And


then I think, I think it was week four that some of the students started, you're going


to social media. And then everything came crashing down in the middle of the course.


And then I had to give out a bunch of refunds, but still had to finish the course to the


end. It was a 10 week course. So we still have to keep going for five weeks after that.


But yeah, I mean, there were still, you know, hundreds of students who stayed in the course.


I don't know if you know that, like, the register made an article on this, but they didn't say


like, it's not like everybody just dropped out all of a sudden. Yeah. So people in the


course, I still had some responsibility.


Yeah. So I maybe briefly summarize these these articles, and, you know, they're, they're


written from a certain angle, right. And that's, that's exactly why I also wanted to get your,


just your side of this story. So these articles, they claim, for example, that, you know, people


started noticing there was no personal supervision, they complained, you, you, you never essentially


showed up in the slack workspaces, well, or infrequently, they all got the same feedback


on their exercise. So that was sort of like a copy paste of like, good job. It was it


was like that, then people started demanding refunds, but were some claim they were even


banned like for demanding refunds. Then it was also claimed that you eventually said


there was a refund period, which was for 14 days. But the article claim you quietly


introduced the refund period 30 days after the course started. So it was essentially


impossible for anyone to have known because there was no refund policy at the beginning,


you introduced a 14 day refund period, 30 days after the code, the course started, you


then and then you know, once once people discovered that there were two different cohorts, and


so on, how, what of these articles is, is true, and what is overdone. So there are also


several several tweets of students that said, Yeah, people claiming refunds were were banned.


Or or that the fact that you introduced this refund period, how did this go down from your


perspective? So Paul, that is true. What I don't I think was overblown is the banning


part. I never personally banned anybody. But I can't speak to whether or not one of the


PAs may or may not have done that. I love that. But yeah, everything else like definitely


on point like it's all a part of the story. Yeah, can't refute any of that.


Yeah. And did you did you get? Did you get scared at any point? Or did you were you still


in this year? Because all of a sudden, people and their money are involved, right? It's


not, I mean, 200 200 bucks is not that much for maybe an American, but it is a lot for


maybe someone in India or something, you know, some place like this. Did you get at some


point, you know, scared because like, wow, there's actual money here, that I may have


to pay back or?


Yeah, I mean, I got scared for a lot of reasons. I was scared that. Yeah, I would like have


to go through some kind of lawsuits, people were saying like, oh, there's going to be


a lawsuit, you're lucky you're not in jail and stuff. And. Yeah, about the refund stuff,


like that 30 day versus sneaking it in and I'm sure I'm sure I did that. I honestly don't


remember it now. Like I'm sure like that's probably what happened. But I mean, when I


look at it now, I'm like, I mean, when you charge money, you need to be very upfront


with people and like, that's how you make a sustainable product. I wasn't thinking very


sustainably in long term. It was a very short term thing. But I was scared. Yeah, I was


here.


Did you? But but your thought was still I can educate these people even if I can't give


them personal supervision? Or, or was it was it all like, you know, like, I'm gonna get


their 200 bucks, I'm going to tell them something so they can't complain? Or did you still think,


you know, I can't like the course has value for the people who are in it?


No, I did think the course had value. I mean, it's weird, because it's like I'm conflating


my bias against academia and the traditional learning path with this course that is, yeah,


it's got a super clickbait title. But, you know, I guess I didn't fully appreciate what


online learning and I'm still learning what online learning really can be in the future.


I thought, well, you know, you don't need to be in a physical classroom to learn like


I think we can all agree to that now like you and watch videos online. But also, you


know, what is personal supervision? And does there need to be x, y and z for someone to


be able to say I learned a lot of learning comes from self motivation. And, you know,


education is not a scarce resource. It's it's abundant. It's the desire to learn that is


scarce. And perhaps that alone, I felt justified, like if I could get them to want to learn


these things, that would be enough. At the time, I felt that way. Now I know like, what


would I change differently besides the obvious part, like the 30 day refund from the start


is to just hire help. Like if I were to give advice to anybody doing anything like this,


like any YouTuber who wants to make a course like hire help, step one, hire help, then


figure everything else out. Don't plan it out yourself. It's too big. It's too big at


scale for one person to do.


What, what happened? Did you end up giving refunds to people or?


I did.


Did you did you still have enough money to give the refunds?


Haha. Um, I, yeah, I did.


What happened to the money? Like, I can imagine you get 200 bucks 1000 people that's like


200k. How where did that go? Did you end up plus or minus or did you spend on refunds?


Did any lawsuit result or?


There were no lawsuits. Everybody who wanted a refund got a refund. There were still a


bunch of students who completed the course to the end like, and I'm very thankful like


despite all the drama, they were loyal to the to the thing and so was it it wasn't negative.


It was positive. It wasn't nearly like probably like 10% what I made at the start.


And and then, you know, I think this as I said, this was within like a month of of every


everything down you you were making lots of videos, the paper, the course all at the same


time. And then everything, everything comes crashing. And I think it's one it's one thing


when you feel bad because life is is crap, right? Because something happened to you that's


bad. And you know, but it's an entirely different thing. When you're you you know, you're responsible


for it. Right? Like that is that is worse. That is like, my life is bad. And I'm to blame


and you know, like it's it's my my doing, right? Like, was this, I guess this was your


experience, right? It you know, whether you thought it was good or bad, it was like, my


life is crap, and I'm responsible. How did you? What did you do at that point? You said


a bit of soul searching and so on? How did you decide to to go forward?


So I moved back to San Francisco, how I was there for a few months, I basically invested


in my friends and family, talk to them, that helped got really to virtual reality that


helped as well, like disassociating from this reality, bring it to a virtual world, where


I was anonymous, and logged off of all social media as well. So that helped as well. And


kind of just gave up with the whole like, you know, million subscriber path that I was


on. And what else? Yeah, just Oh, yeah, focus on my health as well. Like I was like, I'm


just gonna like, try to focus on being healthy, because I can control that I can control what


people think about I can control my health. So that helped.


You made it you made a quite astounding body fitness transformation as well. You were at


the end, you were like in 2019, when it all crashed, you were kind of a like a chopster.


Yeah. Right. And I saw like a before after picture. Was this a conscious effort by you?


Or?


It was it was Yeah, because like, part of like, what you having a desire to live is


to like be able to look in the mirror and, you know, say, like, for me, at least like,


Hey, this is an attractive guy. So that, you know, it's kind of vain, but it definitely


helped for sure. Like, that. Yeah.


And so you eventually you got, let's say, back up on your on your feet after all of


this, what was your or what is your current plan? Or what are you doing right now? You've


you've posted a few videos again here and there, but I'm not so maybe, you know, what's


what are you doing, essentially?


So um, yeah, making videos on this series called AlphaCare about healthcare and AI,


which is kind of always been like my the industry I'm most excited about for AI, like applicability,


like, oh, we can make people healthier. So doing that, I'm almost done with a book I've


been writing for the past three months, which is going to be a free ebook, not going to


charge for it. So that's been interesting. That's also on like deep learning for healthcare


apps for beginners, but examples in there. And once I release that, all of this will


be done in like, three weeks, probably for now. Like the Siri, the video series in the


book, then I have to figure out what the next thing I'm going to do is what I'm most excited


about currently is paying people to be healthy. There's this app called sweat coin. It's out


of the United Kingdom. It pays people in cryptocurrency to walk. I find that really, really interesting


because, you know, two of the most meaningful things to me are keeping people healthy and


reducing poverty. And this kind of does both at the same time. So I'm wondering if there's


a way to create what's called a Dow, a distributed autonomous organization around healthcare


and help data and keeping people healthy, paying them somehow with cryptocurrency to


stay healthy. I just use this service called inside tracker, which cost me like 500 bucks


way too expensive a service for most people to use. But I got a blood test done two weeks


ago using the service. They took 43 biomarkers of mine. And now I have a bunch of health


data like my cholesterol level is apparently way too high because I eat way too much red


meat. So I've got to cut down on that. But something like this, if we could turn into


like a free service that keeps people healthy and actually not just free, but pay them money


and then somehow turn it into a business where also the service makes money, that'd be really


cool. So I'm kind of like thinking like I'm going to start some kind of company around


that or a Dow, I should say. I'm not exactly sure what it looks like, though.


I mean, this is happening in part already with, I don't know, we have like high taxes


on cigarettes, right? So essentially, the smokers, they finance a little bit, the non


smokers via taxes, some health insurances, they already give discounts if you do like


regularly go to a gym or something. So I'm like something like this is definitely in


the realm of possibilities. Now, with respect to cryptocurrency, is this a meme? Or was


there actually a Siraj coin at some point? I haven't found anything like what? What was


that?


Yeah, that was a real thing. I launched a cryptocurrency I think two years ago or something


three, I don't know, called Siraj Coin. And people really didn't like it. So I took down


the video. You could find it if you really search Siraj Coin.


Okay. But it was just it was more like for a video? Or did you think, you know, maybe


I could make some money with launching my own cryptocurrency?


Yeah, both. I mean, this was at the height of the ICO craze. And everybody was doing


it and I felt like, wow, I'm going to do it too. Here we go. Siraj Coin. And the idea


was that you can with Siraj Coin, you can get a meeting, like buy a meeting with me


or like make a music video with me. Just, you know, I am the scarce resource. Like in


these cryptos, there is a scarce resource, the token is how you access the scarce resource.


And yeah, I mean, I'm glad I did it still. Like nobody got hurt from that. It was just


like a fun experiment. And I learned a lot from it as well. Like I still think it's an


interesting idea. Like I do think that we're going to see more individuals create tokens


around themselves. And yeah.


I mean, yeah, a couple of NFTs work this way, right? That there's some kind of like a meeting


with a famous person tagged on Twitter or something like this. Yeah. So with with respect


to your, your book and your new set of videos. And, you know, I guess that the question everyone


asks is, is there still how do you handle citations, plagiarism, things like this? Are


you are you toning it down? Or are you like extra super duper careful? Or what is your


sort of how do you approach this topic? I guess you're in a bit of a special situation.


Not not only are you held to the same standards, but now, you know, people read your name,


they're probably the first thing they do is put something into a plagiarism checker.


Yeah, I'm super careful. I put it in the video description, not just like the GitHub. I say


it verbally. Yeah, I just try to be more careful. Yeah.


And the what's the book about? Can you is there? Is it something you can disclose already?


Yeah, it's on bioinformatics for beginners. I'm also a beginner to bioinformatics. I'm


really interested in multi-omics, like all the omics, genomics, epigenomics, transcriptomics,


and just thinking about how we can integrate all of these different types of data to make


both diagnostic and prognostic predictions for people. And I think that's the future.


I'm really interested in reversing the aging process. David Sinclair at Harvard has a great


book on this called Why We Age and Why We Don't Have To. He has a podcast that he's


going to release next year on this topic. And I just think that there's a great space


for data science and data analyst enthusiast to make a contribution in this field. Because


I do think the future of health care isn't going to be targeting individual diseases


like Alzheimer's or heart disease, but rather that this disease that is upstream of everything


else aging itself.


That's it. I mean, it's a it's a tough task. But yeah, it's a it's a I guess it's a cool,


cool outlook. It seems like a little bit of a rebirth. You know, you told how you were


at the beginning of your video career thinking if I could just, you know, make video about


these cool topics and so on. And it it almost feels or at least to me, it sounds like it's


got a little bit of that same spirit again.


I'd like to think so. I mean, I, I don't have the same, I don't know, I don't have the same


level of or maybe I just feel this way. I don't have the same like energy that I did


back then where it's just like I have to do this or else like the world is going to end


like that level of conviction. I just feel like I mean, I'm really interested in biology


in general. I don't think I'm going to get I honestly don't think this is going to get


me the level of fame or opportunity that talking about deep learning from 2016 to 2020 did.


It's just something I'm interested in. And I'm okay like not reaching a million. I mean,


probably never going to reach a million subscribers. I just want to be interested in this. And


even if you know, if this like company doesn't work out, I'm happy to like take a job somewhere


and just like learn about bioinformatics full time as a bioinformatician, atlist or something.


Yeah. Well, in Yeah, I mean, in many ways, I, I've told you that this this privately,


but in many ways, you were, you're sort of with with all of this happening, you were


still sort of the pioneer of what many of us other ML YouTubers, essentially, that the


path we go is you, you made it a kind of like, I remember when I started making videos, there


was like nothing. And when you started, there must have been like, really, really nothing,


right? And, you know, that for for all the things I think it took, it took balls to go


that way. And you, you certainly hustled, even if it led into like, a wrong direction.


Do you have, I don't know, do you have do you have, because I know that there are quite


a number of people who look at maybe you also me, other YouTubers, a lot of people are starting


their podcasts nowadays. A lot of people also start channels like mine or or similar to


mine. Any advice you have for people starting out in in the in the sphere of online education


or what might what we might call being an influencer, anything like this?


Yeah, I would say that you this is not something you do as a side job. Like a lot of people,


you know, kind of have to because they need a source of income from their day job. But


I would say like, the only way to be successful in this is to pick it to be your one thing


and do that all day. And it's got to feel like play to you, but it's got to look like


work to other people. Like to me, this whole time I've just been playing like really enjoying


myself, like it's not work. And that's honestly why I think I grew as much as I did. I genuinely


enjoy the topics. I genuinely enjoy the video production process, editing, lighting, thinking


about metrics, all that stuff just felt like play to me. And that's how you're going to


be successful. It's not going to be if you feel like it's hard work. You should pivot


or think of some other content to talk about or maybe a different medium. Like, you know,


I had a podcast as well. I did, I think five interviews and then I stopped because it didn't


feel like play to me. Like I don't actually. For some reason, I just don't enjoy being


a podcast host. Like I enjoy monologues and that kind of thing. So I stopped. Whereas


someone like you or, you know, Joe Rogan or other podcasters, they actually enjoy it.


So they're going to they're actually going to be successful. So that's that's my best


advice is like, make sure that it feels like play to you and then you will be you'll probably


be successful.


And when someone finds themselves a bit successful and finds themselves to be sucked and drawn


by the metrics by the clout by because I already I already said it but I'm gonna say it again.


Like this is a this is a thing. I feel it. I like other YouTubers feel it for sure. This


this suck. It's like a it's like a thing drawing you right. And, you know, leading to the kinds


of decisions you made and what is do you have any? I don't know. You know, other than don't


do it. Do you have any you know best the mindset that that creates in a person? Do you have


any any maybe recognition of what could help someone to to get out of it or to resist or


what do you tell yourself when there's like a really easy opportunity to get a lot of


views or clicks?


I would say the best thing you can do is Google Sir Roger Ball and happen to this guy. And


yeah, just be afraid. You don't want that to happen to you for sure. Luckily happened


to me first. So you've got an example in front of you now of what can go wrong when you follow


views and likes too much. You chase cloud too much in the education space. The Internet


gives everybody a voice. You will be held accountable. There is no we are moving into


a world that is much more transparent every day less and less privacy. Yeah, the Internet


gives everybody a voice and power. So yeah, that's that's where I can say use it. Use


it wisely, I guess. Use it wisely.


Well, Sir Roger of all this was this was a pleasure really, truly. I I thank you very


much for for being here with me today. Thanks for coming on. Thanks for being so open and


forward and and and honest. I think it's very valuable. The world also hears from you. And,


you know, in it, not just from articles and and and, you know, reviews and things like


this.


Absolutely. Thank you, Janek.


Intro/basic GUI - PyQt with Python GUI  Programming tutorial

What is going on everybody welcome to another Python tutorial video series. This series is going to be talking about


PyQt with Python so PyQt is a wrapper around the Qt framework


Which is a framework for creating GUIs


We're going to be using the Qt4 framework. There are two frameworks. There's Qt4 and Qt5. Qt5 is


a slightly newer, but it comes with a lot more


Dependencies and it also has a slightly different


Licensing kind of process and Qt4 does Qt4 is much simpler much more black and white and vanilla to use


So therefore I think it would be best to go ahead and use Qt4


so


Qt how does it differ from say Tkinter or WX Python because all of these are GUI


Development frameworks, so why should we use PyQt or learn another framework?


a lot of it's going to come down to user preference or at least developer preference and


Qt is a little more updated. I would say


Back when maybe a few years ago


I wouldn't have really said such a thing but maybe even WX Python would have come first because they were constantly putting updates out


But now I would say Qt is by far the most up-to-date


Framework for creating windows and it's going to look the best on a lot of systems, so this is going to be the most cross


platform


Compatible GUI that you're going to find pretty much


And you can you can easily change the style so like with Tkinter you can really only change to like one


Style until you need to like you have to code the style completely and that takes a lot of code


So with Qt it comes with a bunch of styles, but it also comes with a lot of really intuitive widgets that just make sense


To have pre-built in so obviously there's a lot of stuff with Qt or any framework that they can't


Envision that you want to do but Qt comes pretty well packed with a lot of widgets and a lot of widgets that are


Going to cross platform and make sense per operating system that you use it on


So anyways now that I've sort of sold you on Qt 4 the best thing to do is really just try them out try out


All the frameworks see what suits you best because they're all a little different and most importantly to develop with them is a little different and


Honestly Qt 4 so far for like me if you're going to make a big application


My experience with Qt it so far has been really good actually I like them more than I really liked Tkinter


Initially, but Tkinter makes it pretty hard to make a large application whereas Qt honestly Qt makes it pretty darn simple


So let's go ahead and just jump in so the first thing that you're going to have to do is actually acquire


Qt so to do that you're going to come to riverbank computing the parent company


And you're going to want to go to this link basically software py Qt and download now if you're on windows


That's what you want to do pretty much you come down here and download the installer that suits you okay?


So this would be Qt four point eight point six for Windows 64 and then this would be for Windows 32


now


If you're not on windows if you're on Mac or Linux


You should be able to get by with a simple pseudo app to get install Python Dash


Dash Qt 4 that is all you should have to run and that should install


Qt 4 for you


once you have those things drink your coffee and


You're ready to move on so


You're going to pretty much always have the a couple things you're going to pretty much always import sis


And you're almost always actually you're yeah


There's some scenarios where you might not import the gooey aspect, but anyway from pi


Qt for import


Qt


Gui now


This note the caps here. They're kind of confusing especially gooey. I want to capitalize everything in gooey, but no it's just the capital G


They followed pep 8


Past the whole caps thing anyways


Pi Qt for import Qt. Gooey, so so this is like they basically Qt. Gooey is all the major gooey aspects


There's also Qt. Core which will use kind of not too long


in the future


And then those are the two major ones that you're pretty much always going to use


but there are other ones so


Qt comes with


Things like Qt open gl for including open gl which is really cool


You of course you can always include open gl and your wx python your tkinter


So all you fanboys don't need to jump out at me, but it's a little easier in Qt


So you can do that there's Qt XML Qt SVG so XML for XML obviously SVG support vector graphics


Qt SQL Qt networking and the list goes on so there's a lot of


Kind of like built-in handling for these sorts of things now of course again like with Qt SQL


Do you have to use Qt SQL for your SQL handling no?


But it's going to be a little more useful


So anyways moving on you're going to basically always have these two imports for sure if you're making gooey surely you're going to use the gooey


Anyway


But I can envision some people might just happen to really like I don't know Qt network you like the handling of it


So maybe you use Qt network with something else. I don't know so anyway. You're going to have those two imports now in your application


Your gooey whatever you're always going to have the same kind of three major


Aspects first you're going to have some sort of app definition so the whole application so app equals


Qt gooey dot Q


Application and then here we're going to pass sys.org


V now if you're not familiar with sys.org V you can come to python program in a net


Start learning go to search type in sys and here's what you want and basically what it's going to let you do is


Rv is going to allow you when you could call this application from the command line and you could pass arguments


Via the command line now you don't have to put sys.org V here, but you have to put something


You can't go home, but you can't stay here anyway


You can put in here empty list or something like that, and then you don't have to put anything there


you'll see some people do that, but


Sys.org V makes the most sense to put there anyway


But some people are just like really adamant about not putting sister Harvey's will do this instead so anyway


You do that next you're going to have something that defines the gooey the window whatever it is


So we're going to go ahead and say


For now let's say window


equals


Qt gooey dot Q widget


Now one thing that I do like that pi Qt has done for us is


generally when you're referencing any sort of Qt class


They're going to go ahead and throw a capital Q in front of it


So you can look at your code really quick and know wait is that my method or is that Qt's method?


You look at it, and you'll just know because of the Q


Pretty fast and they do that for a lot of things so I really do appreciate that from pi Qt


So anyways that's just a quick widget and then finally you're going to always have


Window dot show so you're always going to have a dot show


To your application otherwise what happens is everything is created in memory


And then as soon as you call it to show then it shows because that's how graphics work


Graphics are all created kind of in the background


And then when you finally want to show you show because graphics take the most memory and it just makes sense to build it first


In the background and then bring it up, so that's your application


And that's basically you're always going to use this sort of structure now of course there's going to be a lot more


Involved as we go down the road, but for now. This is a basic window


So let's go ahead and save and run that with f5 oh


Blind anyway from pi Qt for import Qt. Gooey


drink more coffee and


Here you have your basic window


Now we've just done this in three lines, but really more than three lines went into this so this window can be resized


It looks like a window


It looks the same as and you can bring this window over it kind of follows that same kind of see-through look it's got the


Same buttons here for anyone who's curious. This is for team viewers, so you probably don't have that button so don't feel like


You're you're falling behind because you don't have that button


But this window resizes it's even got a little bit of a pop-up window


I'm sure you'd call that and most importantly it can exit out


So we already have all these little handling features that we didn't have to code at all for ourselves


Now the other thing I'm going to show really quick before we move on to the next video is


the title to your window and then also


Some the settings for the window and then also the window


So you can see that the window is actually a little bit more detailed than it looks


so we're going to go ahead and just put the title to your window and then also


Some the size and starting point so the once you've defined your window you get this nice window object


what we can do is we can go window dot set


Geometry and


You can say the set geometry takes four parameters. It's the starting point


So starting x starting y and then the dimensions so the x and y or the width in the height


So for example if you work with a GUI application


The starting points are really up here so 0 0 is always like top left


Most GUI applications are like that


Like Keevy is an example of one that's from the bottom left


So that's really annoying for me because I'm used to like everything top left


But if you work with like HTML or anything else


Or all the other GUI frameworks, it's top left anyway, so set geometry


We're going to have a starting point 50 50


pixels wide and 300 pixels tall


The other thing we can do really simply is window dot set


Window title and then you can pass in whatever window title you want. We'll call this pi QT tuts


So let's go ahead and save and run that one


And that comes up and here you have a nice window it followed our rules about being through 500 wide


300 tall and it did start at 50 50 now one thing


I will just mention is you can start a window at 0 0 0 0 0 is going to show us a


Little bit more of a fundamental thing about this window so as you can see my windows off the screen


I film in 1080p


It's my full monitor at least for this monitor


But I have a monitor that sits over here and a monitor that sits up here so above this window


So there's one to the left actually there's one to the right too and one above so


This window didn't really appear on the screen. It's on three total screens


So if you just had one win one monitor chances are it probably appeared like this


But it tried to go up here, but your operating system was like that silly


So it stuffed it down here for you because it has a nice little handling for stupid choices, so


What happened there well 0 0 what is that in reference to well like I was saying you've got an


application and then you've got a window and


You might also hear this generally referred to as you've got a frame


And then you've got a window so the frame is what's around the window and then you've got the actual window which starts


Here and that's why when we said started at zero zero


This point was at zero zero not like this fake corner that should be here without the rounded edges


So keep that in mind it wouldn't matter you could do zero zero all day if you want


But it will show up off-screen for some users, so that's kind of annoying sometimes


Anyway, that's that with this simple application normally


It's going to be you're going to be using object-oriented programming of classes and all this and we really did use object-oriented programming


You just didn't see it yourself, but there's a lot of basically everything here is an object


And we're modifying the object here, so anyway


That's what we're going to get into in the next video is kind of converting this to the sort of


Structure that you're going to tend to use when creating an application


So that's what you guys have to look forward to if you have any questions or comments up to this point


Deep image reconstruction from human brain activity (Paper Explained)

Hi there, today we're looking at deep image reconstruction from human brain activity by


Gowa Shen, Tomoyasu Horikawa, Kei Majima and Yuki Yazoo Kamitani. This is like reading thoughts. So


I was excited when I saw this paper. I saw this on Reddit. And it is a bit older, it is from the


beginning of last year. So I'm sure there have been developments in this area. But basically,


what this paper does is it will have a human look at a picture, as you can see, for example,


right up here, it will measure the MRI activity, then it will use a what they call a feature


decoder in order to map that MRI activity to features of a deep neural network. And then they


will reconstruct the image that is closest to those features in the neural network. And by


reconstruction, they get basically they get out an image of what the human sees. So we're going to


explore this pipeline right here. But it's pretty cool. And if it works, you know, it basically


means that we can read someone's thoughts. But of course, there's going to be issues and problems.


So first of all, this is all visual, they measure the activity in the visual cortex right here. So


let's break it down to the individual parts. The individual parts here, they fMRI that is a machine


we cannot control, right? So you measure the fMRI activity. And that's basically measures which of


the cells in your brain use oxygen. It's it's functional fMRI, it's not structural. So it


measures which ones are active. And that's how you would see which parts of the brain or brains are


active. And I think the resolution on these things has gotten very, very good. So you can make out


very fine grained activation patterns in the neurons. And they measure the visual cortex,


which is the part responsible for basically visual stimuli. So for for seeing things. Now,


they need this, this feature decoder. But because ultimately, what they want to do is they want to


have these features correspond to features in a neural network. This DNN here is good is a is like


a VGG, I think a VGG 16 or a VGG 19. It's called a VGG 16 network. This is a couple of years ago,


these architectures were very popular for ImageNet. And they're fairly basic. So what


that means it's not like a super duper inception net where you have like layers within layers,


and so on. They're pretty straightforward convolutional neural networks with nonlinearities


and pooling. Here, you can see there is a bunch of layers, then there's pooling, there is a bunch


of layers, there's pooling, and so on. So what you want to look at are these layers of the deep


neural network, right, the individual layers. And you want to you want to basically put an image


right here into the neural network, and then observe its features in the neural network.


And then you want to put the same image through the human. And then you observe the MRI features.


And you know that is the same image. So you can basically learn a feature decoder, this is going


to be another sort of machine learned, maybe a neural network. I haven't actually read this


could be just like a linear regression or a neural network, just a regression from that maps the fMRI


to the features. So this is what you have to learn. Basically, they took a bunch of humans,


stuck them in an MRI machine, right, got out their their fMRI data for the same image. So


for a given image x, they got the human fMRI data. And they got the VGG, VGG, they got the features


when they put the x the image through the neural network. And now they learn a function


that minimizes the error. So there is an error. So basically, they run this on a test set


of images. And then you end up with a function that can basically map fMRI features to neural


network features. Now the second step now you you can you can give the human let's say this works,


you can give the human an arbitrary image. And it the human will basically interpret it through its


visual cortex, you measure the activity in the visual cortex. And then you can predict the neural


network features if that image were given to the neural network. Okay, but now you you don't give


the image to the neural network instead of what you do, you do something like deep dream does.


That means you start from a noisy image at the beginning, and you try to find the image. So you


start from this noisy image right here. And through iterative gradient descent, you refine this image,


and that's this arrow right here. You've tried to find the image that as closely as possible in the


internal representation matches the features that you predict from the fMRI signal. Because these


are the features that the neural network should see, right? If your if your feature decoder is


good, these are the features that the neural network should output for that image. So you're


basically trying to find this image right here. But you're not looking at it, you only look at


the features that should be in the neural network. So after a bunch of steps of refining that image,


you hope you basically end up with an image that corresponds to these features. And then you you


can look at it. And it usually looks something like this, we're used to these kinds of things


from neural networks. And if I invite you to look at something like the open AI Atlas, if you want


to understand how this is done, but basically, we can get the image that most faithfully corresponds


to these features. Now, this doesn't always work super well, because there are actually since the


neural network is sort of a dimension reduction technique, there are many images that correspond


to the same features, and they often end up like really weird. So what they do is they have this


deep generator network as a prior, basically, this is the generator from a GAN from a generative


adversarial network. So this network right here is really good at just producing naturally looking


images. And now, because we have that, our task is not going to be to start from this image right


here, our task will be to do the exact same thing, but with the input to the deep generator network.


So basically, what we're trying to do is we're trying to find the input vector to the deep


generator network, such that these features right here correspond to the features that we


predict from the fMRI activity. And because the deep generator network is trained to produce


natural looking images, this will always give us sort of a natural looking image, no matter what


our input vector is. So thereby, we basically constrain the optimization procedure to only


output natural looking images. So let's see how well this works. So these are the reconstructions.


I have to say their training set, their training and testing set. So up here, this procedure where


we learned right here, this would be done on a training set of images, and then they expose the


humans to testing set of images. So this reconstruction right here would happen on images


that the feature decoder wasn't trained on, but the humans are looking at it. So the humans are


would be looking at the picture here on the left. And then this would be to the right, you'll see


as more and more iterations of this process of reconstructing happens, the image gets basically


clearer and clearer. And you can see on the right, you get pretty good looking images for the ones on


the left. Now, these researchers, they tend to, they tout the success like to say, Oh, look,


look at this. But, you know, honestly, like, this is a leopard. This is a dog sheep.


This is an owl. This is a dog. So, you know, this is a fish. And this is a shell, like a muscle.


So I'm, and you can go through these. And basically, you know, this is a sled.


And then this is a truck. So they go basically, and they say, wow, the accuracy is really good. So


they do a pixel correlation accuracy, which is, you know, you just try to pixel correlate things,


but they have human judgment. The accuracy via human judgment is over 95%. That's crazy. But how


do they do it? So basically, they tell you, okay, see, this is the image at the beginning, no,


sorry, they give you this image right here as a human rater. And then they give you two other


images, they say, okay, here are two images, let's say these two, which one did it come from?


And if the human can can determine it correctly counts as a hit. So they say, okay,


determine it correctly counts as a hit. So the baseline probability here is 50%. So basically,


right is so this right here, is it rather the owl? Or is it rather the VCR? And I mean, in that


respect, it's pretty impressive what you can read from a brain, but in no way, like zero way, this is


is reading your thoughts. This isn't like, it seems to basically just reconstruct a example


from the image net training set. And the image net Explorer is down right now. So I, I tried to


look at this. But it seems to me, it's just kind of reconstructing something it knows that sort of


bit resembles the image on the left. Yeah, but it is not it is not reconstructing that image,


not at all, like look, like a bit, but vaguely, vaguely. But they do some more investigation into


this. Okay, so well, first of all, here, you can see what happens without this deep generator


network. So when you have unconstrained search, then it is even worse, right? You get like big


pixel meshes right here. So you need this this kind of prior over natural images, but the prior


I think the prior here comes through a bit much because I the prior might be in part responsible


for why the images just show something else than you see. They go into an investigation of if, and


they discover if you use more layers to reconstruct, the reconstruction gets better. So here,


according to human judgment, if you just reconstruct from the first layer, you don't get


very good reconstruction. But if you incorporate the signal across many layers of the neural


network, so you're basically trying to match to predict many layers of signal different layers,


then the reconstruction gets really good. And, you know, we know this from things like style


transfer, you can modify how close you are to the original or to the target by basically seeing


which layers and how many you reconstruct to which accuracy. So this makes kind of sense. So if you


only take the first layer to match the features, then you get basically this blob here. But if you


get layers one through seven, you get pretty pretty okay ish thing that looks like this thing.


I guess these are without the deep generator network so far. But now that that is interesting.


And I think the novel thing here is one of the novel thing is that they actually use multiple


layers of the neural network to reconstruct. The interesting thing is, I think, think this is this


is pretty cool. They can now do this with these shapes. So these shapes aren't natural images,


and they have not been seen in training. But still, as you can see, when the human sees,


for example, the plus shape, it will get you a pretty clear plus shape. And it happens for a lot


of these things right here. So these are actually I would say, fairly okay ish reconstructions of


what the human sees here neuron, that is fairly neat. And for the alphabetical letters and shapes,


you see again, the even the pixel correlation now is pretty high, but the human judgment


again, high and here the human judgment kind of makes sense, right? If you ask,


is it like this shape or this shape, then it makes more sense to evaluate it like this.


So the shapes, I am fairly impressed that they can reconstruct these shapes. And what they're


now trying to do is they're trying to infer imagined images. So basically, they're telling


a human, please imagine an image and they show it to the human. It's not really imagining,


it's basically recalling, they show you this image, and then you whatever close your eyes,


they take the image away, and you just try to imagine that. And you can see through the


reconstruction process, that works out, you know, sort sort of ish, right, you can see that the


cross here, it kind of comes through. And this plus kind of it sort of comes through. And so


these are the high accuracy, these are actually the samples where where it worked. And there are


also samples where it didn't work like here, where you see it doesn't really come through.


So there either there is, you know, really a difference between imagining something and


seeing something, or this method just isn't very good, per se, and you actually need or humans


aren't really good at imagining like there's lots of explanations. And here is the same thing. If


you imagine these images, now they report that if humans just recall or imagine these images,


then the reconstruction doesn't work at all. So that might be to the fact that, you know,


you cannot in your recollection, you basically just remember the important things about something,


you don't remember the exact pixel values, and therefore, your visual cortex doesn't respond in


the same way. But I mean, it's interesting, even per se to think about it. But I have my doubts


about you know, this entire system. So I don't want to make too many conclusions here about these


things. Suffice to say that sometimes it actually can read your thoughts. Because if you just think


so this is the stuff up here, if you just think of a shape, it can sort of kind of a bit make out


the shape that you're thinking about. Alright, this was it for this paper, I'm basically mainly


wanted to show you what I found. And I'm I'm I have to say I'm pretty impressed with this,


even though like, this is a laptop. This is not a VCR. This is a VCR. It's, it's more of a nearest


neighbor thing, really, than I reconstruction, I think, but that's my opinion, right? Yes. So


if you like this, give it a like, subscribe if you're still here. And I look forward to next


OpenAI tackles Math - Formal Mathematics Statement Curriculum Learning (Paper Explained)

Can AI do math? And I don't mean two plus two, I mean pure mathematics. The paper we're going


to look at today is called formal mathematics statement curriculum learning and presents an


automated system to prove mathematical theorems in a symbolic fashion. What's even more crazy is


that this system was able to solve two problems of the International Mathematical Olympiad,


which is a contest that real gifted high school students get to take part in. This system is way


beyond previous systems that have attempted anything like this, because formal mathematics


and automated mathematics that uses algorithms to prove things lags a lot behind the informal


mathematics that you might know. A lot of previous techniques relied on proof searching,


essentially brute forcing their way to a proof guided by some heuristics. And this paper improves


on that drastically. It uses language models to guide the proof search. And it uses a technique


called expert iteration to build itself automatically a curriculum of harder and harder


statements to prove. Now the implications of this are cool for math, but it goes way beyond math.


This is essentially symbolic reasoning. It's the model teaching itself to learn more and more. And


that's exciting for many fields of AI. So here's how it goes. This video right here is a paper


review a comprehensive review of me going through the paper explaining to you what is in the paper,


what its main contributions are, what I think are the weaknesses and strengths of the paper,


and much more. After this video, you should have a good understanding of what is in the paper.


Otherwise, I haven't done my job. In the next video released tomorrow, I'll be interviewing the


first author of this paper, which is a huge privilege. Because if you watch this video,


you'll see that I have many open questions. I'm a noob at formal mathematics. And I suppose many


people are and therefore, even though the paper is written really well, I had a lot of questions,


I even had some criticisms. And all of that was answered when I spoke to the author. So if you


watch tomorrow's video, you'll get an insight into the behind the scenes of this research,


how it came about, what worked, what didn't, how problems were solved during the research


process, and much more. The author I'm interviewing has actually seen my paper review and is directly


able to answer to any questions that are raised there. Please let me know how you like these


formats in the comments. If you do like the video, please leave a like tell someone to subscribe,


and I'll see you around. Bye. Hello there. Today we're looking at formal mathematics statement


curriculum learning by researchers of open AI, EPFL, and Cambridge. This paper presents or applies


the technique of expert iteration to the domain of proving formal mathematics statements. This is


not enough yet. They also bring language modeling into the picture. So you have a proof searcher in


this paper or a proof search procedure that is guided by language models to focus, to search for


mathematics proofs. And then the expert iteration procedure makes the system better and better and


better by always incorporating new statements that it has been able to prove into its training set.


And so the domain or the difficulty of statements that it is able to prove expands iteration by


iteration. The culmination of this is that they're able to solve two problems, I believe,


of the IMO of the International Mathematics Olympiad, which is a difficult math challenge


for high school students. And this has this has implications beyond just math. So this can be


applied anywhere where agents need to reason over some sort of symbolic structure. And, you know,


this is wide ranging. This could be agents acting in the real world. This could be reinforcement


learning things. This could be, I don't know, assistance for clinical trials and whatnot,


essentially anywhere where such a more formal system, more logical type of reasoning is required.


So we're going to look into this paper and what they do. This builds on a bit of other work,


but I think it can be looked at in isolation. So they claim right here in the introduction that


deep learning has been very good at sort of many tasks like, you know, language modeling,


there's vision, image generation. However, they say it has not yet enjoyed a comparable success


in tasks that require extensive planning and symbolic reasoning. And the domain of mathematics


proves is a good domain, because it has these challenges. But also, you don't exactly rely


on external data that much like you can, you can prove things in mathematics, kind of by yourself


in the basement, or in this case, you can verify a proof pretty quickly. So the challenges in this


domain are, it has an extremely large search space, and an infinite action space. When you


prove a statement in mathematics, there are many things you could potentially do like infinitely


many things. It's not only about manipulating the symbols that are there. Often you need to introduce


new symbols, they, for example, they say, you could generate a witness like there exists an


X that fulfills some things where X was never a symbol before. So you have like infinite things


at your disposal. Now the question is, how do you prove a statement? Maybe we'll just direct a little


bit go into how these mathematics proving things work if you really do them formally. So in their


types of system, they have some kind of statement to be proven. So I'm going to call that statement


S. That is a formal statement that just is essentially is the formalization, the exact


writing down of something like a theorem, as you would find it in a textbook. But instead of using


words and language, it uses like a defined syntax in a predefined system. So how to prove the system


in order to prove the system, what you need to do is you need to build up a tree. So you need to


decompose this system in some way into multiple sub statements. And the way you do this is as you


would do as a human, you, you know, you'd have some sort of a proof. And then you say, okay,


in order to prove that I need the following three things to be true, right. So these would be the


three things like this is a sub statement, one, the sub statement to a sub statement three, and


generally the derivation from such like from this to this, I believe that's called a tactic. So you


can apply tactics to sort of reformulate things into its sub into its sub things in. I'm speaking


very informally right here, because as you might guess, I'm also a noob in this domain. And I hope


the interview will tell us a little bit more about how these things work. But as far as I understand,


you want to decompose these things into sub statements. And then the sub statements, again,


you can decompose into stuff. And this is a context free grammar, right? So this sub statement


like this, should be provable by itself independently of the other sub statements.


And you build this tree for as long as you want until the leaves right here are either the sort


of the preconditions for the theorem. So a theorem could be, you know, for any two rational numbers.


So if the leaf right here says, you know, this is a rational number, then we're done, because that's


a precondition for the theorem. Also, if it's like some sort of a lemma that I already know,


or if it's like a fundamental, how do you how do you call them an axiom? If it's a fundamental


axiom, I also stop. So I'm going to build up this proof tree until every single leaf is either


something that I already know, or something that I can assume to be true. And then I have proven


the I've proven the original statement, because the tree represents the proof. Now how to build


the tree? That is the question, right? I could, I could derive many different sub groups, I could


derive many different sub statements from the from the top statement, the fact that I derived


these particular ones that then lead me to approve that is the magic of proving things in mathematics,


right? That's what mathematicians do for a job. And you can already see that this is not an easy,


an easy thing. You might think of something like alpha, alpha zero, alpha go, and that is a good


guess. But whereas alpha go has defined actions, so all of these things that alpha go could do,


are pretty defined, like how it could expand the tree. Not in the case of mathematical proofs,


there are there's a complex and infinite set of tactics potentially involving exogenous


mathematical terms that have to be generated. So quite a challenging domain. The other one,


so there is the infinite action space, which is one of the tragedies problems. And the other


problem is this no direct self play setup. So whereas in something like alpha zero, I can train


with self play. In mathematics proving there is no adversary, I cannot have a two player game and


the two players get better and better and better. It's a statement, you can either prove it or not,


like that it has the difficulty that it has, there is no there's no opponent that can be hard or easy.


However, so they say this, the is it prevents the naive application of the symmetric self play


objective. However, they say that they observe that the key role of self play is to provide an


unsupervised curriculum. And I'm not exactly sure, honestly, how they arrive at that statement,


if that is just sort of their, their hypothesis right here, and the sort of the paper validates


it. I don't see any exogenous reason why I might be true, but it is a reasonable statement to make


right. The self play, self play is really good, because both opponents start very weak. And then


they all get sort of better in steps. And that is essentially a curriculum. So the question is,


how can we come up with an automated way to generate a curriculum for proving formal math


statements? That is going to be one of the challenges. The other challenge, the challenge of


infinite action space, they say that this has been addressed in past work. By sampling from a language


model, we're going to look a little bit into how this is done. But this is by the same authors. So


they have previously dealt with this by having the proof search, like the thing that decides what


node to expand in the proof tree, be guided by a language model that has been trained on a number


of proofs, and that sort of takes a good guess at what to do next. So it kind of guides the search,


much like the value and policy networks in like alpha zero guide the tree search,


because that is also inherently too large. So they say, they empirically show that


when the difficulty of the auxiliary problems is varied, oh, sorry, we skipped a part. So they


they say we propose to supply auxiliary set of problem statements without requiring proofs of


varying difficulty. We show that when the difficulty of these auxiliary statements is


varied enough, a simple expert iteration procedure is able to solve a curriculum of increasingly


difficult problems. And so what they're saying is they're going to provide. So here here is maybe,


you know, statement one, statement two, statement three, that I want to prove ultimately, and these


are really difficult. So what I'm going to do is I'm just going to put like statement four,


statement five, I'm going to put these statements in here, I don't know what's wrong with the with


the pen. Sorry. I'm just going to put these statements in in there. And as long as they


vary in difficulty, so there is like a difficulty gradient, and I just fill sort of the space with


statement six, statement seven, with with various difficulty statements, what I can do is I can do


an expert iteration procedure. So what does the expert iteration procedure do? Essentially, it just


says that I start with some sort of a model that can solve, you know, some kind of a difficulty of


statements that say s six and s seven are the easiest ones, then I take the results of that


system and the proofs that generated to retrain the same system. And that would result in a better


system. And the better system that would be able to solve slightly more hard statements. And, you


know, since I now solve the slightly more hard statements, I can feed the proofs that I found


back into the system, right, train them on those proofs, because I now know the proofs because I